Risk content detection method, device, equipment, storage medium and program product

Abstract

This invention provides a method, apparatus, device, storage medium, and program product for detecting risky content, relating to the field of artificial intelligence technology. The method includes: determining the real-time risk profile of a large model in the current session; the real-time risk profile characterizing the output risk of the large model; if, based on the real-time risk profile, batch review is required, batch review is performed based on the session context of the large model in the current session to obtain risky content detection results; the session context includes reviewed content and currently unreviewed streaming content generated by the large model, the reviewed content including the output content of the large model; if, based on the real-time risk profile, streaming review is required, streaming review is performed based on the currently unreviewed streaming content generated by the large model to obtain risky content detection results. This invention can balance the real-time performance and accuracy of risky content detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for detecting risky content. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, generative artificial intelligence has been widely applied in various scenarios such as intelligent assistants, content creation, and coding. Large models typically employ a next-token-prediction-based architecture. When responding to user requests, to reduce user waiting time and improve the interactive experience, they generally adopt a streaming output approach, that is, generating a character or fragment and immediately outputting it to the front end for display, rather than waiting for the entire response to be generated before outputting it. However, the randomness and uncontrollability of the content generated by large models bring content security risks, such as politically sensitive or value-biased content. To ensure content security, real-time review of the output of large models is necessary.

[0003] Currently, to match the streaming output speed of large models, the output content is segmented into independent text fragments (slices) for individual detection. However, while this streaming segmentation review method is fast and has low latency, it suffers from a serious lack of context. Streaming review focuses on a limited amount of content, making it difficult to identify risks that require contextual analysis (such as semantic combination risks across segments or implicit risks in specific contexts). This easily leads to missed detections of security risks, meaning the detection accuracy of risky content is low. Summary of the Invention

[0004] This invention provides a method, apparatus, device, storage medium, and program product for detecting risky content, in order to overcome the shortcomings of low accuracy in the prior art for detecting risky content and to achieve real-time, high-accuracy detection of risky content.

[0005] This invention provides a method for detecting risky content, comprising: Determine the real-time risk posture of the large model in the current session; the real-time risk posture is used to characterize the output risk of the large model. If, based on the real-time risk situation, it is determined that batch review is required, the risk content detection results are obtained by batch review based on the session context of the current session using the large model; the session context includes the reviewed content and the currently pending streaming content generated by the large model, and the reviewed content includes the output content of the large model; If, based on the real-time risk situation, it is determined that streaming review is required, streaming review is performed on the streaming content to be reviewed currently generated by the large model to obtain risk content detection results.

[0006] According to the risk content detection method provided by the present invention, the real-time risk situation includes risk situations corresponding to multiple classification scenarios; When it is determined that batch review is required based on the real-time risk situation, the batch review is performed based on the large model within the session context of the current session to obtain risk content detection results, including: If, among the risk situations corresponding to at least one target classification scenario, the risk situation is greater than the risk situation threshold, then each content fragment associated with the at least one target classification scenario is determined from the reviewed content. Remove all content except the aforementioned content fragments from the reviewed content to obtain the filtered session context; Based on the filtered session context, batch review is performed to obtain risk content detection results.

[0007] According to a risky content detection method provided by the present invention, determining the content segments associated with the at least one target classification scenario from the reviewed content includes: The reviewed content is asynchronously categorized according to scenario; The content segments were identified from the categorized and reviewed content.

[0008] According to the risk content detection method provided by the present invention, different risk status thresholds are corresponding to different classification scenarios, and the risk status corresponding to the target classification scenario is greater than the risk status threshold corresponding to the target classification scenario.

[0009] According to a risk content detection method provided by the present invention, determining the real-time risk status of a large model in the current session includes: The reviewed content is input into the streaming risk detection model to obtain the real-time risk coefficient output by the streaming risk detection model; The real-time risk situation is determined based on the real-time risk coefficient output by the streaming risk detection model. The streaming risk detection model is trained based on the reviewed content of the sample.

[0010] According to a risky content detection method provided by the present invention, the session context further includes unoutput content generated by the large model; the unoutput content is content generated after the streaming content.

[0011] According to the risk content detection method provided by the present invention, when the streaming content is the content first generated by the large model in the current session, the real-time risk situation is the real-time risk coefficient; When the streaming content is the content generated by the large model for the Nth time in the current session, the real-time risk situation is the rate of increase of the real-time risk coefficient; N is a positive integer greater than 1, and the rate of increase of the real-time risk coefficient is determined based on the difference between the real-time risk coefficient of the Nth time and the real-time risk coefficient of the (N-1)th time.

[0012] According to the risk content detection method provided by the present invention, the reviewed content also includes the input content of the large model.

[0013] According to a risky content detection method provided by the present invention, after obtaining the risky content detection result, the method further includes: If the streaming content is deemed to have passed review based on the risk content detection results, the streaming content is output. If the streaming content fails the review based on the risk content detection results, the output of the streaming content will be suspended.

[0014] According to a risky content detection method provided by the present invention, the session context further includes unoutput content generated by the large model; the unoutput content is content generated after the streaming content; The output of the streaming content includes: Output the streaming content and the unoutput content; The streaming content and the unoutput content are identified as reviewed content.

[0015] The present invention also provides a risky content detection device, comprising: The risk assessment module is used to determine the real-time risk status of the large model in the current session; the real-time risk status is used to characterize the output risk of the large model. The batch review module is used to perform batch review and obtain risk content detection results based on the session context of the current session and the large model when it is determined that batch review is required based on the real-time risk situation. The session context includes the reviewed content and the streaming content to be reviewed generated by the large model. The reviewed content includes the output content of the large model. The streaming review module is used to perform streaming review on the currently pending streaming content generated by the large model when it is determined that streaming review is required based on the real-time risk situation, and obtain the risk content detection result.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the risk content detection method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the risk content detection method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the risk content detection method as described above.

[0019] The present invention provides a risk content detection method, apparatus, device, storage medium, and program product. It determines the real-time risk status of a large model within the current session. When the real-time risk status indicates that batch review is required, it performs batch review based on the session context of the large model within the current session to obtain risk content detection results. The session context includes reviewed content and currently pending stream content generated by the large model. Reviewed content includes the output content of the large model. Therefore, when a potentially risky dialogue is identified based on the real-time risk status, batch review is used to avoid the output of risky content, thus improving the accuracy of risk content detection and ultimately ensuring the security of human-computer interaction. Furthermore, when the real-time risk status indicates that streaming review is required, it performs streaming review based on the currently pending stream content generated by the large model to obtain risk content detection results. Therefore, when the real-time risk status indicates that the current dialogue is safe, streaming review is used to avoid full review, thereby improving the efficiency and cost of risk content detection. Streaming review also ensures the timeliness of risk content detection, ultimately enhancing the user experience. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the risk content detection method provided by the present invention.

[0022] Figure 2 This is the second flowchart of the risk content detection method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the risk content detection device provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Currently, while streaming segmented review is fast and has low latency, it suffers from a serious lack of context. Streaming review focuses on a limited amount of content, making it difficult to identify risks that require contextual analysis (such as semantic combination risks across segments or implicit risks in specific contexts). This can easily lead to missed security risks, meaning that the detection accuracy of risky content is not high.

[0027] To address the shortcomings of the aforementioned solutions, this invention proposes a method of recombining all accumulated historical output content (including user input) for a full-scale inspection when generating each new segment. However, while this approach can cover the context and ensure the accuracy of content security detection, the computational load for review increases linearly or even exponentially with the length of the generated content, resulting in extremely high computing costs. More seriously, the lengthy full-scale inspection significantly increases the latency of streaming output, causing lag on the user's end and severely disrupting the smooth experience of streaming interaction, thus impacting user experience. Furthermore, it can lead to the problem of untimely risk detection.

[0028] Therefore, how to effectively solve the problem of missed judgments due to lack of context while ensuring a low-latency experience for streaming output of large models, and avoid the high computing power consumption and high latency caused by full-scale review, is a technical problem that urgently needs to be solved.

[0029] To address the above problems, the present invention proposes the following embodiments. The following is a detailed explanation... Figures 1-2 The present invention describes a method for detecting risky content.

[0030] Figure 1 This is one of the flowcharts illustrating the risky content detection method provided by the present invention, such as... Figure 1 As shown, the risk content detection method includes the following steps 110, 120 and 130.

[0031] Step 110: Determine the real-time risk posture of the large model in the current session.

[0032] Here, the large model refers to the one that requires risk content detection. This means that the output content (generated content) of the large model needs to be checked for risk content to ensure that only content that passes the review process is output to the user (frontend). Furthermore, this large model is based on a Next Token Prediction architecture, outputting content in a streaming manner, word by word or segment by segment.

[0033] Here, the current session refers to one or more rounds of interaction between the user and the large model. During this process, the user inputs a Prompt, and the large model continuously generates Responses. It should be noted that only the real-time risk situation of the current session is considered; other sessions are not taken into account.

[0034] The real-time risk situation is used to characterize the output risk of the large model. This real-time risk situation differs from a simple risk / safety binary judgment; it is a comprehensive indicator used to dynamically characterize the risk tendency, risk level, or risk accumulation rate of the current output content of the large model. It can reflect not only the risk coefficient at the current instant but also the trend of risk change. This real-time risk situation can be a real-time risk coefficient (such as a real-time risk value) or it can be the rate at which the real-time risk coefficient increases.

[0035] In one embodiment, the real-time risk situation includes risk situations corresponding to multiple classification scenarios.

[0036] In one specific embodiment, when the large model generates the streaming content to be reviewed, the real-time risk status of the large model in the current session is determined.

[0037] It should be understood that by calculating the real-time risk situation, a risk early warning mechanism has been established. This makes the review process no longer passively waiting for obvious violations to appear before blocking, but rather able to perceive potential risks in advance based on the signs (situation) of the risk, providing a basis for decision-making on whether to intervene in a deeper review. This achieves an upgrade from static detection to dynamic perception, thus combining the real-time nature of streaming review with the accuracy of batch review, thereby improving the accuracy of risk content detection.

[0038] Step 120: If it is determined that batch review is required based on the real-time risk situation, batch review is performed based on the large model in the session context of the current session to obtain risk content detection results.

[0039] It should be noted that triggering conditions are preset, such as setting a risk status threshold. When the real-time risk status exceeds this threshold (for example, real-time risk coefficient > 0.8, or risk growth rate > 0.2), the current situation is determined to be in a high-risk tendency state. Simple streaming review may not be able to accurately judge or intercept this situation, so batch review is required.

[0040] Here, batch review refers to the process of taking multiple text segments or a long complete text segment (i.e., the conversation context) as input and making a one-time overall judgment.

[0041] The session context includes both reviewed content and currently pending review streaming content generated by the large model. The reviewed content includes the output content of the large model. This session context refers to background information used to assist in determining whether the current streaming content poses a risk.

[0042] In one embodiment, the reviewed content includes all reviewed content from all conversations in the current session, thereby improving the accuracy of detecting risky content. Of course, the reviewed content may also include only the reviewed content from the current conversation in the current session.

[0043] Here, the currently pending streaming content generated by the large model refers to the latest content fragment generated by the large model that has not yet been finalized for approval; that is, streaming content that has been generated but not yet approved. This streaming content corresponds to a token or a small block of tokens.

[0044] Here, "output content" refers to the content that is located before the current stream content to be reviewed in the streaming content sequence generated by the large model, and has been determined to be safe and output to the user (i.e., historical output).

[0045] In one specific embodiment, when the real-time risk situation is greater than the risk situation threshold, the risk content detection results are obtained by performing batch review based on the session context of the current session using a large model.

[0046] In one specific embodiment, the reviewed content (as above) is concatenated with the currently pending streaming content (as below), and the risk content detection results are obtained by batch reviewing the concatenated content.

[0047] In one specific embodiment, to avoid excessively long contexts affecting efficiency, the reviewed content does not necessarily have to be the entire history, but rather key fragments (i.e., the filtered session context) selected through scenario classification or semantic relevance. For example, if the current risk situation indicates political risks, only reviewed content fragments related to political tags will be recalled from the history.

[0048] In one specific embodiment, the session context is input into a batch review model to obtain the risk content detection results output by the batch review model. This batch review model can be trained based on sample session contexts and can be a large model or a small / medium-sized model. The batch review model comprehensively analyzes the preceding context and the generation tendency of the following text to output the final risk content detection results (such as safe, unsafe, questionable, approved, disapproved, etc.).

[0049] It should be understood that by introducing session context, batch review can identify subtle attacks that may seem harmless when viewed individually, but reveal malicious intent when combined with the preceding text (for example, a user asks how to make XX dangerous goods, and the model answers with specific chemical steps; these steps may seem harmless on their own, but combined with the preceding text, they constitute a tutorial on making dangerous goods). Based on this, risky content can be avoided from being missed, thereby improving the accuracy of risky content detection.

[0050] Step 130: If it is determined that streaming review is required based on the real-time risk situation, streaming review is performed on the current streaming content to be reviewed generated by the large model to obtain the risk content detection result.

[0051] It should be noted that when the real-time risk situation does not reach the preset risk situation threshold (e.g., the risk coefficient is low and stable), the current environment is determined to be relatively safe, and the default streaming review mode is maintained.

[0052] Here, streaming auditing refers to the real-time detection of every token or tiny token block generated by a large model, which usually requires extremely low detection latency (e.g., milliseconds) in order to avoid blocking the smoothness of streaming output.

[0053] In one specific embodiment, when the real-time risk situation is less than or equal to the risk situation threshold, the current streaming content to be reviewed, generated based on the large model, is used to perform streaming review to obtain the risk content detection result.

[0054] In one specific embodiment, streaming content is input into a streaming moderation model to obtain the risk content detection results output by the streaming moderation model. This streaming moderation model can be trained based on sample streaming content and can be a large model or a medium-sized model. This streaming moderation model can be the same model as the batch moderation model, or it can be a different model.

[0055] In one embodiment, the real-time risk situation includes risk situations corresponding to multiple classification scenarios; when the risk situations corresponding to multiple classification scenarios are all less than the risk situation threshold, the current streaming content to be reviewed is generated based on the large model, and streaming review is performed to obtain the risk content detection result.

[0056] It should be noted that the normal mode can be a streaming review mode, and the switch should be made when it is necessary to switch to the batch review mode.

[0057] It should be understood that in the vast majority of normal dialogue interactions (low-risk scenarios), the system maintains a streaming mode. Because the streaming mode is small, fast, and does not require loading historical context, it can minimize first-character latency and generation stuttering, while significantly saving computing resources, thereby ensuring user experience and reducing costs.

[0058] It should be understood that the embodiments of the present invention resolve the contradiction between accuracy and real-time performance. By introducing only the necessary data (session context) when necessary, it breaks the rigid thinking in the prior art that all contexts must be fully reviewed, and achieves adaptive adjustment of review granularity and review depth.

[0059] It should be understood that this embodiment of the invention introduces a real-time risk situation as a switching mechanism, constructing a dual-modal detection architecture of streaming review and batch review. When the risk situation is low, streaming review is used to ensure high real-time performance and low latency of large model output, maintaining user experience; when the real-time risk situation increases, it automatically switches to batch review, introducing context information to solve the problem of missed risk detection caused by the lack of context in streaming segmented review, thus balancing real-time performance and accuracy. Moreover, unlike the prior art that performs full historical detection for every output, this embodiment of the invention only loads historical context when needed, greatly reducing average computing power consumption and review costs.

[0060] The risk content detection method provided in this invention determines the real-time risk status of a large model in the current session. When batch review is required based on the real-time risk status, batch review is performed based on the session context of the large model in the current session to obtain risk content detection results. The session context includes reviewed content and currently pending stream content generated by the large model. Reviewed content includes the output content of the large model. Therefore, when a potentially risky dialogue is determined based on the real-time risk status, batch review is used to avoid the output of risky content, thus improving the accuracy of risk content detection and ultimately ensuring the security of human-computer interaction. Furthermore, when streaming review is required based on the real-time risk status, streaming review is performed based on the currently pending stream content generated by the large model to obtain risk content detection results. Therefore, when a safe dialogue is determined based on the real-time risk status, streaming review is used to avoid full review, thereby improving the efficiency and cost of risk content detection. Streaming review also ensures the timeliness of risk content detection, ultimately enhancing the user experience.

[0061] Based on any of the above embodiments, in this method, the real-time risk situation includes risk situations corresponding to multiple classification scenarios; step 120 includes: If, among the risk situations corresponding to at least one target classification scenario, the risk situation is greater than the risk situation threshold, then each content fragment associated with the at least one target classification scenario is determined from the reviewed content. Remove all content except the aforementioned content fragments from the reviewed content to obtain the filtered session context; Based on the filtered session context, batch review is performed to obtain risk content detection results.

[0062] Here, "categorization scenario" refers to different categories based on the content's security or business attributes, such as harmful content involving politics, advertisements, or prohibited items. Based on this, each category scenario corresponds to an independent risk detection dimension. For example, for the same streamed content, the risk coefficient in a politically sensitive scenario might be 0.85 (high risk), while the risk coefficient in an advertising scenario might be only 0.01 (low risk).

[0063] Here, the target classification scenario refers to the specific scenario category that triggered the risk warning at the current moment, that is, the scenario in which the real-time risk situation exceeds the set safety red line.

[0064] Here, the risk situation threshold is preset. This risk situation threshold includes a risk coefficient threshold or a risk coefficient increase rate threshold.

[0065] Here, the category scene label of the content fragment is the same as one of the target category scenes.

[0066] In one embodiment, a content fragment refers to the smallest semantic or structural unit within the reviewed content. For example, a content fragment can be a complete sentence, or it can be a semantic block divided based on semantic integrity.

[0067] For example, the risk profile of the current streaming content in each category scenario is first obtained; then, it is determined whether the risk profile of one or more categories scenario exceeds a preset risk profile threshold; if so, these categories scenario are identified as target categories scenario. For example, if the risk coefficients of both the political and biased categories scenario exceed the threshold, then both categories scenario are target categories scenario; next, the historical reviewed content is traversed. Since the reviewed content has been tagged with scenario tags during generation or subsequent processing, all content fragments marked as belonging to the target categories scenario are retrieved and extracted based on the tag index.

[0068] It should be understood that batch review based on the simplified session context significantly improves review speed because the length of the input data is significantly smaller than the full historical data, thereby increasing review efficiency and reducing review costs.

[0069] The risk content detection method provided in this invention achieves an on-demand recall mechanism by loading only the content fragments most relevant to the current risk, rather than loading the entire content. This significantly reduces the amount of input data for batch review, thereby greatly reducing computing power consumption and significantly shortening the review time. In turn, while ensuring the depth of review, it minimizes the impact on user experience (delay). That is, it improves the detection efficiency of risk content, reduces the detection cost of risk content, and ultimately improves the user experience.

[0070] Based on any of the above embodiments, in this method, determining the content fragments associated with the at least one target classification scenario from the reviewed content includes: The reviewed content is asynchronously categorized according to scenario; The content segments were identified from the categorized and reviewed content.

[0071] It should be noted that asynchronous means that this processing step is executed in parallel or decoupled from the main process of generating streaming content from the large model and outputting it to the user. The execution time of this step is not included in the main process's time consumption, and the execution status of this step (such as calculating or completing the calculation) will not block or pause the normal output of content in the main process.

[0072] Specifically, classification algorithms or models are used to classify the approved content that has been reviewed and output, and to determine one or more category scenarios to which it belongs (such as politics, entertainment, technology, sensitive topics, etc.).

[0073] For example, at the system architecture level, a backend classification service or bypass processing module is set up independently of the main review process. When a segment of streaming content generated by the large model passes streaming review and is output to the user as reviewed content, it is placed in a message queue or buffer. The backend classification service reads the reviewed content from the queue and calls the classification model to perform scene identification. After classification, the content segment and its corresponding scene tags are stored in an indexed historical record database for subsequent retrieval. When subsequent streaming content triggers a high-risk situation requiring batch review, a query request is initiated. At this time, since the historically reviewed content has already undergone preprocessing and classification tagging, only a tag-based quick retrieval based on the currently determined target classification scene is needed to directly extract the corresponding content segments from the historical record database.

[0074] It should be understood that because the categorization process is executed asynchronously, it does not need to be completed instantly when the content is output to the user. Even if the categorization takes a long time, it will not cause any stuttering or delay in the content seen by the user. This ensures that under normal interaction (streaming review approval), users can always have an extremely smooth response experience, that is, it does not affect the review delay.

[0075] It should be understood that because the tedious classification calculations have already been distributed and completed asynchronously during previous idle time, when a recall is actually needed, an extremely fast table lookup or index retrieval operation is performed, rather than real-time classification of all historical records. This drastically compresses the overall response time from triggering a risk to completing a batch review, thereby improving review efficiency, i.e., improving the efficiency of detecting risky content.

[0076] The risk content detection method provided in this invention asynchronously classifies reviewed content according to scenarios, thereby quickly identifying each content fragment from the classified reviewed content. This ensures that no additional delay is added when no risk is triggered, while when a risk is triggered, the required context information can be quickly obtained with extremely low time complexity, maximizing review efficiency, thus improving the detection efficiency of risk content and ultimately enhancing the user experience.

[0077] Based on any of the above embodiments, in this method, different classification scenarios correspond to different risk status thresholds, and the risk status corresponding to the target classification scenario is greater than the risk status threshold corresponding to the target classification scenario.

[0078] It should be noted that instead of using a globally uniform threshold, a configuration table or strategy set is maintained to specify an independent risk status threshold for each specific category scenario (such as political, advertising, etc.).

[0079] For example, in politically sensitive scenarios, because such risks are absolute red lines with extremely low tolerance, a lower risk threshold is set. This means that as soon as streaming content shows even a slight political inclination, it will immediately switch to batch review mode. In advertising scenarios, because such risks usually do not involve legal boundaries and have relatively high tolerance, a higher risk threshold is set. This means that batch review will only be triggered when streaming content exhibits very obvious advertising characteristics, thereby reducing disruption to users' normal communication.

[0080] The risk content detection method provided in this invention can accurately adapt to the risk tolerance of different business scenarios by configuring different risk status thresholds for different classification scenarios. This avoids excessive false blocking due to excessively low global thresholds, as well as security vulnerabilities due to excessively high global thresholds. In other words, it meets the differentiated needs of different scenarios, thereby improving the accuracy of risk content detection.

[0081] Based on any of the above embodiments, in this method, step 110 includes: The reviewed content is input into the streaming risk detection model to obtain the real-time risk coefficient output by the streaming risk detection model; The real-time risk situation is determined based on the real-time risk coefficient output by the streaming risk detection model.

[0082] The streaming risk detection model is trained based on the content of the reviewed samples. This streaming risk detection model refers to a machine learning or deep learning model specifically designed for processing streaming data with extremely fast response times.

[0083] Here, the real-time risk coefficient refers to the probability value or confidence score (e.g., a floating-point number between 0 and 1) that represents the risk of the reviewed content as output by the model.

[0084] It should be noted that as the large model continuously generates new streaming content, the reviewed content can be updated in real time and continuously fed into the streaming risk detection model as input.

[0085] In one embodiment, when the streaming content is the first content generated by the large model in the current session, the real-time risk posture is the real-time risk coefficient. When the streaming content is the Nth time the large model generates content in the current session, the real-time risk posture is the rate of increase of the real-time risk coefficient, where N is a positive integer greater than 1, and the rate of increase of the real-time risk coefficient is determined based on the difference between the real-time risk coefficient of the Nth time and the real-time risk coefficient of the (N-1)th time.

[0086] Furthermore, the reviewed content and the currently pending streaming content are input into the streaming risk detection model to obtain the real-time risk coefficient output by the streaming risk detection model.

[0087] The risk content detection method provided in this invention uses reviewed content as input, and the streaming risk detection model can obtain certain local contextual information, thereby better distinguishing semantics, improving the detection accuracy of real-time risk situation, reducing the frequency of falsely triggering batch review mode, thereby improving the detection efficiency of risk content, and ultimately enhancing the user experience.

[0088] Based on any of the above embodiments, in this method, the session context further includes the unoutput content generated by the large model; the unoutput content is content generated after the streaming content.

[0089] Here, "unoutputted content" refers to content that has been generated by the large model but is intercepted in the internal buffer and has not yet been sent to the user's terminal for display. On the timeline, it appears after the currently pending streaming content.

[0090] Here, the session context includes reviewed content, currently pending streaming content, and unoutputted content.

[0091] For example, when a batch review mode is triggered based on a real-time risk situation, not only is the output of the current streaming content paused, but the large model is also controlled to continue inferring and generating subsequent content. The unoutput content corresponding to the token sequence generated by the large model is temporarily written to an internal cache instead of being pushed to the front-end interface, thereby avoiding the exposure of more risky content through caching.

[0092] In one embodiment, unoutput content is continuously cached until the length of the unoutput content reaches a preset number of tokens (e.g., continue generating and caching 50 tokens). This ensures that a sufficiently long amount of subsequent semantics can be seen.

[0093] In another embodiment, unoutput content is continuously cached until an end-of-line symbol or a complete period / question mark is detected. This ensures that a complete semantic unit (such as a sentence) is seen.

[0094] The risk content detection method provided in this invention introduces unoutput content, enabling batch review to move beyond relying solely on the preceding text and the current word. It allows for the acquisition of subsequent textual information, thus determining the complete semantic flow and effectively resolving misinterpretations or misjudgments caused by incomplete sentences. This significantly improves the robustness of the review process, further enhancing the accuracy of risk content detection. Furthermore, it caches unoutput content, caching subsequently generated content as unoutput content during the review process. This internally mitigates the risk; if ultimately determined to be a violation, this cached content is directly discarded, ensuring the user cannot see these potentially harmful fragments. This avoids the unpleasant experience of outputting and then retracting content, completely eliminating any exposure of risk content on the user's end, thereby improving the user experience.

[0095] Based on any of the above embodiments, in this method, when the streaming content is the content first generated by the large model in the current session, the real-time risk situation is the real-time risk coefficient; when the streaming content is the content generated by the large model for the Nth time in the current session, the real-time risk situation is the real-time risk coefficient increase rate; N is a positive integer greater than 1, and the real-time risk coefficient increase rate is determined based on the difference between the real-time risk coefficient of the Nth time and the real-time risk coefficient of the (N-1)th time.

[0096] Here, the first generated content refers to the content corresponding to the first token or the first streaming shard generated by the large model for the user's prompt in the current session. At this time, there is no previous historical state, that is, no output content.

[0097] Here, the real-time risk coefficient is used to characterize the probability of the existence of risk or the confidence score (e.g., a floating-point number between 0 and 1).

[0098] Here, the rate of increase of the real-time risk coefficient refers to the slope or growth gradient of the risk coefficient within a unit of generation (such as each token).

[0099] In one specific embodiment, the difference between the real-time risk coefficient of the Nth time and the real-time risk coefficient of the (N-1)th time is determined, as well as the difference in the number of tokens between the content generated by the large model in the Nth time in the current session and the content generated by the large model in the (N-1)th time in the current session; based on the ratio of the difference to the difference in the number of tokens, the rate of increase of the real-time risk coefficient is determined.

[0100] The risk content detection method provided in this invention uses the real-time risk coefficient increase rate as a risk status indicator for subsequent content. This allows for the keen detection of abrupt risk changes. At the moment the large model generates risk content, the real-time risk coefficient typically undergoes a dramatic jump. Compared to waiting for the risk coefficient to slowly accumulate to an absolute high value (e.g., 0.9), judgment based on the growth rate can identify anomalies earlier, allowing for timely intervention or cut-off before the risk content is fully formed. This improves the accuracy of risk content detection, prevents missed risk content, and ultimately enhances user experience. Furthermore, separating the first character judgment from subsequent judgments aligns with the objective laws of streaming generation. The first character lacks historical reference and can only be considered in terms of absolute value; subsequent characters have historical reference, and the speed better reflects the generation trend. This phased strategy improves the scientific rigor and rationality of the detection logic.

[0101] Based on any of the above embodiments, in this method, the reviewed content further includes the input content of the large model. The reviewed content includes both the input and output content of the large model.

[0102] Here, "input content" refers to the instructions, questions, or dialogue text that the user sends to the large model in the current session. Before large model inference, this input content has usually passed risk checks on the user side and is therefore considered safe content that has been reviewed.

[0103] In one specific embodiment, the input content of the large model, the output content of the large model, and the currently pending streaming content are concatenated according to the dialogue rounds, and batch review is performed based on the concatenated complete sequence.

[0104] It should be understood that the output of a large model is often a response to user input, and the two are strongly logically related. By including the input content in the review scope, batch review can understand the intent and context of the current output, thereby making more accurate detection of risky content, i.e., improving the accuracy of risky content detection. Furthermore, embodiments of this invention help detect whether a large model follows user instructions with malicious intent; for example, if a user induces the large model to output sensitive content through a carefully designed prompt, the input prompt must be considered to determine whether the output violates regulations.

[0105] The risk content detection method provided in this embodiment of the invention includes the input content of the large model in the reviewed content, thereby ensuring more complete contextual information and improving the accuracy of batch review, that is, improving the accuracy of risk content detection.

[0106] Based on any of the above embodiments Figure 2 This is the second flowchart of the risk content detection method provided by the present invention, as shown below. Figure 2 As shown, after obtaining the risk content detection result, the method further includes: Step 140: If the streaming content is deemed to have passed review based on the risk content detection results, output the streaming content.

[0107] It should be noted that passing the streaming content review means that the risk content detection result is safe, compliant, or the risk score is below the interception threshold. Outputting streaming content means sending the streaming content to the user's terminal or front-end for display.

[0108] Furthermore, streaming content will be identified as content that has already been reviewed.

[0109] For example, when the streaming or batch review determines that the current streaming content is safe, the lock on the streaming content is released, the output interface is called, and the streaming content (and any subsequent cached content) is pushed to the client in sequence. At the same time, the streaming content is marked as reviewed content and stored in the history to support the context construction of subsequent rounds.

[0110] Step 150: If the streaming content fails the review based on the risk content detection results, the output of the streaming content is suspended.

[0111] It should be noted that failing the streaming content review means that the content is detected as unsafe, violates regulations, or has a risk score exceeding the blocking threshold. Suspending the output of streaming content means terminating the sending of current and subsequent content to the user's terminal.

[0112] For example, when the risk content detection results determine that the streaming content poses a risk, an interception action is immediately executed. Specifically, the currently pending streaming content and all generated but unoutputted content (cached content) are discarded to prevent them from being leaked to users.

[0113] Furthermore, after pausing output, subsequent actions are performed according to preset security policies, such as refusing to answer or answering on behalf of the user. Refusing to answer means directly terminating the connection or stopping the generation of the large model and returning a generic error code to the user. Answering on behalf of the user means calling a preset security response template (e.g., "I'm sorry, I cannot answer this question because it involves sensitive information") and sending this security message as new output to the user, replacing the original inappropriate content; this provides a more user-friendly interactive experience.

[0114] It should be understood that by explicitly outputting or pausing instructions, the review results are transformed into actual control actions. This ensures that all content that ultimately reaches the user is verified and safe, completely blocking the path of harmful information dissemination.

[0115] The risk content detection method provided in this invention suspends the output of streaming content when the streaming content fails the review based on the risk content detection result, thereby ensuring that all content that finally reaches the user is verified safe content, thus improving the user experience.

[0116] Based on any of the above embodiments, in this method, the session context further includes the unoutput content generated by the large model; the unoutput content is content generated after the streaming content.

[0117] Accordingly, the output of the streaming content includes: Output the streaming content and the unoutput content; The streaming content and the unoutput content are identified as reviewed content.

[0118] In one specific embodiment, a series of text sequences accumulated in the buffer (i.e., streaming content and unoutput content) are sent to the user all at once or at an accelerated pace.

[0119] For example, in batch review mode, since the review takes time, the large model may have generated dozens of unoutputted tokens in the background and accumulated in the buffer. When the batch review result is passed, it means that the entire context (including streaming content and unoutputted content) is safe. Therefore, it is no longer sent word by word, but the streaming content and unoutputted content are merged and sent to the front end as a whole data packet; or it is pushed continuously at a very high frequency so that the user can see the completed text instantly.

[0120] It should be noted that the next streaming review will begin with the first new token after no content has been output.

[0121] It should be understood that since batch review inevitably introduces brief pauses, users will feel that the accumulated unreleased content is being displayed all at once, catching up instantly. This fast-forward effect psychologically compensates for the previous waiting time, keeping the overall interactive experience smooth and thus improving the user experience.

[0122] It should be understood that timely updates to the status of reviewed content ensure that the next round of review is based on the latest and most comprehensive historical data, thus ensuring the accuracy of risk content detection.

[0123] The risk content detection method provided in this embodiment of the invention takes into account that batch review has already covered the unoutput content, so it directly marks it as reviewed when releasing it, avoiding the need to go back and perform streaming review on these contents later, saving computing resources, improving the detection efficiency of risk content, and reducing the detection cost of risk content.

[0124] The risk content detection device provided by the present invention is described below. The risk content detection device described below and the risk content detection method described above can be referred to in correspondence.

[0125] Figure 3 This is a schematic diagram of the risky content detection device provided by the present invention, as shown below. Figure 3 As shown, the risk content detection device includes: a risk determination module 310, a batch review module 320, and a streaming review module 330.

[0126] The risk determination module 310 is used to determine the real-time risk status of the large model in the current session; the real-time risk status is used to characterize the output risk of the large model.

[0127] The batch review module 320 is used to perform batch review to obtain risk content detection results based on the session context of the current session and the large model when it is determined that batch review is required based on the real-time risk situation. The session context includes the reviewed content and the currently pending streaming content generated by the large model. The reviewed content includes the output content of the large model.

[0128] The streaming review module 330 is used to perform streaming review based on the current streaming content to be reviewed generated by the large model when it is determined that streaming review is required based on the real-time risk situation to obtain risk content detection results.

[0129] The risk content detection device provided in this invention determines the real-time risk status of a large model in the current session. When the real-time risk status indicates that batch review is required, it performs batch review based on the session context of the large model in the current session to obtain risk content detection results. The session context includes reviewed content and currently pending stream content generated by the large model. Reviewed content includes the output content of the large model. Therefore, when a potentially risky dialogue is identified based on the real-time risk status, batch review is used to avoid the output of risky content, thus improving the accuracy of risk content detection and ultimately ensuring the security of human-computer interaction. Furthermore, when the real-time risk status indicates that streaming review is required, it performs streaming review based on the currently pending stream content generated by the large model to obtain risk content detection results. Therefore, when the real-time risk status indicates that the current dialogue is safe, streaming review is used to avoid full review, thereby improving the efficiency and cost of risk content detection. Streaming review also ensures the timeliness of risk content detection, ultimately enhancing the user experience.

[0130] Based on any of the above embodiments, the real-time risk situation includes risk situations corresponding to multiple classification scenarios; the batch review module 320 is specifically used for: If, among the risk situations corresponding to at least one target classification scenario, the risk situation is greater than the risk situation threshold, then each content fragment associated with the at least one target classification scenario is determined from the reviewed content. Remove all content except the aforementioned content fragments from the reviewed content to obtain the filtered session context; Based on the filtered session context, batch review is performed to obtain risk content detection results.

[0131] Based on any of the above embodiments, in this device, determining the content segments associated with the at least one target classification scenario from the reviewed content includes: The reviewed content is asynchronously categorized according to scenario; The content segments were identified from the categorized and reviewed content.

[0132] Based on any of the above embodiments, in this device, different risk status thresholds correspond to different classification scenarios, and the risk status corresponding to the target classification scenario is greater than the risk status threshold corresponding to the target classification scenario.

[0133] Based on any of the above embodiments, in this device, the risk determination module 310 is specifically used for: The reviewed content is input into the streaming risk detection model to obtain the real-time risk coefficient output by the streaming risk detection model; The real-time risk situation is determined based on the real-time risk coefficient output by the streaming risk detection model. The streaming risk detection model is trained based on the reviewed content of the sample.

[0134] Based on any of the above embodiments, in this device, the session context further includes the unoutput content generated by the large model; the unoutput content is content generated after the streaming content.

[0135] Based on any of the above embodiments, when the streaming content is the content first generated by the large model in the current session, the real-time risk situation is the real-time risk coefficient; when the streaming content is the content generated by the large model for the Nth time in the current session, the real-time risk situation is the real-time risk coefficient increase rate; N is a positive integer greater than 1, and the real-time risk coefficient increase rate is determined based on the difference between the real-time risk coefficient of the Nth time and the real-time risk coefficient of the (N-1)th time.

[0136] Based on any of the above embodiments, in this device, the reviewed content also includes the input content of the large model.

[0137] Based on any of the above embodiments, the device further includes: The content output module is used to output the streaming content when the streaming content is determined to have passed the review based on the risk content detection results. The output pause module is used to pause the output of the streaming content if the streaming content fails the review based on the risk content detection results.

[0138] Based on any of the above embodiments, in this device, the session context further includes the unoutput content generated by the large model; the unoutput content is content generated after the streaming content; the content output module is specifically used for: Output the streaming content and the unoutput content; The streaming content and the unoutput content are identified as reviewed content.

[0139] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a risk content detection method. This method includes: determining the real-time risk posture of a large model in the current session; the real-time risk posture is used to characterize the output risk of the large model; if it is determined based on the real-time risk posture that batch review is required, batch review is performed based on the session context of the large model in the current session to obtain risk content detection results; the session context includes reviewed content and currently unreviewed streaming content generated by the large model, the reviewed content including the output content of the large model; if it is determined based on the real-time risk posture that streaming review is required, streaming review is performed based on the currently unreviewed streaming content generated by the large model to obtain risk content detection results.

[0140] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the risk content detection method provided by the above methods. The method includes: determining the real-time risk status of a large model in the current session; the real-time risk status is used to characterize the output risk of the large model; if it is determined based on the real-time risk status that batch review is required, batch review is performed based on the session context of the large model in the current session to obtain risk content detection results; the session context includes reviewed content and currently unreviewed streaming content generated by the large model, the reviewed content including the output content of the large model; if it is determined based on the real-time risk status that streaming review is required, streaming review is performed based on the currently unreviewed streaming content generated by the large model to obtain risk content detection results.

[0142] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the risk content detection method provided by the above methods. The method includes: determining the real-time risk profile of a large model in the current session; the real-time risk profile characterizing the output risk of the large model; if, based on the real-time risk profile, it is determined that batch review is required, performing batch review based on the session context of the large model in the current session to obtain risk content detection results; the session context includes reviewed content and currently pending review streaming content generated by the large model, the reviewed content including the output content of the large model; if, based on the real-time risk profile, it is determined that streaming review is required, performing streaming review based on the currently pending review streaming content generated by the large model to obtain risk content detection results.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of detecting risky content, characterized by, include: Determine the real-time risk profile of the large model in the current session; The real-time risk situation is used to characterize the output risk of the large model; If, based on the real-time risk situation, it is determined that batch review is required, the risk content detection results are obtained by batch review based on the session context of the current session using the large model; the session context includes the reviewed content and the currently pending streaming content generated by the large model, and the reviewed content includes the output content of the large model; If, based on the real-time risk situation, it is determined that streaming review is required, streaming review is performed on the streaming content to be reviewed currently generated by the large model to obtain risk content detection results.

2. The risk content detection method of claim 1, wherein, The real-time risk situation includes risk situations corresponding to multiple classification scenarios; When it is determined that batch review is required based on the real-time risk situation, the batch review is performed based on the large model within the session context of the current session to obtain risk content detection results, including: If, among the risk situations corresponding to at least one target classification scenario, the risk situation is greater than the risk situation threshold, then each content fragment associated with the at least one target classification scenario is determined from the reviewed content. Remove all content except the aforementioned content fragments from the reviewed content to obtain the filtered session context; Based on the filtered session context, batch review is performed to obtain risk content detection results.

3. The risk content detection method of claim 2, wherein, The step of identifying content fragments associated with the at least one target classification scenario from the reviewed content includes: The reviewed content is asynchronously categorized according to scenario; The content segments were identified from the categorized and reviewed content.

4. The risk content detection method of claim 2, wherein, Different risk status thresholds correspond to different classification scenarios, and the risk status corresponding to the target classification scenario is greater than the risk status threshold corresponding to the target classification scenario.

5. The risk content detection method according to claim 1, characterized in that, Determining the real-time risk posture of the large model in the current session includes: The reviewed content is input into the streaming risk detection model to obtain the real-time risk coefficient output by the streaming risk detection model; The real-time risk situation is determined based on the real-time risk coefficient output by the streaming risk detection model. The streaming risk detection model is trained based on the reviewed content of the sample.

6. The risk content detection method according to any one of claims 1 to 5, characterized in that, The session context also includes the unoutput content generated by the large model; the unoutput content is the content generated after the streaming content.

7. The risk content detection method according to any one of claims 1 to 5, characterized in that, In the case that the streaming content is the content first generated by the large model in the current session, the real-time risk situation is the real-time risk coefficient; When the streaming content is the content generated by the large model for the Nth time in the current session, the real-time risk situation is the rate of increase of the real-time risk coefficient; N is a positive integer greater than 1, and the rate of increase of the real-time risk coefficient is determined based on the difference between the real-time risk coefficient of the Nth time and the real-time risk coefficient of the (N-1)th time.

8. The risk content detection method according to any one of claims 1 to 5, characterized in that, The reviewed content also includes the input content of the large model.

9. The risk content detection method according to any one of claims 1 to 5, characterized in that, After obtaining the risk content detection results, the method further includes: If the streaming content is deemed to have passed review based on the risk content detection results, the streaming content is output. If the streaming content fails the review based on the risk content detection results, the output of the streaming content will be suspended.

10. The risk content detection method according to claim 9, characterized in that, The session context also includes the unoutput content generated by the large model; The unoutput content refers to the content generated after the streaming content. The output of the streaming content includes: Output the streaming content and the unoutput content; The streaming content and the unoutput content are identified as reviewed content.

11. A risky content detection device, characterized in that, include: The risk assessment module is used to determine the real-time risk situation of the large model in the current session; The real-time risk situation is used to characterize the output risk of the large model; The batch review module is used to perform batch review and obtain risk content detection results based on the session context of the current session and the large model when it is determined that batch review is required based on the real-time risk situation. The session context includes the reviewed content and the streaming content to be reviewed generated by the large model. The reviewed content includes the output content of the large model. The streaming review module is used to perform streaming review on the currently pending streaming content generated by the large model when it is determined that streaming review is required based on the real-time risk situation, and obtain the risk content detection result.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the risk content detection method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the risk content detection method as described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the risk content detection method as described in any one of claims 1 to 10.

Images