Risk detection method, device, equipment, storage medium and product

Abstract

The application discloses a risk detection method and device, equipment, a storage medium and a product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a risk sample set, the risk sample set comprising a plurality of risk samples, the risk samples being used for guiding a large language model to generate risk content; screening a target sample from the risk sample set by using a risk detection large model, the risk type detection difficulty of the target sample being greater than the risk type detection difficulty of other samples in the risk sample set; acquiring a risk type labeled for the target sample; and training the risk detection large model based on the target sample and the risk type labeled for the target sample. The method can enhance the detection capability of the risk detection large model and improve the detection accuracy of risk problems.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a risk detection method, apparatus, device, storage medium, and product. Background Technology

[0002] Large language models, as leaders in the field of generative models, have demonstrated outstanding performance in many tasks such as natural language processing and dialogue generation. However, when running in open environments, they also expose some security vulnerabilities. For example, the models may generate risky content such as politically sensitive or socially ethically conflicting content under various attacks.

[0003] To effectively address this issue, a large-scale risk detection model has been introduced. Leveraging its powerful detection capabilities, it filters input questions into a large language model, proactively identifying and intercepting issues that may lead to the generation of risky content, and then taking corresponding preventative measures. However, current large-scale risk detection models frequently experience missed detections and false positives in practical applications. Therefore, improving the accuracy of large-scale risk detection models to enable them to more accurately detect risky issues has become a crucial and urgent problem to be solved.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a risk detection method, apparatus, equipment, storage medium, and product that can enhance the detection capability of large-scale risk detection models and improve the accuracy of risk problem detection.

[0006] To achieve the above objectives, this application proposes a risk detection method, the method comprising:

[0007] Obtain a risk sample set, which includes multiple risk samples, and the risk samples are used to guide the large language model to generate risk content.

[0008] Using a large-scale risk detection model, target samples are selected from the risk sample set, and the risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set.

[0009] Obtain the risk type labeled for the target sample;

[0010] The risk detection model is trained based on the target sample and the risk type labeled for the target sample.

[0011] Optionally, the step of selecting target samples from the risk sample set using a large-scale risk detection model includes:

[0012] The risk detection model is used to detect the risk type of each risk sample in the risk sample set, and the detection result corresponding to each risk sample is obtained.

[0013] Based on the detection results corresponding to each risk sample, the target sample is selected from the risk sample set.

[0014] Optionally, the detection results include the risk type to which each risk sample belongs and the probability that each risk sample belongs to the detected risk type;

[0015] The step of selecting the target sample from the risk sample set based on the detection results corresponding to each risk sample includes:

[0016] Based on the probability that each risk sample belongs to the detected risk type, risk samples with a probability lower than the reference probability are selected from the risk sample set.

[0017] The selected risk samples are identified as the target samples.

[0018] Optionally, the step of detecting the risk type of each risk sample in the risk sample set using the risk detection model to obtain the detection result corresponding to each risk sample includes:

[0019] Using multiple risk detection models, the risk type of each risk sample in the risk sample set is detected, and multiple detection results are obtained for each risk sample.

[0020] The step of selecting the target sample from the risk sample set based on the detection results corresponding to each risk sample includes:

[0021] Based on multiple detection results corresponding to each risk sample, risk samples with inconsistent detection results are selected from the risk sample set.

[0022] The selected risk samples are identified as the target samples.

[0023] Optionally, training the risk detection model based on the target sample and the risk type labeled for the target sample includes:

[0024] Multiple target samples are concatenated into a fixed-length training sequence, and the feature vector of each word in the training sequence is determined.

[0025] The risk detection big model determines the attention weights between each word and its associated word in the training sequence, and updates the feature vector of each word based on the determined attention weights, wherein the associated word of each word belongs to the same sample as each word and is located before each word in the sample.

[0026] The risk detection model is used to detect the risk type of each target sample in the training sequence based on the updated feature vectors of each word.

[0027] The risk detection model is trained based on the detection risk type and labeled risk type of each target sample.

[0028] Optionally, the step of determining the attention weights between each word and its associated word in the training sequence using the risk detection big model, and updating the feature vector of each word based on the determined attention weights, includes:

[0029] The risk detection big model is used to determine a similarity matrix, where the elements of the similarity matrix represent the feature similarity between lexical units in the training sequence.

[0030] Obtain a mask matrix, wherein the elements in the mask matrix represent the attention mask between words in the training sequence, wherein the attention mask between each word and its associated words is 0, and the attention mask between each word and other words besides its associated words is negative infinity;

[0031] The similarity matrix and the mask matrix are added together and converted into a probability distribution to obtain the attention matrix. The elements in the attention matrix represent the attention weights between word units in the training sequence, wherein the attention weight between each word unit and other word units other than the word unit associated with each word unit is 0.

[0032] Based on the attention matrix, update the feature vector of each word.

[0033] Optionally, obtaining the mask matrix includes:

[0034] Determine the position information of each target sample in the training sequence, wherein the position information includes the start position, end position, and position of each word element in each target sample;

[0035] The mask matrix is ​​generated based on the position information of each target sample in the training sequence.

[0036] Optionally, the method further includes:

[0037] Based on the detection results corresponding to each risk sample, false detection samples are selected from the risk sample set. The false detection samples are those that are incorrectly detected in terms of risk type.

[0038] The false detection samples, the detection risk types of the false detection samples, and the labeled risk types constitute the training samples;

[0039] The risk detection model is optimized based on the training samples.

[0040] Optionally, the method further includes:

[0041] Based on the detection results corresponding to each risk sample, false detection samples are selected from the risk sample set. The false detection samples are those that are incorrectly detected in terms of risk type.

[0042] Construct a new risk sample of the same type as the false detection sample;

[0043] The new risk sample is added to the risk sample set.

[0044] Optionally, after training the risk detection model based on the target sample and the risk types labeled for the target sample, the method further includes:

[0045] In response to a risk detection command, extract the target issue and the business type to which the target issue belongs from the risk detection command;

[0046] The risk detection model is used to detect the risk type of the target problem.

[0047] Based on the risk type and business type of the target problem, determine the target handling method for the target problem.

[0048] Optionally, determining the target handling method for the target problem based on the risk type and business type to which the target problem belongs includes:

[0049] Obtain strategy configuration information that matches the business type, wherein the strategy configuration information contains the problem handling method corresponding to each risk type;

[0050] Query the risk type to which the target problem belongs and the corresponding problem handling method from the strategy configuration information;

[0051] The problem-solving methods found in the query are determined as the target problem-solving methods.

[0052] Optionally, the problem-handling methods covered in the policy configuration information include at least two of the following:

[0053] The responses can be rejected, a response generated using a secure response model, or a response generated using a default response model. The secure response model is a response model that has been trained and optimized using secure data.

[0054] Optionally, obtaining the risk sample set includes:

[0055] Extract risk samples from the business log dataset of the question-and-answer system; or, extract risk samples from publicly available domain datasets used for building model content security; or, obtain manually constructed risk samples; or, obtain risk samples generated by attacking large models.

[0056] The risk sample set is constructed based on the acquired risk samples.

[0057] Furthermore, to achieve the above objectives, this application also proposes a risk detection device, the device comprising:

[0058] The set acquisition module is used to acquire a risk sample set, which includes multiple risk samples. These risk samples are used to guide the large language model to generate risk content.

[0059] The sample screening module is used to screen target samples from the risk sample set using a risk detection big data model. The risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set.

[0060] The type labeling module is used to obtain the risk type labeled for the target sample;

[0061] The model training module is used to train the risk detection big model based on the target sample and the risk type labeled for the target sample.

[0062] Optionally, the sample screening module includes:

[0063] The result acquisition unit is used to detect the risk type of each risk sample in the risk sample set through the risk detection big model, and obtain the detection result corresponding to each risk sample.

[0064] The sample screening unit is used to screen the target sample from the risk sample set based on the detection results corresponding to each risk sample.

[0065] Optionally, the detection results include the risk type to which each risk sample belongs and the probability that each risk sample belongs to the detected risk type;

[0066] The sample screening unit is used to screen risk samples whose corresponding probability is lower than the reference probability from the risk sample set based on the probability that each risk sample belongs to the detected risk type; and to determine the screened risk samples as the target samples.

[0067] Optionally, the result acquisition unit is used to detect the risk type of each risk sample in the risk sample set through multiple risk detection models, and obtain multiple detection results corresponding to each risk sample.

[0068] The sample screening unit is used to screen out multiple risk samples with inconsistent test results from the risk sample set based on multiple test results corresponding to each risk sample; and to determine the screened risk samples as the target samples.

[0069] Optionally, the model training module includes:

[0070] A vector determination unit is used to concatenate multiple target samples into a fixed-length training sequence and determine the feature vector of each word in the training sequence.

[0071] The weight determination unit is used to determine the attention weight between each word and its associated word in the training sequence through the risk detection big model, and update the feature vector of each word based on the determined attention weight, wherein the associated word of each word belongs to the same sample as each word and is located before each word in the sample.

[0072] The type detection unit is used to detect the risk type of each target sample in the training sequence based on the updated feature vector of each word using the risk detection big model.

[0073] The model training unit is used to train the risk detection model based on the detection risk type and labeled risk type of each target sample.

[0074] Optionally, the weight determination unit includes:

[0075] The first subunit is used to determine a similarity matrix through the risk detection big model, wherein the elements in the similarity matrix represent the feature similarity between lexical units in the training sequence;

[0076] The second subunit is used to obtain a mask matrix, wherein the elements in the mask matrix represent the attention mask between words in the training sequence, wherein the attention mask between each word and its associated words is 0, and the attention mask between each word and other words besides its associated words is negative infinity.

[0077] The third subunit is used to add the similarity matrix and the mask matrix and convert them into a probability distribution to obtain an attention matrix. The elements in the attention matrix represent the attention weights between word units in the training sequence, wherein the attention weight between each word unit and other word units other than the word units associated with each word unit is 0.

[0078] The fourth subunit is used to update the feature vector of each word based on the attention matrix.

[0079] Optionally, the second subunit is used to determine the position information of each target sample in the training sequence, the position information including the start position, end position and position of each word in each target sample; and to generate the mask matrix based on the position information of each target sample in the training sequence.

[0080] Optionally, the device further includes:

[0081] The preference optimization module is used to select false detection samples from the risk sample set based on the detection results corresponding to each risk sample. The false detection samples are samples with incorrect risk type detection. The false detection samples, the detected risk type of the false detection samples, and the labeled risk type constitute training samples. The preference optimization is performed on the risk detection model based on the training samples.

[0082] Optionally, the device further includes:

[0083] The sample update module is used to filter out false detection samples from the risk sample set based on the detection results corresponding to each risk sample, wherein the false detection samples are samples with incorrect risk type detection; construct a new risk sample with the same type as the false detection sample; and add the new risk sample to the risk sample set.

[0084] Optionally, the device further includes:

[0085] The instruction response module is used to respond to a risk detection instruction by extracting the target issue and the business type to which the target issue belongs from the risk detection instruction.

[0086] The type detection module is used to detect the risk type of the target problem using the risk detection model.

[0087] The method determination module is used to determine the target processing method for the target problem based on the risk type and the business type to which the target problem belongs.

[0088] Optionally, the method determination module is used to obtain strategy configuration information matching the business type, wherein the strategy configuration information includes problem handling methods corresponding to each risk type; query the problem handling method corresponding to the risk type to which the target problem belongs from the strategy configuration information; and determine the queried problem handling method as the target handling method for the target problem.

[0089] Optionally, the problem-handling methods covered in the policy configuration information include at least two of the following:

[0090] The responses can be rejected, a response generated using a secure response model, or a response generated using a default response model. The secure response model is a response model that has been trained and optimized using secure data.

[0091] Optionally, the set acquisition module is used to extract risk samples from the business log dataset of the question-answering system; or, extract risk samples from the publicly available domain dataset used for model content security construction; or, acquire manually constructed risk samples; or, acquire risk samples generated by attacking large models; and construct the risk sample set based on the acquired risk samples.

[0092] In addition, to achieve the above objectives, this application also proposes a risk detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the risk detection method as described above.

[0093] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the risk detection method described above.

[0094] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the risk detection method described above.

[0095] One or more technical solutions proposed in this application have at least the following technical effects:

[0096] The risk detection scheme provided in this application first obtains a risk sample set, which includes multiple risk samples used to guide a large language model in generating risk content, providing a data foundation for subsequent sample selection. Then, the large-scale risk detection model selects target samples from the risk sample set, where the risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set. In other words, the more challenging risk samples are selected from the risk sample set. Considering that false negatives and missed detections often concentrate on these challenging target samples, the risk types of these target samples are specifically labeled, and the large-scale risk detection model is trained based on these labeled target samples. This enhances the model's ability to identify complex risks, reduces false negatives and missed detections, and improves detection accuracy. Furthermore, compared to training all risk samples indiscriminately, targeted training by selecting target samples utilizes computational resources and time more efficiently. Focusing on more challenging target samples allows for faster improvement of model performance, avoiding excessive resource waste on simple samples. Attached Figure Description

[0097] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0098] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0099] Figure 1 This is a schematic diagram of an implementation environment for the risk detection method of this application;

[0100] Figure 2 This is a flowchart illustrating the first embodiment of the risk detection method of this application;

[0101] Figure 3 This is a flowchart illustrating the second embodiment of the risk detection method of this application;

[0102] Figure 4 This is a flowchart illustrating the third embodiment of the risk detection method of this application;

[0103] Figure 5 This is a flowchart illustrating the fourth embodiment of the risk detection method of this application;

[0104] Figure 6 This is a flowchart illustrating the fifth embodiment of the risk detection method of this application;

[0105] Figure 7 This is a flowchart illustrating the sixth embodiment of the risk detection method of this application;

[0106] Figure 8 A schematic diagram illustrating a problem-solving process provided for some embodiments of this application;

[0107] Figure 9 This is a schematic diagram of the module structure of the risk detection device according to an embodiment of this application;

[0108] Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the risk detection method in the embodiments of this application.

[0109] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0110] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0111] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0112] Figure 1 This is a schematic diagram illustrating an implementation environment provided by an embodiment of this disclosure. See also... Figure 1 The implementation environment includes a user terminal 101, a training terminal 102, and a server 103. The user terminal 101 and the training terminal 102 are connected to the server 103 via a wireless or wired network. For example, the user terminal 101 and the training terminal 102 can be a computer, mobile phone, tablet computer, or other terminal.

[0113] For example, a target application provided by server 103 is installed on user terminal 101. For example, the target application is either an application within the operating system of user terminal 101 or a target application provided by a third party. This target application has a question-and-answer function, capable of generating corresponding answers based on input user questions. For example, the target application could be a search application, a short video application, or a chat application. For example, server 103 is the backend server corresponding to this target application. Accordingly, server 103 could be a search application server, a short video application server, a chat application server, etc.

[0114] In this application, training terminal 102 is used to acquire a risk sample set, which includes multiple risk samples used to guide a large language model to generate risky content. Then, the risk detection model filters target samples from the risk sample set, where the risk type detection difficulty of the target sample is greater than that of other samples in the set. Next, the risk type labeled for the target sample is obtained, and the target sample and its labeled risk type are sent to the server. Server 103 receives the target sample and its labeled risk type. Based on the target sample and its labeled risk type, the risk detection model is trained. Afterward, server 103 can use the trained risk detection model to perform risk detection on various questions from user terminal 101, thereby filtering questions input to the response model and identifying and intercepting questions that may lead to the generation of risky content in advance. For normal questions, server 103 generates corresponding answers using the response model and returns the generated answers to user terminal 101.

[0115] Alternatively, the aforementioned risk detection scheme can also be performed independently by the training terminal 102 or the server 103. This application embodiment does not impose any limitations on this.

[0116] The solution provided in this application has a wide range of applications and can help improve the security and reliability of the output content of large language models in multiple fields. For example, in intelligent customer service scenarios, intelligent customer service systems may encounter various problems during communication with users. Some problems may contain sensitive information or lead the customer service system to generate inappropriate responses. Using this solution to train a large-scale risk detection model can perform risk detection on user-input questions, filtering out risky questions in advance, preventing the customer service system from generating inappropriate answers, and improving service quality and user experience. Another example is in education scenarios. When providing learning assistance to students, intelligent tutoring systems need to ensure the correctness and legality of the answers. Using the large-scale risk detection model provided in this application to detect students' questions can effectively prevent the generation of erroneous or harmful information, ensuring the quality of education.

[0117] Figure 2 This is a flowchart illustrating the first embodiment of the risk detection method of this application. (Refer to...) Figure 2 Taking the training terminal as the executing entity as an example, this risk detection method includes the following steps S10 to S40:

[0118] Step S10: Obtain a risk sample set, which includes multiple risk samples. These risk samples are used to guide the large language model to generate risk content.

[0119] The risk sample set is a compilation of risk samples. It is collected through various means and contains sample data of different forms and characteristics. A common feature of these sample data is that if they are fed as input to a large language model, they are highly likely to guide the model to generate risky content. As the foundational data source for subsequent selection of target samples and training of the risk detection model, it provides the model with a rich variety of risk scenarios and features, enabling the model to learn the manifestations of different types of risks.

[0120] A risk sample is a single element in a risk sample set; it is a data individual with specific characteristics and content. It is specifically designed or collected to test or trigger risky outputs from large language models. For example, in a content generation scenario, a risk sample might be a text prompt containing malicious intent.

[0121] Large Language Models (LLMs) are language processing models with a massive parameter scale built using deep learning techniques. Trained on large-scale text data, they learn the syntax, semantics, and functions of language, enabling them to perform various natural language processing tasks such as text generation, question answering, and translation. In this approach, the large language model is the primary source of potentially risky content, and the risk samples are designed specifically to target it. Examples of large language models include GPT-4 (Generative Pretrained Transformer 4) and Claude.

[0122] Optionally, obtaining a risk sample set includes: extracting risk samples from the business log dataset of the question-answering system; or extracting risk samples from publicly available domain datasets used for model content security construction; or obtaining manually constructed risk samples; or obtaining risk samples generated by attacking large models; and constructing a risk sample set based on the obtained risk samples.

[0123] The business log dataset of the question-and-answer system is a collection of various business-related data recorded during the system's operation. It includes detailed information about user interactions with the system, such as user questions and system responses. Since the question-and-answer system may encounter various user inputs, including those with malicious intent, these attack-related records constitute a potential source of risk samples. For example, malicious users may attempt to induce the question-and-answer system to output sensitive information or inappropriate content through specific questions; these interaction records can be extracted from the business log dataset as risk samples.

[0124] Domain-specific public datasets for building model content security are datasets publicly shared by research institutions or enterprises to improve model content security. These datasets focus on various content security risks that models may face, collecting and organizing a large amount of risk-related sample data, including various types of risk samples. For example, domain-specific public datasets include SafetyBench, Safety-Prompts, and SORRY-Bench (Safety and Openness Reliability Risk Yield-Benchmark).

[0125] Artificially constructed risk samples are risk samples designed and created by professionals based on actual risk scenarios or model vulnerabilities. For example, security experts may construct a series of samples that bypass model security filtering mechanisms by cleverly configuring instructions, in order to test the model's security.

[0126] Risk samples generated by attack models refer to risk samples generated by attack models through designed attack methods, such as prompting manipulation, role-playing, and command injection. An attack model is a large language model that has been trained specifically to generate risk samples.

[0127] In this embodiment, risk samples are obtained from three different sources: the business log dataset of the question-answering system, the publicly available domain dataset, and manually constructed samples, as well as large attack models. This significantly enriches the content of the risk sample set. The business log dataset reflects real-world attack scenarios; the publicly available domain dataset gathers industry research findings on model security risks; manually constructed samples can specifically simulate various potential attack scenarios; and the attack models generate a wide variety of attack samples, covering a broad range of risk scenarios. This fusion of multi-source data makes the risk sample set more comprehensive in terms of the types of risks it covers.

[0128] Step S20: Using a large-scale risk detection model, target samples are selected from the risk sample set. The risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set.

[0129] The risk detection big data model is an optimized large language model whose main task is to assess and detect risks in input content. By learning from a large number of risk samples and their corresponding risk types, the model can identify whether there are potential risks in the input content. For example, the risk detection big data model is a neural network model based on the Transformer architecture, such as the GPT model. It can extract and analyze features from the input text to determine whether it contains sensitive information, violates ethical standards, etc., thereby outputting the risk type of the text.

[0130] Target samples are a special subset of samples selected from the risk sample set. These samples represent a relatively high level of risk type detection difficulty, typically exhibiting more subtle and complex risk characteristics. These target samples serve as key training data, enhancing the risk detection model's ability to identify complex risks. By specifically analyzing and learning from these target samples, the model can better handle various complex risk situations, improving overall detection accuracy and generalization ability.

[0131] Risk types are the criteria for classifying risks. Based on different application scenarios and risk characteristics, risks are divided into different categories to more accurately identify and manage them. For example, risk types include political sensitivities, violence and pornography, misinformation, and ethical conflicts.

[0132] For example, a large-scale risk detection model can select target samples from a set of risk samples based on the text complexity of each risk sample, such as text length, lexical richness, and grammatical complexity. Higher text complexity means a more difficult-to-determine risk type for the sample. Alternatively, the large-scale risk detection model can select target samples from the set of risk samples based on the prediction probability of predicting the risk type for each risk sample. Lower prediction probabilities indicate a more difficult-to-determine risk type for the sample, and thus a greater difficulty in detection.

[0133] Step S30: Obtain the risk type labeled for the target sample.

[0134] The risk type labeled for the target sample refers to the risk type labeled for the target sample by professional labelers. These labelers possess relevant knowledge and experience, understand the definitions and judgment criteria of different risk types, and are thus able to accurately label the risk type to which the target risk sample belongs.

[0135] For example, the training terminal can display a type labeling interface. Based on this interface, labelers can select a target sample and its corresponding risk type. Accordingly, the training terminal obtains the risk type labeled for the target sample based on the user's labeling actions.

[0136] For example, to improve the accuracy of annotation, a multi-round review process can be adopted. First, one annotator performs the initial annotation, and then another annotator reviews it. For samples with disputes, experts can be organized to discuss and make a final determination.

[0137] It is understood that this application does not manually annotate all risk samples. Instead, it first uses a large-scale risk detection model to screen for challenging, high-quality samples, and then manually annotates these samples. This semi-automated annotation method can significantly reduce labor costs. Through this semi-automated annotation method, this application has mined and annotated over 200,000 high-quality training corpora.

[0138] Step S40: Train the risk detection big model based on the target sample and the risk type labeled for the target sample.

[0139] By labeling the target samples with risk types, the risk category represented by each sample is clearly defined. This provides explicit supervision information for model training, enabling the model to learn the characteristics and patterns of different risk types. During training, the model continuously learns the mapping relationship between the features of the target samples and their corresponding risk types, optimizing its parameters and structure. Through repeated training, the model can gradually improve its ability to identify complex risk types.

[0140] For example, before training a large-scale risk detection model based on target samples and the risk types labeled for those samples, it is pre-trained using a risk category structure. The risk category structure is a system for classifying and organizing the risks faced by the large language model. It not only includes multiple specific risk categories but also clarifies the hierarchical relationships between these categories. For instance, in a comprehensive risk classification system, "content security risk" can be a high-level category, further subdivided into more specific risk categories such as "politically sensitive content risk" and "vulgar and pornographic content risk." This hierarchical structure clearly demonstrates the logical relationships and hierarchical architecture between risks. For example, guided by the "Interim Measures for the Administration of Generative Artificial Intelligence Services" and the "Basic Requirements for the Security of Generative Artificial Intelligence Services," a risk category structure including over 100 risk categories such as terrorism, illegal activities, discrimination, pornography, and technological ethics is developed. Through deep learning of various risk contents within the risk category structure, the model is ensured to maintain efficient and accurate detection capabilities across multiple risk scenarios.

[0141] The risk detection scheme provided in this application first obtains a risk sample set, which includes multiple risk samples used to guide a large language model in generating risk content, providing a data foundation for subsequent sample selection. Then, the large-scale risk detection model selects target samples from the risk sample set, where the risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set. In other words, the more challenging risk samples are selected from the risk sample set. Considering that false negatives and missed detections often concentrate on these challenging target samples, the risk types of these target samples are specifically labeled, and the large-scale risk detection model is trained based on these labeled target samples. This enhances the model's ability to identify complex risks, reduces false negatives and missed detections, and improves detection accuracy. Furthermore, compared to training all risk samples indiscriminately, targeted training by selecting target samples utilizes computational resources and time more efficiently. Focusing on more challenging target samples allows for faster improvement of model performance, avoiding excessive resource waste on simple samples.

[0142] Based on the first embodiment described above, a second embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 3 In the second embodiment, step S20 includes steps S201 to S202:

[0143] Step S201: Using the risk detection big model, detect the risk type of each risk sample in the risk sample set and obtain the detection result corresponding to each risk sample.

[0144] The detection result is the judgment output by the risk detection model after analyzing the risk sample, indicating which risk type the risk sample is determined to belong to. It may include information such as the specific risk type and the probability of belonging to each risk type.

[0145] Step S202: Based on the detection results corresponding to each risk sample, select target samples from the risk sample set. The risk type detection difficulty of the target sample is greater than the risk type detection difficulty of other samples in the risk sample set.

[0146] Optionally, the detection results include the risk type of each risk sample and the probability that each risk sample belongs to the detected risk type. Accordingly, target samples are selected using uncertainty sampling. That is, based on the probability that each risk sample belongs to the detected risk type, risk samples with probabilities lower than the reference probability are selected from the risk sample set; the selected risk samples are then determined as target samples.

[0147] When a risk detection model determines whether a risk sample belongs to a certain risk type, it provides a probability to represent the confidence level of its judgment. This probability ranges from 0 to 1; the closer it is to 1, the more confident the model is that the sample belongs to the detected risk type; the closer it is to 0, the higher the uncertainty of the model in its judgment.

[0148] The reference probability is a pre-set threshold that serves as the standard for selecting target samples. When the probability of a risk sample belonging to the detected risk type is lower than this reference probability, the risk type of that sample is considered difficult to detect and requires further attention and analysis. For example, the reference probability can be 0.6, 0.7, etc. Its specific value can be set according to the accuracy requirements of the large-scale risk detection model.

[0149] The probability that each risk sample output by the model belongs to the detected risk type directly reflects the uncertainty of the model in judging the risk type of the sample. Therefore, based on probability screening, the selected target samples are those that the model truly struggles to handle. Targeted training on these samples can more effectively optimize the model and improve its risk detection capability.

[0150] Optionally, a large-scale risk detection model is used to detect the risk type of each risk sample in the risk sample set, obtaining detection results for each risk sample. This includes using multiple large-scale risk detection models to detect the risk type of each risk sample in the risk sample set, obtaining multiple detection results for each risk sample. Correspondingly, target samples are selected through a query-by-committee voting method. That is, based on the multiple detection results for each risk sample, risk samples with inconsistent detection results are selected from the risk sample set; the selected risk samples are then determined as target samples.

[0151] The inconsistent detection results from multiple risk detection models indicate that the risk characteristics of this sample are not obvious, making it highly deceptive and a truly difficult sample to detect. By selecting these samples as target samples, we can focus our attention on the most challenging samples and train them specifically, avoiding wasting time and resources on simple and easily judged samples. This improves the training efficiency and effectiveness of the risk detection model, ultimately increasing its accuracy in risk detection.

[0152] In this embodiment, target samples are selected based on detection results, directly addressing the difficulties encountered by large-scale risk detection models in actual detection processes. The detection results reflect the model's accuracy in judging the risk type of samples, intuitively showcasing samples that the model struggles to handle—that is, samples with truly challenging risk types. This selection method focuses on the model's actual performance, and the selected target samples accurately reflect the model's shortcomings. By training these samples specifically, the model can better learn the characteristics of complex risks, improve its detection capabilities, and more effectively handle risk detection tasks in practical applications.

[0153] Based on the second embodiment of this application described above, a third embodiment of this application is proposed. Contents that are the same as or similar to the second embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 4 In the third embodiment, after step S40, steps S501 to S503 are also included.

[0154] Step S501: Based on the detection results corresponding to each risk sample, select false detection samples from the risk sample set. False detection samples are samples that are incorrectly detected in terms of risk type.

[0155] False positives are samples where the risk type is incorrectly detected during the risk detection process. In other words, the risk detection model's assessment of the risk type for these samples does not match the actual situation. For example, it might misclassify a risky sample as a normal sample, or misclassify one risk type as another.

[0156] Step S502: Construct a new risk sample of the same type as the false positive sample.

[0157] New risk samples are constructed based on the characteristics and types of false positive samples. These new samples share the same risk type as the false positive samples, with the aim of increasing the quantity and diversity of that type of risk sample in the risk sample set. For example, new risk samples can be constructed by manually mimicking false positive samples, or by attacking a large model to simulate false positive samples and construct new risk samples.

[0158] Step S503: Add the new risk sample to the risk sample set.

[0159] It's important to note that by continuously identifying false positives in the model and constructing new risk samples of the same type, the risk sample set can be continuously improved. This iterative optimization of the model using risk samples from the risk sample set leads to a stable improvement in the model's risk detection performance.

[0160] In this embodiment, by filtering out false positive samples, it is possible to identify which samples the risk detection model misjudged, thus revealing the model's weaknesses. New risk samples of the same type as the false positive samples are constructed and added to the risk sample set, allowing the model to encounter more different manifestations of this type of risk during subsequent training. This enables the model to better learn and adapt to these types of risk problems. This helps the model learn more comprehensive and accurate risk characteristics, reduces false positives, and improves the overall detection accuracy of the model.

[0161] Based on the second embodiment of this application described above, a fourth embodiment of this application is proposed. Contents that are the same as or similar to the second embodiment can be referred to the above description, and will not be repeated hereafter. See also... Figure 5 In the fourth embodiment, after step S40, steps S601 to S603 are also included.

[0162] Step S601: Based on the detection results corresponding to each risk sample, select false detection samples from the risk sample set. False detection samples are samples with incorrect risk type detection.

[0163] Step S602: The false detection samples, the detection risk type of the false detection samples, and the labeled risk type are used to form training samples.

[0164] The detected risk type is the risk type determined by the risk detection model after it has detected a risk sample. This judgment is based on the model's algorithm and learned features. The labeled risk type is the result of manual labeling of the true risk type of the risk sample. It represents the actual risk type of the sample and serves as a reference standard for measuring the accuracy of the model's detection.

[0165] Training samples consist of a combination of false positive samples, the detection risk type of each false positive sample, and the labeled risk type. Training samples are used to further train and optimize the risk detection model, helping it learn the correct risk judgment rules.

[0166] Step S603: Optimize the risk detection model based on the training samples.

[0167] Preference optimization, also known as direct preference optimization (DPO), is the process of adjusting and improving a large-scale risk detection model based on training samples. By allowing the model to learn false positive samples and their correct risk types, the model can correct previous erroneous judgments, enhance its ability to accurately identify different risk types, and thus be more inclined to make correct judgments in subsequent detections.

[0168] Traditional reinforcement learning from human feedback (RLHF) typically involves multiple steps and is relatively complex when optimizing large language models, including training reward models. Preference optimization, however, simplifies the training process and significantly improves model training efficiency.

[0169] False positives often possess unique characteristics or exist in complex contexts, making the model prone to errors. By constructing training samples from false positives, their detection risk types, and labeled risk types, the model can be optimized for biases. Through continuous optimization of parameters and structure, the model can correct judgment biases and achieve sustained performance improvement.

[0170] Based on the first embodiment of this application described above, a fifth embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. See also... Figure 6 In the fifth embodiment, step S40 includes steps S401 and S404.

[0171] Step S401: Concatenate multiple target samples into a training sequence of fixed length, and determine the feature vector of each word in the training sequence.

[0172] A training sequence is a fixed-length sequence formed by concatenating multiple target samples. This sequence consists of a series of tokens. In natural language processing tasks, the text lengths of risk samples vary considerably. To facilitate batch processing and learning by the model, the length of the risk samples used for training needs to be consistent. If short risk samples are directly padded with the longest sequence using padding tokens, a large amount of unnecessary computation will occur. Therefore, this embodiment concatenates multiple short target samples into a fixed-length training sequence, reducing the use of padding tokens. This facilitates batch processing and learning by the model and reduces redundant computation.

[0173] A lexical unit is the basic unit of text segmentation in natural language processing; it can be a word or a character. For example, in English, a word can be considered a lexical unit; in Chinese, a Chinese character or a meaningful word segment can be treated as a lexical unit.

[0174] A feature vector is a vector obtained by transforming or encoding a word. It maps each word to a multi-dimensional vector space, enabling the model to better capture the features and patterns of the data. Feature vectors not only contain information about the word itself, but can also incorporate location information and relationships with other words, allowing the model to process and analyze the data.

[0175] Step S402: Using the risk detection big model, determine the attention weights between each word and its associated word in the training sequence, and update the feature vector of each word based on the determined attention weights. The associated word of each word belongs to the same sample as each word and is located before each word in the sample.

[0176] For each word in the training sequence, its associated word is a word that belongs to the same sample as the current word and is located before it in the sample. The information of the associated word can provide contextual information for the understanding and processing of the current word.

[0177] Attention weights are used to measure the degree of association between each word and its related words in a training sequence. In large-scale risk detection models, the attention mechanism calculates attention weights, enabling the model to pay more attention to related words associated with the current word, thereby better capturing semantic and contextual information in the text.

[0178] Optionally, using a large-scale risk detection model, the attention weights between each word and its associated word in the training sequence are determined, and the feature vectors of each word are updated based on the determined attention weights, including the following steps:

[0179] (1) By using the risk detection big model, the similarity matrix is ​​determined. The elements in the similarity matrix represent the feature similarity between word units in the training sequence.

[0180] For example, the attention layer of the risk detection model converts the feature vector of each word into a query vector and a key vector respectively; the dot product operation of each query vector and all key vectors is performed to obtain a similarity matrix. Each element (i, j) in the similarity matrix represents the similarity between the i-th query vector and the j-th key vector, where i and j are both natural numbers greater than 0.

[0181] A query vector is a vector obtained by performing a linear transformation on the original feature vector. In attention mechanisms, the query vector is used to perform matching calculations with the key vector to determine the degree of attention the feature vector at the current position has to feature vectors at other positions. The query vector can be viewed as a "query tool" used to search for relevant information throughout the sequence.

[0182] A key vector is a vector obtained by linearly transforming the original feature vector. Key vectors are used in conjunction with query vectors. Each key vector at a given location represents a characteristic of that location; by performing a dot product operation with the query vector, the correlation between feature vectors at different locations can be reflected.

[0183] In attention mechanisms, the dot product operation quickly calculates the similarity between the query vector and the key vector. The similarity matrix is ​​obtained by performing the dot product operation on the query vector and the key vector. The size of the similarity matrix is ​​N×N, where N is the number of feature vectors, and it shows the degree of correlation between the feature vectors of each word in the sequence.

[0184] (2) Obtain the mask matrix. The elements in the mask matrix represent the attention mask between words in the training sequence. The attention mask between each word and its associated words is 0, and the attention mask between each word and other words besides its associated words is negative infinity.

[0185] The mask matrix is ​​also a two-dimensional matrix, with the same dimensions as the similarity matrix. The elements in the mask matrix represent the attention masks between words in the training sequence, used to control the degree of attention the model gives to different words when calculating attention weights. In this scheme, the attention mask between each word and its associated word is set to 0, while the attention mask between each word and other words outside of its associated word is negative infinity.

[0186] For example, each element (i, j) in the mask matrix represents the attention mask between the i-th query vector and the j-th key vector. Furthermore, the element (i, j) is 0 when the i-th query vector and the j-th key vector belong to the same sample, and the term corresponding to the j-th key vector precedes the term corresponding to the i-th query vector, or the j-th key vector and the i-th query vector correspond to the same term. The element (i, j) is negative infinity when the i-th query vector and the j-th key vector do not belong to the same sample, or when the term corresponding to the j-th key vector follows the term corresponding to the i-th query vector.

[0187] Attention masks are used to limit the scope of a model's attention when calculating attention. When the attention mask is 0, it means that the model can normally pay attention to the corresponding word; when the attention mask is negative infinity, it means that the model will not pay attention to the corresponding word, that is, the model ignores the word.

[0188] Optionally, obtaining the mask matrix includes: determining the position information of each target sample in the training sequence, the position information including the start position, end position and position of each word in each target sample; and generating the mask matrix based on the position information of each target sample in the training sequence.

[0189] For example, the positional information of each target sample in the training sequence has been recorded when the training sequence is constructed. When generating the mask matrix, the positional relationship between the i-th query vector and the j-th key vector in element (i, j) is determined by traversing this positional information, thereby filling the value corresponding to element (i, j) in the mask matrix.

[0190] (3) Add the similarity matrix and the mask matrix and convert them into a probability distribution to obtain the attention matrix. The elements in the attention matrix represent the attention weights between word units in the training sequence. The attention weight between each word unit and other word units other than the associated word units is 0.

[0191] The attention matrix is ​​a probability distribution matrix obtained by adding the similarity matrix and the mask matrix and then transforming them. The elements in the matrix represent the attention weights between words in the training sequence, reflecting the degree of attention the model pays to other words when processing a particular word. In this scheme, due to the mask matrix, the attention weights between each word and other words besides its associated words are 0 or close to 0.

[0192] Attention weights reflect the degree of importance the model places on other words in the training sequence when processing a particular word. Higher attention weights indicate that the model pays more attention to the corresponding word when processing the current word, which helps the model capture long-distance dependencies and contextual information in the text.

[0193] (4) Update the feature vector of each word based on the attention matrix.

[0194] Optionally, in addition to converting the feature vector of each word into a query vector and a key vector, step (1) above also needs to convert each feature vector into a value vector. After obtaining the attention matrix, the attention matrix is ​​multiplied by the value vector matrix to obtain the updated feature vector matrix. Each row in the value vector matrix is ​​a value vector. Each row in the updated feature vector matrix is ​​the updated feature vector of each word, which incorporates the feature vectors of related words in the sequence.

[0195] Step S403: Using the risk detection big model, detect the risk type of each target sample in the training sequence based on the updated feature vectors of each word.

[0196] For example, the risk detection big model distinguishes the feature vectors of different samples in the training sequence based on the location information of each target sample in the training sequence, and then determines the risk type of each target sample based on the feature vector of the word in each target sample.

[0197] Step S404: Train a large-scale risk detection model based on the detection risk type and labeled risk type of each target sample.

[0198] Training a large-scale risk detection model based on the detection risk type and labeled risk type of each target sample refers to: determining the model's loss value using the detection risk type and labeled risk type of each target sample, and then adjusting the model parameters based on this loss value to reduce the loss value, thereby improving the model's detection performance. This training method is called supervised fine-tuning (SFT) of the large-scale risk detection model.

[0199] For example, the risk detection model in this application uses a decoder model based on a transformer architecture. Therefore, the attention weights between different samples can be set to 0 based on the 4D mask method supported by transformers. "4D" indicates that the mask has four dimensions, corresponding to the training batch size, the number of heads in the multi-head attention mechanism, and the sequence length, respectively, thereby precisely controlling the information interaction and computation process of the model at different samples, different attention heads, and different positions in the sequence. The method of setting the attention weights between different samples to 0 is the same as the implementation method in step S402.

[0200] It should be noted that the embodiments of this application apply the Packing technique, that is, combine multiple short target samples of different lengths into a fixed-length training sequence for processing, thereby reducing the waste of computing resources caused by padding, and improving the training efficiency of the risk detection model by 8 times.

[0201] For example, mixed-precision training and gradient accumulation techniques can be used during training to fine-tune model parameters and ensure model recognition performance. Mixed-precision training combines single-precision and half-precision data for computation during training. It leverages the speed and low memory usage of half-precision data to improve training speed and reduce memory consumption. Single-precision data is used in critical steps to ensure numerical stability, thereby improving training efficiency without significantly sacrificing model accuracy. Gradient accumulation, on the other hand, does not immediately update model parameters based on gradients calculated for each mini-batch. Instead, it calculates gradients on multiple mini-batch datasets and accumulates them. Once a sufficient number of mini-batch datasets have been accumulated, the accumulated gradients are used to update the model parameters. This technique can simulate larger batch sizes with limited hardware resources, thereby improving model training effectiveness.

[0202] In this embodiment, multiple target samples are concatenated into a fixed-length training sequence, facilitating batch processing by the model and significantly reducing computational overhead during training, thereby improving training efficiency. Furthermore, during training, an attention mechanism is used to update the feature vectors of each word, essentially fusing the features of related words into the features of each word in the training sequence. This achieves the fusion of contextual information into the word features of each sample while preventing information interference between different samples, allowing the model to more comprehensively and accurately understand the semantics of the text, thus improving the accuracy of risk detection.

[0203] Based on the first embodiment of this application described above, a sixth embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. See also... Figure 7 In the sixth embodiment, after step S40, steps S701 to S703 are also included.

[0204] Step S701: In response to the risk detection instruction, extract the target issue and the business type to which the target issue belongs from the risk detection instruction.

[0205] A risk detection command is a message initiated by a user or system to trigger a risk detection operation. It serves as the starting signal for the entire risk detection process, containing the specific content of the risk to be detected and related information, such as the target issue and its associated business type.

[0206] The target problem is the core issue extracted from the risk detection instructions; it is the specific object that needs to be risk-detected. It can be a piece of text, a request, a task description, etc., and will manifest in different forms depending on the business scenario.

[0207] Business type refers to the classification of the specific business area or business scenario to which the target question belongs. Different business types have different rules, risk characteristics, and processing requirements. For example, the business types in this application embodiment include children's smartwatch business, browser business, and nano search business. Specifically, the children's smartwatch business refers to the business of generating answers to questions asked on a children's smartwatch. The browser business is the business of generating answers to questions asked in a browser. The nano search business refers to the business of generating answers to questions asked in a nano search tool. The nano search tool is a tool focused on academic literature searching. Of course, business types can be divided according to business needs, and this application embodiment does not impose any limitations on this.

[0208] Step S702: Detect the risk type of the target problem using the risk detection model.

[0209] After training, the risk detection big model can accurately identify and judge the risk type. Therefore, by inputting the target question into the risk detection big model, the risk type to which the target question belongs can be obtained.

[0210] Step S703: Based on the risk type and business type of the target problem, determine the target handling method for the target problem.

[0211] The target handling method is the specific approach determined based on the risk type and business type of the target issue. Handling methods can include refusing to answer, manual review of the question, issuing a warning, continuing processing, and generating an answer using a specific, security-trained model. Different business types have varying degrees of stringency in controlling risky content. For example, for questions related to pornography or violent crime, the target handling method for a children's smartwatch business would be refusing to answer; while for a search engine business, the target handling method could be generating an answer using a security-trained model.

[0212] Optionally, based on the risk type and business type to which the target problem belongs, the target handling method for the target problem is determined, including: obtaining strategy configuration information that matches the business type, wherein the strategy configuration information contains the problem handling methods corresponding to each risk type; querying the problem handling methods corresponding to the risk type to which the target problem belongs from the strategy configuration information; and determining the queried problem handling methods as the target handling method for the target problem.

[0213] Strategy configuration information is a collection of rules that matches a specific business type. It details how different risk types should be handled under that specific business type. The handling methods for the same risk type may differ across strategy configuration information for different business types. Strategy configuration information can be pre-defined and stored in the system's configuration files or database, and can be adjusted and updated according to business needs and actual circumstances.

[0214] In this embodiment, by pre-setting strategy configuration information that matches the business type, corresponding handling methods are clearly defined for issues of different risk types. This allows the system to process target issues according to established rules, enhancing the standardization of business processing. Furthermore, the entire process eliminates the need for complex manual analysis and decision-making. This automated approach significantly reduces problem-solving time and improves business processing efficiency.

[0215] Optionally, the issue handling methods covered in the policy configuration information include at least two of the following: refusing to answer, generating an answer through a secure response big model, and generating an answer through a default response big model, wherein the secure response big model is a response big model trained with security data for security optimization.

[0216] The secure response model is a specially trained response model, optimized using a large amount of security data. This security data has been rigorously screened and labeled, and does not contain any sensitive, illegal, or harmful information. By training with this security data, the secure response model can generate responses that meet security standards and business requirements, effectively reducing the likelihood of risky content appearing in the responses.

[0217] The default large response model is a response model used when there are no special requirements or specific optimizations. It can be trained based on general training data and methods, and it does not have special optimizations for security when generating responses, so the generated responses may have certain risks.

[0218] The strategy configuration information includes multiple problem-handling methods, allowing for flexible selection of the appropriate method based on the risk type of the target problem. In this embodiment, the risk type of the problem can be further divided into three levels: secure, sensitive, and prohibited. For secure problems, a default response model is used to generate an answer; for sensitive problems, a secure response model is used; and for prohibited problems, an answer is refused. It should be noted that classifying a risk type as secure does not imply absolute security, but rather that the risk type poses a lower threat and is relatively safe under a specific business scenario.

[0219] In this embodiment, the processing methods for various types of questions are flexibly configured through policy configuration information. For high-risk questions, refusing to answer can directly avoid potential risks and prevent the leakage of sensitive information or the spread of illegal content. For low-risk questions, a secure response model is used to generate answers, leveraging its security-optimized training to ensure that the generated answers meet security standards. For normal questions, a default response model is used to generate answers, improving the efficiency of question processing. This flexibility allows for a balance between efficiency and security needs in different scenarios.

[0220] Figure 8 This is a schematic diagram illustrating a problem-solving process provided for some embodiments of this application. (Reference) Figure 8 After a user submits a question, a risk detection model is first used to assess the risk level of the question. If the risk level falls under the prohibited category in the current business context, the question will not be answered. If the risk level falls under the sensitive category, a secure response model will be used to generate an answer. If the risk level falls under the safe category, a default response model will be used to generate an answer; this default response model can also be referred to as the backbone model.

[0221] In this embodiment, the target processing method is determined based on the risk type and business type of the target problem, enabling personalized processing of different problems. Since different businesses have varying tolerances and processing requirements for the same risk type, and the impact of the same risk type differs in different business scenarios, this personalized processing method can better meet business needs and improve the efficiency and quality of business processing.

[0222] It should be noted that the key technical indicators of the risk detection big data model in the embodiments of this application cover the following aspects:

[0223] Accuracy: The model achieves an accuracy rate of over 95% in detecting various types of risk issues, enabling efficient classification and identification of risk problems.

[0224] Recall: The model maintains a recall rate of over 98%, which can minimize false negatives and false positives.

[0225] Performance optimization: For efficient processing of massive amounts of data and diverse content categories, the model performs exceptionally well in both real-time and batch processing, with an average response time of less than 80 milliseconds. Furthermore, the model employs a distributed cluster deployment, achieving a concurrent processing capacity of 1000 QPS (Queries Per Second).

[0226] Cross-domain adaptability: The model has a high degree of generalization and adaptability in different languages ​​and application scenarios, including detection support for more than 29 languages ​​such as Chinese, English, and French. Moreover, it can achieve good detection results in various business types such as Nano Search, AI (Artificial Intelligence) browser, and children's watches.

[0227] After deployment, the risk detection model processes over 20 million requests daily, cumulatively blocking over 50 million high-risk requests, maintaining a 98% success rate for normal requests while ensuring content security. Moving forward, we will continue to optimize the timeliness of risk perception and the lightweight nature of the model, providing the industry with verifiable and auditable content security solutions.

[0228] Another point to note is that the above examples are only for understanding this application and do not constitute a limitation on the risk detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0229] This application also provides a risk detection device, please refer to... Figure 9 The device includes:

[0230] The set acquisition module 10 is used to acquire a risk sample set, which includes multiple risk samples. These risk samples are used to guide the large language model to generate risk content.

[0231] The sample screening module 20 is used to screen target samples from the risk sample set using a risk detection model. The risk type detection difficulty of the target sample is greater than that of other samples in the risk sample set.

[0232] Type labeling module 30 is used to obtain the risk type labeled for the target sample;

[0233] The model training module 40 is used to train a large risk detection model based on the target sample and the risk type labeled for the target sample.

[0234] Optionally, the sample screening module 20 includes:

[0235] The result acquisition unit is used to detect the risk type of each risk sample in the risk sample set through the risk detection big model, and obtain the detection result corresponding to each risk sample.

[0236] The sample screening unit is used to screen target samples from the risk sample set based on the detection results corresponding to each risk sample.

[0237] Optionally, the detection results include the risk type to which each risk sample belongs and the probability that each risk sample belongs to the detected risk type;

[0238] The sample screening unit is used to screen risk samples from the risk sample set whose corresponding probability is lower than the reference probability based on the probability that each risk sample belongs to the detected risk type; and to determine the screened risk samples as target samples.

[0239] Optionally, the result acquisition unit is used to detect the risk type of each risk sample in the risk sample set through multiple risk detection models, and obtain multiple detection results corresponding to each risk sample.

[0240] The sample screening unit is used to screen out risk samples with inconsistent test results from the risk sample set based on the multiple test results corresponding to each risk sample; and to determine the screened risk samples as target samples.

[0241] Optionally, the model training module 40 includes:

[0242] The vector determination unit is used to concatenate multiple target samples into a fixed-length training sequence and determine the feature vector of each word in the training sequence.

[0243] The weight determination unit is used to determine the attention weight between each word and its associated word in the training sequence through the risk detection big model, and update the feature vector of each word based on the determined attention weight, wherein the associated word of each word belongs to the same sample as each word and is located before each word in the sample.

[0244] The type detection unit is used to detect the risk type of each target sample in the training sequence based on the updated feature vectors of each word using the risk detection big model.

[0245] The model training unit is used to train a large-scale risk detection model based on the detected risk type and labeled risk type of each target sample.

[0246] Optionally, the weight determination unit includes:

[0247] The first subunit is used to determine the similarity matrix through the risk detection big model. The elements in the similarity matrix represent the feature similarity between words in the training sequence.

[0248] The second subunit is used to obtain the mask matrix. The elements in the mask matrix represent the attention mask between words in the training sequence. The attention mask between each word and its associated words is 0, and the attention mask between each word and other words besides its associated words is negative infinity.

[0249] The third subunit is used to add the similarity matrix and the mask matrix and convert them into a probability distribution to obtain the attention matrix. The elements in the attention matrix represent the attention weights between word units in the training sequence. The attention weights between each word unit and other word units other than the word units associated with each word unit are 0.

[0250] The fourth subunit is used to update the feature vector of each word based on the attention matrix.

[0251] Optionally, the second sub-unit is used to determine the position information of each target sample in the training sequence. The position information includes the start position, end position, and position of each word in each target sample. Based on the position information of each target sample in the training sequence, a mask matrix is ​​generated.

[0252] Optionally, the device further includes:

[0253] The preference optimization module is used to filter out false detection samples from the risk sample set based on the detection results corresponding to each risk sample. False detection samples are samples that are incorrectly detected in terms of risk type. The false detection samples, the detected risk type of the false detection samples, and the labeled risk type are used to form training samples. The preference optimization is performed on the risk detection model based on the training samples.

[0254] Optionally, the device further includes:

[0255] The sample update module is used to filter out false positive samples from the risk sample set based on the detection results corresponding to each risk sample. False positive samples are samples with incorrect risk type detection; construct new risk samples with the same type as the false positive samples; and add the new risk samples to the risk sample set.

[0256] Optionally, the device further includes:

[0257] The instruction response module is used to respond to risk detection instructions and extract the target issue and the business type to which the target issue belongs from the risk detection instructions;

[0258] The type detection module is used to detect the risk type of the target problem using a large-scale risk detection model.

[0259] The method determination module is used to determine the target handling method for the target problem based on the risk type and business type to which the target problem belongs.

[0260] Optionally, the method determination module is used to obtain strategy configuration information that matches the business type, the strategy configuration information containing the problem handling methods corresponding to each risk type; query the problem handling methods corresponding to the risk type to which the target problem belongs from the strategy configuration information; and determine the queried problem handling methods as the target handling methods for the target problem.

[0261] Optionally, the problem handling methods covered in the policy configuration information include at least two of the following:

[0262] The options are: refusing to answer, generating an answer using the secure response model, and generating an answer using the default response model. The secure response model is a response model that has been trained with secure data for security optimization.

[0263] Optionally, the set acquisition module 10 is used to extract risk samples from the business log dataset of the question-and-answer system; or, extract risk samples from the publicly available domain dataset used for model content security construction; or, acquire manually constructed risk samples; or, acquire risk samples generated by attack large models; and construct a risk sample set based on the acquired risk samples.

[0264] The risk detection device provided in this application, employing the risk detection method described in the above embodiments, can solve the technical problem of low detection accuracy of large-scale risk detection models in related technologies. Compared with the prior art, the beneficial effects of the risk detection device provided in this application are the same as those of the risk detection method provided in the above embodiments, and other technical features in the risk detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0265] This application provides a risk detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the risk detection method in Embodiment 1 above.

[0266] The following is for reference. Figure 10 The diagram illustrates a structural schematic of a risk detection device suitable for implementing embodiments of this application. The risk detection device in this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The risk detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0267] like Figure 10 As shown, the risk detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the risk detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the risk detection device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show risk detection devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0268] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0269] The risk detection device provided in this application, employing the risk detection method described in the above embodiments, can solve the technical problem of low detection accuracy of large-scale risk detection models in related technologies. Compared with the prior art, the beneficial effects of the risk detection device provided in this application are the same as those of the risk detection method provided in the above embodiments, and other technical features of this risk detection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0270] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0271] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0272] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the risk detection method described in the above embodiments.

[0273] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0274] The aforementioned computer-readable storage medium may be included in the risk detection device; or it may exist independently and not be assembled into the risk detection device.

[0275] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the risk detection device, the risk detection device causes the risk detection device to: acquire a risk sample set, which includes multiple risk samples used to guide a large language model to generate risk content; select target samples from the risk sample set using the risk detection large model, wherein the risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set; acquire the risk type labeled for the target samples; and train the risk detection large model based on the target samples and the risk types labeled for the target samples.

[0276] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0277] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0278] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0279] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described risk detection method, which can solve the technical problem of low detection accuracy of large-scale risk detection models in related technologies. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the risk detection method provided in the above embodiments, and will not be repeated here.

[0280] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the risk detection method described above.

[0281] The computer program product provided in this application can solve the technical problem of low detection accuracy of large-scale risk detection models in related technologies. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the risk detection method provided in the above embodiments, and will not be repeated here.

[0282] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A risk detection method, characterized in that, The method includes: Obtain a risk sample set, which includes multiple risk samples, and the risk samples are used to guide the large language model to generate risk content. Using a large-scale risk detection model, target samples are selected from the risk sample set, and the risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set. Obtain the risk type labeled for the target sample; The risk detection model is trained based on the target sample and the risk type labeled for the target sample. The step of selecting target samples from the risk sample set using a large-scale risk detection model includes: The risk detection model is used to detect the risk type of each risk sample in the risk sample set, and the detection result corresponding to each risk sample is obtained. Based on the detection results corresponding to each risk sample, the target sample is selected from the risk sample set; The method further includes: Based on the detection results corresponding to each risk sample, false detection samples are selected from the risk sample set. The false detection samples are those that are incorrectly detected in terms of risk type. The false detection samples, the detected risk type of the false detection samples, and the labeled risk type are used to form training samples. The risk detection model is then optimized based on the training samples. Construct a new risk sample of the same type as the false positive sample; add the new risk sample to the risk sample set.

2. The method as described in claim 1, characterized in that, The detection results include the risk type to which each risk sample belongs and the probability that each risk sample belongs to the detected risk type; The step of selecting the target sample from the risk sample set based on the detection results corresponding to each risk sample includes: Based on the probability that each risk sample belongs to the detected risk type, risk samples with a probability lower than the reference probability are selected from the risk sample set. The selected risk samples are identified as the target samples.

3. The method as described in claim 1, characterized in that, The process involves using the large-scale risk detection model to detect the risk type of each risk sample in the risk sample set, obtaining the detection result for each risk sample, including: Using multiple risk detection models, the risk type of each risk sample in the risk sample set is detected, and multiple detection results are obtained for each risk sample. The step of selecting the target sample from the risk sample set based on the detection results corresponding to each risk sample includes: Based on multiple detection results corresponding to each risk sample, risk samples with inconsistent detection results are selected from the risk sample set. The selected risk samples are identified as the target samples.

4. The method as described in claim 1, characterized in that, The training of the risk detection model based on the target sample and the risk type labeled for the target sample includes: Multiple target samples are concatenated into a fixed-length training sequence, and the feature vector of each word in the training sequence is determined. The risk detection big model determines the attention weights between each word and its associated word in the training sequence, and updates the feature vector of each word based on the determined attention weights, wherein the associated word of each word belongs to the same sample as each word and is located before each word in the sample. The risk detection model is used to detect the risk type of each target sample in the training sequence based on the updated feature vectors of each word. The risk detection model is trained based on the detection risk type and labeled risk type of each target sample.

5. The method as described in claim 4, characterized in that, The process of determining the attention weights between each word and its associated word in the training sequence using the risk detection model, and updating the feature vector of each word based on the determined attention weights, includes: The risk detection big model is used to determine a similarity matrix, where the elements of the similarity matrix represent the feature similarity between lexical units in the training sequence. Obtain a mask matrix, wherein the elements in the mask matrix represent the attention mask between words in the training sequence, wherein the attention mask between each word and its associated words is 0, and the attention mask between each word and other words besides its associated words is negative infinity; The similarity matrix and the mask matrix are added together and converted into a probability distribution to obtain the attention matrix. The elements in the attention matrix represent the attention weights between word units in the training sequence, wherein the attention weight between each word unit and other word units other than the word unit associated with each word unit is 0. Based on the attention matrix, update the feature vector of each word.

6. A risk detection device, characterized in that, The device includes: The set acquisition module is used to acquire a risk sample set, which includes multiple risk samples. These risk samples are used to guide the large language model to generate risk content. The sample screening module is used to screen target samples from the risk sample set using a risk detection big data model. The risk type detection difficulty of the target samples is greater than that of other samples in the risk sample set. The type labeling module is used to obtain the risk type labeled for the target sample; The model training module is used to train the risk detection big model based on the target sample and the risk type labeled for the target sample; The sample screening module includes: The result acquisition unit is used to detect the risk type of each risk sample in the risk sample set through the risk detection big model, and obtain the detection result corresponding to each risk sample. A sample screening unit is used to screen the target sample from the risk sample set based on the detection results corresponding to each risk sample; The device further includes: The preference optimization module is used to filter out false detection samples from the risk sample set based on the detection results corresponding to each risk sample. The false detection samples are samples that are incorrectly detected in terms of risk type. The false detection samples, the detected risk type of the false detection samples, and the labeled risk type are used to form training samples. The preference optimization is performed on the risk detection model based on the training samples. The sample update module is used to filter out false detection samples from the risk sample set based on the detection results corresponding to each risk sample, wherein the false detection samples are samples with incorrect risk type detection; construct a new risk sample with the same type as the false detection sample; and add the new risk sample to the risk sample set.

7. A risk detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the risk detection method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the risk detection method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the risk detection method as described in any one of claims 1 to 5.

Images