Electric power large language model output text security detection method and related device

Abstract

The invention belongs to the field of electric power intelligence, and discloses an electric power large language model output text security detection method and a related device, and the method comprises the steps: obtaining a to-be-detected embedded vector, a current detection false alarm rate and a current detection missing report rate, and constructing a current state vector; calling a pre-trained benefit value evaluation network and a pre-trained risk value evaluation network according to the current state vector to obtain strategy probability distribution of a preset strategy set, and determining a target strategy according to the strategy probability distribution; and calling a pre-trained output security classifier according to the to-be-detected embedded vector to obtain a current security score, and obtaining a security detection result of a current output text of the to-be-detected electric power large language model in combination with the target strategy. A current security score is obtained through a lightweight output security classifier, a security detection result is obtained in combination with a target strategy, prison break attacks are defended with low calculation overhead, reinforcement learning is utilized to optimize a classification threshold to improve the accuracy and generalization under unknown attack distribution, and security reasoning and response generation are supported.

Description

Technical Field

[0001] This invention belongs to the field of power intelligence and relates to a method and related device for security detection of output text of a large power language model. Background Technology

[0002] Large language model output security detection has been used to identify harmful or biased outputs and misinformation generated by jailbreak attacks, such as through prefix hint injection, adversarial suffixes, and information encoding. Currently, some output security detection methods employ instruction-fine-tuned large language models for end-to-end discrimination of model outputs. For example, some methods use instruction-fine-tuned large language models based on training samples to identify malicious intent in user input prompts, detect security risks in the large model output, and evaluate the model's rejection rate; others use open-source evaluation models to extract semantic representations of the large model output and classify them into harmful and harmless categories, improving the accuracy of the evaluation and the model's generalization ability; still others concatenate text prompts and corresponding large model outputs from dataset samples containing five security dimensions (privacy, bias, toxicity, authenticity, and legitimacy) into an input template and use manually labeled scores as labels to train and evaluate the large model output score. Furthermore, some methods are based on large language model input / output security review models, which fine-tune the large language model by constructing a security risk classification system to identify harmful content in prompts and model-generated content, improving the accuracy and generalization ability of the large model's content review. However, large model fine-tuning and inference result in significant computational overhead, making online deployment in power scenarios difficult.

[0003] Based on this, some output security detection methods employ lightweight output security review methods, utilizing classifiers to discriminate intermediate layer representations of large models. For example, a content moderation scheme for large language models uses a three-layer perceptron to classify the word units output by the first and last layers of the large model being evaluated, and employs red team data augmentation to resist jailbreak attacks. Furthermore, other methods integrate the last layer hidden representation during the large language model's inference process into a multilayer perceptron and train it using label disambiguation to achieve real-time word-level security review with millisecond-level low latency; other methods obtain the refusal activation strength of each layer by calculating the similarity between the refusal semantics in the hidden state and the refusal vector constructed based on context, identifying unsafe prompts and resisting jailbreak attacks; still other methods perform singular value decomposition on the embedding matrix extracted from the large model to construct a malicious subspace, and use the distance from the sample to this subspace to generate a malice estimation score, thereby automatically classifying data into malicious and benign categories without manual annotation, and then training a classifier based on this to detect malicious prompts.

[0004] However, these classifier-based large language model output security detection methods require retraining of the classifier according to new business scenarios and new attack types, which increases the computational overhead. Furthermore, they lack generalization ability when facing constantly evolving jailbreak attacks or unknown risk scenarios, making it difficult to achieve accurate and effective security detection of power large language model outputs and unable to support security reasoning and response generation based on power large language models. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and related apparatus for security detection of output text of a large power language model.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention provides a method for security detection of output text from a power big data language model, comprising: acquiring a target embedding vector and the current false alarm rate and false negative rate, and constructing a current state vector; wherein the target embedding vector is the embedding vector corresponding to the current output text extracted from the last layer of the decoder during the inference phase of the power big data language model; based on the current state vector, calling a pre-trained benefit value evaluation network and a risk value evaluation network to obtain a policy probability distribution of a preset policy set, and determining a target policy based on the policy probability distribution; wherein the policies in the policy set are security detection classification thresholds quantized into several levels; based on the target embedding vector, calling a pre-trained output security classifier to obtain the current security score, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model.

[0007] Optionally, the current false alarm rate and false negative rate are obtained by: obtaining the accuracy of the most recent preset number of security detection results; wherein, the accuracy of the security detection results is obtained by: calling a preset large language model to obtain the security discrimination result of the output text, updating the uncertain security discrimination result in the security discrimination result to the manual review result, and obtaining the accuracy of the most recent preset number of security detection results based on the final security discrimination result and security detection result; based on the accuracy of the most recent preset number of security detection results, taking the proportion of security detection results that are incorrectly detected as harmful in the most recent preset number of detections as the current false alarm rate; and taking the proportion of security detection results that are incorrectly detected as safe in the most recent preset number of detections as the current false negative rate.

[0008] Optionally, constructing the current state vector includes: concatenating the embedding vector to be detected, the current false alarm rate, and the false negative rate to obtain the current state vector.

[0009] Optionally, the step of calling the pre-trained benefit value evaluation network and risk value evaluation network based on the current state vector to obtain the policy probability distribution of the preset policy set, and determining the current policy based on the policy probability distribution, includes obtaining the policy probability distribution of the preset policy set using the following formula:

[0010] in, Let be the policy probability of the current policy. Let this be the current state vector. As the current strategy, For benefit value evaluation network, The weight parameters of the benefit value evaluation network, for and The benefit value below, For risk weighting coefficients, For risk value assessment network, The weight parameters for the risk value assessment network. for and The risk value below, For a pre-defined strategy set, For strategy; for and The benefit value below, for and The risk value is determined by the strategy probability distribution; the strategy with the highest probability is selected as the target strategy.

[0011] Optionally, the pre-trained benefit value evaluation network and risk value evaluation network are obtained through the following method: constructing an initial benefit value evaluation network and an initial risk value evaluation network based on a neural network; obtaining a training experience replay pool, and training the initial benefit value evaluation network and the initial risk value evaluation network according to the training experience replay pool to obtain the pre-trained benefit value evaluation network and risk value evaluation network; wherein, the training experience replay pool includes several training samples, and the training samples are... , For the first Each training output text corresponds to a state vector. for Corresponding target strategy for Corresponding calculation benefits, for Corresponding calculation risk, For the first Each training output text corresponds to a state vector.

[0012] Optionally, the Corresponding calculation benefits We obtain it from the following formula:

[0013] in, for Corresponding false alarm rate To preset weights, for Corresponding false negative rate.

[0014] The Corresponding calculation risk We obtain it from the following formula:

[0015] in, For indicator functions, when When satisfied =1, otherwise =0; For the first State vectors Corresponding false negative rate This is the preset threshold for the false negative rate.

[0016] Optionally, the step of training the initial benefit value evaluation network and the initial risk value evaluation network according to the training experience replay pool to obtain the pre-trained benefit value evaluation network and risk value evaluation network includes: sampling from the training experience replay pool. The training samples are used, and the weight parameters of the initial benefit value evaluation network and the initial risk value evaluation network are updated according to the following formula:

[0017]

[0018] in, The weight parameters of the benefit value evaluation network, The weight parameters for the risk value assessment network. for and The benefit value below, for and The benefit value below, As a discount factor, for and The risk value below, for and The risk value below, For a pre-defined strategy set, For benefit value evaluation network, For risk value assessment networks.

[0019] Repeat the above steps until the false alarm rate and the false alarm rate are both less than the preset threshold, to obtain the pre-trained benefit evaluation network and risk evaluation network.

[0020] Optionally, it also includes: acquiring several detection samples based on several detection output texts and storing them in the training experience replay pool to obtain an updated experience replay pool, and updating the pre-trained benefit value evaluation network and risk value evaluation network according to the updated experience replay pool.

[0021] Optionally, both the benefit value evaluation network and the risk value evaluation network consist of an input layer, two hidden layers, and an output layer connected in sequence; the output safety classifier is constructed using a multilayer perceptron.

[0022] Optionally, the step of calling the pre-trained output security classifier based on the embedding vector to be detected to obtain the current security score, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected includes: when the current security score is less than the target policy, the security detection result of the current output text of the power big data language model to be detected is harmful; when the current security score is not less than the target policy, the security detection result of the current output text of the power big data language model to be detected is safe.

[0023] In a second aspect, the present invention provides a security detection system for the output text of a power big data language model, comprising: a state vector module, used to acquire the embedding vector to be detected, the current false alarm rate, and the false negative rate, and to construct a current state vector; wherein the embedding vector to be detected is the embedding vector corresponding to the current output text extracted from the last layer of the decoder during the inference stage of the power big data language model to be detected; a strategy selection module, used to, based on the current state vector, call a pre-trained benefit value evaluation network and a risk value evaluation network to obtain a strategy probability distribution of a preset strategy set, and determine a target strategy based on the strategy probability distribution; wherein the strategies in the strategy set are security detection classification thresholds quantized into several levels; and a security detection module, used to, based on the embedding vector to be detected, call a pre-trained output security classifier to obtain a current security score, and combine it with the target strategy to obtain the security detection result of the current output text of the power big data language model to be detected.

[0024] Optionally, the current false alarm rate and false negative rate are obtained by: obtaining the accuracy of the most recent preset number of security detection results; wherein, the accuracy of the security detection results is obtained by: calling a preset large language model to obtain the security discrimination result of the output text, updating the uncertain security discrimination result in the security discrimination result to the manual review result, and obtaining the accuracy of the most recent preset number of security detection results based on the final security discrimination result and security detection result; based on the accuracy of the most recent preset number of security detection results, taking the proportion of security detection results that are incorrectly detected as harmful in the most recent preset number of detections as the current false alarm rate; and taking the proportion of security detection results that are incorrectly detected as safe in the most recent preset number of detections as the current false negative rate.

[0025] Optionally, constructing the current state vector includes: concatenating the embedding vector to be detected, the current false alarm rate, and the false negative rate to obtain the current state vector.

[0026] Optionally, the step of calling the pre-trained benefit value evaluation network and risk value evaluation network based on the current state vector to obtain the policy probability distribution of the preset policy set, and determining the current policy based on the policy probability distribution, includes obtaining the policy probability distribution of the preset policy set using the following formula:

[0027] in, Let be the policy probability of the current policy. Let this be the current state vector. As the current strategy, For benefit value evaluation network, The weight parameters of the benefit value evaluation network, for and The benefit value below, For risk weighting coefficients, For risk value assessment network, The weight parameters for the risk value assessment network. for and The risk value below, For a pre-defined strategy set, For strategy; for and The benefit value below, for and The risk value is determined by the strategy probability distribution; the strategy with the highest probability is selected as the target strategy.

[0028] Optionally, the pre-trained benefit value evaluation network and risk value evaluation network are obtained through the following method: constructing an initial benefit value evaluation network and an initial risk value evaluation network based on a neural network; obtaining a training experience replay pool, and training the initial benefit value evaluation network and the initial risk value evaluation network according to the training experience replay pool to obtain the pre-trained benefit value evaluation network and risk value evaluation network; wherein, the training experience replay pool includes several training samples, and the training samples are... , For the first Each training output text corresponds to a state vector. for Corresponding target strategy for Corresponding calculation benefits, for Corresponding calculation risk, For the first Each training output text corresponds to a state vector.

[0029] Optionally, the Corresponding calculation benefits We obtain it from the following formula:

[0030] in, for Corresponding false alarm rate To preset weights, for Corresponding false negative rate.

[0031] The Corresponding calculation risk We obtain it from the following formula:

[0032] in, For indicator functions, when When satisfied =1, otherwise =0; For the first State vectors Corresponding false negative rate The preset false negative rate threshold is used; Optionally, the step of training the initial benefit value evaluation network and the initial risk value evaluation network according to the training experience replay pool to obtain the pre-trained benefit value evaluation network and risk value evaluation network includes: sampling from the training experience replay pool. The training samples are used, and the weight parameters of the initial benefit value evaluation network and the initial risk value evaluation network are updated according to the following formula:

[0033]

[0034] in, The weight parameters of the benefit value evaluation network, The weight parameters for the risk value assessment network. for and The benefit value below, for and The benefit value below, As a discount factor, for and The risk value below, for and The risk value below, For a pre-defined strategy set, For benefit value evaluation network, For risk value assessment networks.

[0035] Repeat the above steps until the false alarm rate and the false alarm rate are both less than the preset threshold, to obtain the pre-trained benefit evaluation network and risk evaluation network.

[0036] Optionally, it also includes: acquiring several detection samples based on several detection output texts and storing them in the training experience replay pool to obtain an updated experience replay pool, and updating the pre-trained benefit value evaluation network and risk value evaluation network according to the updated experience replay pool.

[0037] Optionally, both the benefit value evaluation network and the risk value evaluation network consist of an input layer, two hidden layers, and an output layer connected in sequence; the output safety classifier is constructed using a multilayer perceptron.

[0038] Optionally, the step of calling the pre-trained output security classifier based on the embedding vector to be detected to obtain the current security score, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected includes: when the current security score is less than the target policy, the security detection result of the current output text of the power big data language model to be detected is harmful; when the current security score is not less than the target policy, the security detection result of the current output text of the power big data language model to be detected is safe.

[0039] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described power big language model output text security detection method.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described power big data language model output text security detection method.

[0041] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a method for security detection of output text from a large-scale power language model. During the inference phase of the large-scale power language model, the embedding vector corresponding to the current output text is extracted from the last layer of the decoder. This vector, combined with the current false alarm rate and false negative rate, is used to construct the current state vector and serve as feature input. Then, pre-trained benefit and risk evaluation networks are invoked to obtain the policy probability distribution of a preset policy set. Based on this distribution, a target policy is determined, which reflects the most suitable security detection classification threshold. Finally, a pre-trained output security classifier is invoked based on the embedding vector to be detected to obtain the current security score. Combined with the target policy, the security detection result of the current output text of the large-scale power language model is obtained. This method can determine whether the output is secure using a lightweight output security classifier, resisting jailbreak attacks with low computational overhead. Furthermore, it utilizes reinforcement learning to optimize the current security detection classification threshold (target policy), improving the accuracy and generalization of the output security classifier under unknown attack distributions, thereby enhancing the accuracy of security detection of the output text from the large-scale power language model and supporting secure inference and response generation based on the large-scale power language model. Attached Figure Description

[0042] Figure 1 This is a flowchart of the power big language model output text security detection method according to an embodiment of the present invention.

[0043] Figure 2 This is a block diagram of the power big language model output text security detection system according to an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a method for security detection of output text from a large power language model is provided, which can effectively improve the accuracy and generalization of security detection under unknown attack distributions.

[0047] Specifically, the power big data language model output text security detection method of the present invention includes the following steps: S1: Obtain the embedding vector to be detected, as well as the current false alarm rate and false negative rate, and construct the current state vector; where the embedding vector to be detected is the embedding vector corresponding to the current output text extracted from the last layer of the decoder during the inference stage of the power big language model to be detected.

[0048] S2: Based on the current state vector, call the pre-trained benefit value evaluation network and risk value evaluation network to obtain the policy probability distribution of the preset policy set, and determine the target policy based on the policy probability distribution; wherein, the policy in the policy set is a security detection classification threshold quantized into several levels.

[0049] S3: Based on the embedding vector to be detected, call the pre-trained output security classifier to obtain the current security score, and combine it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected.

[0050] This invention discloses a method for security detection of output text from a large-scale power language model. During the inference phase of the large-scale power language model, the embedding vector corresponding to the current output text is extracted from the last layer of the decoder. This vector, combined with the current false alarm rate and false negative rate, is used to construct the current state vector and serve as feature input. Then, pre-trained benefit and risk evaluation networks are invoked to obtain the policy probability distribution of a preset policy set. Based on this distribution, a target policy is determined, which reflects the most suitable security detection classification threshold. Finally, a pre-trained output security classifier is invoked based on the embedding vector to be detected to obtain the current security score. Combined with the target policy, the security detection result of the current output text of the large-scale power language model is obtained. This method can determine whether the output is secure using a lightweight output security classifier, resisting jailbreak attacks with low computational overhead. Furthermore, it utilizes reinforcement learning to optimize the current security detection classification threshold (target policy), improving the accuracy and generalization of the output security classifier under unknown attack distributions, thereby enhancing the accuracy of security detection of the output text from the large-scale power language model and supporting secure inference and response generation based on the large-scale power language model.

[0051] In one possible implementation, the current false alarm rate and false negative rate are obtained by: obtaining the accuracy of the most recent preset number of security detection results; wherein the accuracy of the security detection results is obtained by: calling a preset large language model to obtain the security discrimination result of the output text, updating the uncertain security discrimination result in the security discrimination result to the manual review result, and obtaining the accuracy of the most recent preset number of security detection results based on the final security discrimination result and security detection result; based on the accuracy of the most recent preset number of security detection results, taking the proportion of security detection results that are incorrectly detected as harmful in the most recent preset number of detections as the current false alarm rate; and taking the proportion of security detection results that are incorrectly detected as safe in the most recent preset number of detections as the current false negative rate.

[0052] Interpretive methods employ a human-machine collaborative review mechanism to verify the accuracy of security detection results. First, a large language model, such as GPT-4, is used to perform security discrimination on the responses corresponding to the most recent preset number of embedding vectors, i.e., the most recent preset number of output texts. Then, the security discrimination results of the large language model are manually reviewed to ensure the accuracy of the verification of the security detection results.

[0053] In one possible implementation, constructing the current state vector includes: concatenating the embedding vector to be detected, the current false alarm rate, and the false negative rate to obtain the current state vector.

[0054] For example, the current state vector can be represented as ,in, Let be the embedding vector to be detected. The current false alarm rate, This represents the current false negative rate.

[0055] Explanatoryly, the current false alarm rate and false negative rate are calculated using a sliding window method, generally based on the most recent data. W Security check results for (e.g., 100) output texts.

[0056] In one possible implementation, the step of calling a pre-trained benefit evaluation network and risk evaluation network based on the current state vector to obtain the policy probability distribution of a preset policy set, and determining the current policy based on the policy probability distribution, includes obtaining the policy probability distribution of the preset policy set using the following formula:

[0057] in, Let be the policy probability of the current policy. Let this be the current state vector. As the current strategy, For benefit value evaluation network, The weight parameters of the benefit value evaluation network, for and The benefit value below, For risk weighting coefficients, For risk value assessment network, The weight parameters for the risk value assessment network. for and The risk value below, For a pre-defined strategy set, For strategy; for and The benefit value below, for and The risk value below.

[0058] Based on the policy probability distribution, the policy with the highest policy probability is selected as the target policy.

[0059] Interpretive calculations, considering both benefits and risks, are computationally complex and uncertain, and are obtained using a neural network fitting approach. This involves calculating the policy probability distribution of a predefined policy set under the current state vector, and then selecting the policy with the highest probability. The target strategy represents the current optimal security detection and classification threshold. Based on this, a dynamic security detection and classification threshold setting is implemented to accurately identify the type of the current security score output by the output security classifier in the current state.

[0060] In one possible implementation, the pre-trained benefit value evaluation network and risk value evaluation network are obtained by: constructing an initial benefit value evaluation network and an initial risk value evaluation network based on a neural network; obtaining a training experience replay pool, and training the initial benefit value evaluation network and the initial risk value evaluation network according to the training experience replay pool, thereby obtaining the pre-trained benefit value evaluation network and risk value evaluation network; wherein, the training experience replay pool includes several training samples, and the training samples are... , For the first Each training output text corresponds to a state vector. for Corresponding target strategy for Corresponding calculation benefits, for Corresponding calculation risk, For the first Each training output text corresponds to a state vector.

[0061] The explanatory, benefit value assessment network and risk value assessment network were pre-trained with full consideration of the first... Each training output text corresponds to a state vector. Corresponding target strategy Corresponding calculation benefits, Corresponding calculation risk and the first Each training output text corresponds to a state vector, enabling the benefit value evaluation network and the risk value evaluation network to learn from historical decision-making experience how to accurately evaluate the benefits and risks of taking each action under different states. The finally trained network can dynamically select the optimal security detection classification threshold based on the real-time state during actual deployment, achieving an adaptive balance between false alarm rate and false negative rate.

[0062] In one possible implementation, the Corresponding calculation benefits We obtain it from the following formula:

[0063] in, for Corresponding false alarm rate To preset weights, for Corresponding false negative rate.

[0064] The Corresponding calculation risk We obtain it from the following formula:

[0065] in, For indicator functions, when When satisfied =1, otherwise =0; For the first State vectors Corresponding false negative rate This is the preset threshold for the false negative rate.

[0066] Explained, this benefit-risk calculation design quantifies false alarm rate and false negative rate into benefit value and risk value, constructing a quantifiable, optimizable and risk-controllable evaluation framework. It not only transforms complex security performance into numerical targets that can be directly optimized by reinforcement learning, enabling the system to automatically balance the contradiction between false alarms and false negatives, but also monitors in real time whether the false negative rate exceeds the safety threshold through the risk value formula, ensuring that the system always stays within the preset safety boundary while pursuing maximum benefit.

[0067] In one possible implementation, training the initial benefit evaluation network and the initial risk evaluation network based on the training experience replay pool to obtain the pre-trained benefit evaluation network and risk evaluation network includes: sampling from the training experience replay pool. The training samples are used, and the weight parameters of the initial benefit value evaluation network and the initial risk value evaluation network are updated according to the following formula:

[0068]

[0069] in, The weight parameters of the benefit value evaluation network, The weight parameters for the risk value assessment network. for and The benefit value below, for and The benefit value below, As a discount factor, for and The risk value below, for and The risk value below, For a pre-defined strategy set, For benefit value evaluation network, For risk value assessment networks.

[0070] Repeat the above steps until the false alarm rate and the false alarm rate are both less than the preset threshold, to obtain the pre-trained benefit evaluation network and risk evaluation network.

[0071] For example, the false alarm rate is initialized during pre-training. and underreporting rate Initialize the learning rate Discount Factor Experience replay pool and the number of samples used The initial false alarm rate threshold and the false negative rate threshold are both 0.01.

[0072] In one possible implementation, the power big language model output text security detection method further includes: acquiring several detection samples based on several detection output texts and storing them in the training experience replay pool to obtain an updated experience replay pool, and updating the pre-trained benefit value evaluation network and risk value evaluation network according to the updated experience replay pool.

[0073] Interpretive, by continuously collecting detection samples and updating the experience replay pool in practical applications, the pre-trained benefit value evaluation network and risk value evaluation network can be further optimized using the newly added real-time data. This ensures that the detection can dynamically adapt to the distribution changes of the output text of the power big language model and new security threats, realize the continuous adaptive adjustment of the detection threshold, and continuously improve the accuracy and robustness of the detection.

[0074] In one possible implementation, both the benefit value evaluation network and the risk value evaluation network consist of an input layer, two hidden layers, and an output layer connected in sequence; the output safety classifier is constructed using a multilayer perceptron.

[0075] For example, both the benefit value assessment network and the risk value assessment network consist of an input layer, two hidden layers, and an output layer connected sequentially. The input layer has 10 neurons, the hidden layer has 128 neurons, and the output layer has 50 neurons. A multilayer perceptron is used to construct the output safety classifier, which is trained on the training dataset, and a safety score is output using the sigmoid function. .

[0076] For example, policies within a policy set This represents the security detection classification threshold, which is quantized into L levels. For example, if it is uniformly quantized into 10 levels, then 10 strategies are generated.

[0077] In one possible implementation, the step of calling the pre-trained output security classifier based on the embedding vector to be detected to obtain the current security score, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected includes: when the current security score is less than the target policy, the security detection result of the current output text of the power big data language model to be detected is harmful; when the current security score is not less than the target policy, the security detection result of the current output text of the power big data language model to be detected is safe.

[0078] Interpretive, the target strategy actually reflects the security detection classification threshold. This invention realizes the dynamic setting of the security detection classification threshold through reinforcement learning, so as not to rely on the adaptation of the output security classifier to different scenario states, but to achieve adaptation to different scenario states from the security detection classification threshold.

[0079] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0080] See Figure 2 In another embodiment of the present invention, a power big language model output text security detection system is provided, which can be used to implement the above-mentioned power big language model output text security detection method. Specifically, the power big language model output text security detection system includes a state vector module, a strategy selection module, and a security detection module.

[0081] The system comprises the following modules: a state vector module for acquiring the embedding vector to be detected, the current false alarm rate, and the false negative rate, and constructing the current state vector; wherein the embedding vector to be detected is the embedding vector corresponding to the current output text extracted from the last layer of the decoder during the inference stage of the power big data language model to be detected; a strategy selection module for calling the pre-trained benefit value evaluation network and risk value evaluation network based on the current state vector to obtain the strategy probability distribution of the preset strategy set, and determining the target strategy based on the strategy probability distribution; wherein the strategies in the strategy set are security detection classification thresholds quantized into several levels; and a security detection module for calling the pre-trained output security classifier based on the embedding vector to be detected to obtain the current security score, and combining it with the target strategy to obtain the security detection result of the current output text of the power big data language model to be detected.

[0082] In one possible implementation, the current false alarm rate and false negative rate are obtained by: obtaining the accuracy of the most recent preset number of security detection results; wherein the accuracy of the security detection results is obtained by: calling a preset large language model to obtain the security discrimination result of the output text, updating the uncertain security discrimination result in the security discrimination result to the manual review result, and obtaining the accuracy of the most recent preset number of security detection results based on the final security discrimination result and security detection result; based on the accuracy of the most recent preset number of security detection results, taking the proportion of security detection results that are incorrectly detected as harmful in the most recent preset number of detections as the current false alarm rate; and taking the proportion of security detection results that are incorrectly detected as safe in the most recent preset number of detections as the current false negative rate.

[0083] In one possible implementation, constructing the current state vector includes: concatenating the embedding vector to be detected, the current false alarm rate, and the false negative rate to obtain the current state vector.

[0084] In one possible implementation, the step of calling a pre-trained benefit evaluation network and risk evaluation network based on the current state vector to obtain the policy probability distribution of a preset policy set, and determining the current policy based on the policy probability distribution, includes obtaining the policy probability distribution of the preset policy set using the following formula:

[0085] in, Let be the policy probability of the current policy. Let this be the current state vector. As the current strategy, For benefit value evaluation network, The weight parameters of the benefit value evaluation network, for and The benefit value below, For risk weighting coefficients, For risk value assessment network, The weight parameters for the risk value assessment network. for and The risk value below, For a pre-defined strategy set, For strategy; for and The benefit value below, for and The risk value is determined by the strategy probability distribution; the strategy with the highest probability is selected as the target strategy.

[0086] In one possible implementation, the pre-trained benefit value evaluation network and risk value evaluation network are obtained by: constructing an initial benefit value evaluation network and an initial risk value evaluation network based on a neural network; obtaining a training experience replay pool, and training the initial benefit value evaluation network and the initial risk value evaluation network according to the training experience replay pool, thereby obtaining the pre-trained benefit value evaluation network and risk value evaluation network; wherein, the training experience replay pool includes several training samples, and the training samples are... , For the first Each training output text corresponds to a state vector. for Corresponding target strategy for Corresponding calculation benefits, for Corresponding calculation risk, For the first Each training output text corresponds to a state vector.

[0087] In one possible implementation, the Corresponding calculation benefits We obtain it from the following formula:

[0088] in, for Corresponding false alarm rate To preset weights, for Corresponding false negative rate.

[0089] The Corresponding calculation risk We obtain it from the following formula:

[0090] in, For indicator functions, when When satisfied =1, otherwise =0; For the first State vectors Corresponding false negative rate The preset false negative rate threshold is used; In one possible implementation, training the initial benefit evaluation network and the initial risk evaluation network based on the training experience replay pool to obtain the pre-trained benefit evaluation network and risk evaluation network includes: sampling from the training experience replay pool. The training samples are used, and the weight parameters of the initial benefit value evaluation network and the initial risk value evaluation network are updated according to the following formula:

[0091]

[0092] in, The weight parameters of the benefit value evaluation network, The weight parameters for the risk value assessment network. for and The benefit value below, for and The benefit value below, As a discount factor, for and The risk value below, for and The risk value below, For a pre-defined strategy set, For benefit value evaluation network, For risk value assessment networks.

[0093] Repeat the above steps until the false alarm rate and the false alarm rate are both less than the preset threshold, to obtain the pre-trained benefit evaluation network and risk evaluation network.

[0094] In one possible implementation, the power big language model output text security detection system further includes: acquiring several detection samples based on several detection output texts and storing them in the training experience replay pool to obtain an updated experience replay pool, and updating the pre-trained benefit value evaluation network and risk value evaluation network according to the updated experience replay pool.

[0095] In one possible implementation, both the benefit value evaluation network and the risk value evaluation network consist of an input layer, two hidden layers, and an output layer connected in sequence; the output safety classifier is constructed using a multilayer perceptron.

[0096] In one possible implementation, the step of calling the pre-trained output security classifier based on the embedding vector to be detected to obtain the current security score, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected includes: when the current security score is less than the target policy, the security detection result of the current output text of the power big data language model to be detected is harmful; when the current security score is not less than the target policy, the security detection result of the current output text of the power big data language model to be detected is safe.

[0097] All relevant content of each step involved in the aforementioned embodiments of the power big language model output text security detection method can be referenced to the functional description of the corresponding functional module of the power big language model output text security detection system in the embodiments of the present invention, and will not be repeated here.

[0098] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0099] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a power large-scale language model output text security detection method.

[0100] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the power large language model output text security detection method in the above embodiments.

[0101] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for security detection of output text from a large-scale power language model, characterized in that, include: Obtain the embedding vector to be detected, as well as the current false alarm rate and false negative rate, and construct the current state vector; where the embedding vector to be detected is the embedding vector corresponding to the current output text extracted from the last layer of the decoder during the inference stage of the power big language model to be detected. Based on the current state vector, the pre-trained benefit value evaluation network and risk value evaluation network are invoked to obtain the policy probability distribution of the preset policy set, and the target policy is determined based on the policy probability distribution; wherein, the policies in the policy set are security detection classification thresholds quantized into several levels. The current security score is obtained by calling the pre-trained output security classifier based on the embedding vector to be detected, and combined with the target policy, the security detection result of the current output text of the power big data language model to be detected is obtained.

2. The method for security detection of output text from a large power language model according to claim 1, characterized in that, The current false alarm rate and false negative rate are obtained in the following way: Obtain the accuracy of the most recent preset number of security detection results; wherein, the accuracy of the security detection results is obtained by calling the preset large language model to obtain the security judgment result of the output text, updating the uncertain security judgment result in the security judgment result to the manual review result, and obtaining the accuracy of the most recent preset number of security detection results based on the final security judgment result and security detection result; Based on the accuracy of the most recent preset number of security detection results, the proportion of security detection results that were incorrectly detected as harmful in the most recent preset number of detections is taken as the current false alarm rate; and the proportion of security detection results that were incorrectly detected as safe in the most recent preset number of detections is taken as the current false alarm rate.

3. The method for security detection of output text from a large power language model according to claim 1, characterized in that, The construction of the current state vector includes: The current state vector is obtained by concatenating the embedding vector to be detected, the current false alarm rate, and the false negative rate.

4. The method for security detection of output text from a large power language model according to claim 1, characterized in that, The step of calling the pre-trained benefit evaluation network and risk evaluation network based on the current state vector to obtain the policy probability distribution of the preset policy set, and determining the current policy based on the policy probability distribution includes: The policy probability distribution of the preset policy set can be obtained by the following formula: in, Let be the policy probability of the current policy. Let this be the current state vector. As the current strategy, For benefit value evaluation network, The weight parameters of the benefit value evaluation network, for and The benefit value below, For risk weighting coefficients, For risk value assessment network, The weight parameters for the risk value assessment network. for and The risk value below, For a pre-defined strategy set, For strategy; for and The benefit value below, for and The risk value below; Based on the policy probability distribution, the policy with the highest policy probability is selected as the target policy.

5. The method for security detection of output text from a large power language model according to claim 1, characterized in that, The pre-trained benefit evaluation network and risk evaluation network are obtained in the following manner: Construct an initial benefit value assessment network and an initial risk value assessment network based on neural networks; Obtain a training experience replay pool, and train an initial benefit value evaluation network and an initial risk value evaluation network based on the training experience replay pool to obtain pre-trained benefit value evaluation networks and risk value evaluation networks; wherein, the training experience replay pool includes several training samples, and the training samples are... , For the first Each training output text corresponds to a state vector. for Corresponding target strategy for Corresponding calculation benefits, for Corresponding calculation risk, For the first Each training output text corresponds to a state vector.

6. The method for security detection of output text from a large power language model according to claim 5, characterized in that, The Corresponding calculation benefits We obtain it from the following formula: in, for Corresponding false alarm rate To preset weights, for Corresponding false negative rate; The Corresponding calculation risk We obtain it from the following formula: in, For indicator functions, when When satisfied =1, otherwise =0; For the first State vectors Corresponding false negative rate This is the preset threshold for the false negative rate.

7. The method for security detection of output text from a large power language model according to claim 5, characterized in that, The process of training the initial benefit value evaluation network and the initial risk value evaluation network based on the training experience replay pool to obtain the pre-trained benefit value evaluation network and risk value evaluation network includes: Sampling from the training experience replay pool The training samples are used, and the weight parameters of the initial benefit value evaluation network and the initial risk value evaluation network are updated according to the following formula: in, The weight parameters of the benefit value evaluation network, The weight parameters for the risk value assessment network. for and The benefit value below, for and The benefit value below, As a discount factor, for and The risk value below, for and The risk value below, For a pre-defined strategy set, For benefit value evaluation network, For risk value assessment network; Repeat the above steps until the false alarm rate and the false alarm rate are both less than the preset threshold, to obtain the pre-trained benefit evaluation network and risk evaluation network.

8. The method for security detection of output text from a large power language model according to claim 7, characterized in that, Also includes: Several detection samples based on several detection output texts are obtained and stored in the training experience replay pool to obtain the updated experience replay pool. The pre-trained benefit value evaluation network and risk value evaluation network are updated according to the updated experience replay pool.

9. The method for security detection of output text from a large power language model according to claim 1, characterized in that, Both the benefit value assessment network and the risk value assessment network consist of an input layer, two hidden layers, and an output layer connected in sequence; the output safety classifier is constructed using a multilayer perceptron.

10. The method for security detection of output text from a large power language model according to claim 1, characterized in that, The process of obtaining the current security score by calling the pre-trained output security classifier based on the embedding vector to be detected, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected includes: When the current security score is less than the target policy, the security detection result of the current output text of the power big data language model to be detected is harmful; when the current security score is not less than the target policy, the security detection result of the current output text of the power big data language model to be detected is safe.

11. A power large language model output text security detection system, characterized in that, include: The state vector module is used to obtain the embedding vector to be detected, as well as the current false alarm rate and false negative rate, and to construct the current state vector; wherein, the embedding vector to be detected is the embedding vector corresponding to the current output text extracted from the last layer of the decoder during the inference stage of the power big language model to be detected. The strategy selection module is used to call the pre-trained benefit value evaluation network and risk value evaluation network according to the current state vector to obtain the strategy probability distribution of the preset strategy set, and determine the target strategy according to the strategy probability distribution; wherein, the strategies in the strategy set are security detection classification thresholds quantized into several levels. The security detection module is used to call the pre-trained output security classifier based on the embedding vector to be detected to obtain the current security score, and combine it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected.

12. The power large language model output text security detection system according to claim 11, characterized in that, The current false alarm rate and false negative rate are obtained in the following way: Obtain the accuracy of the most recent preset number of security detection results; wherein, the accuracy of the security detection results is obtained by calling the preset large language model to obtain the security judgment result of the output text, updating the uncertain security judgment result in the security judgment result to the manual review result, and obtaining the accuracy of the most recent preset number of security detection results based on the final security judgment result and security detection result; Based on the accuracy of the most recent preset number of security detection results, the proportion of security detection results that were incorrectly detected as harmful in the most recent preset number of detections is taken as the current false alarm rate; and the proportion of security detection results that were incorrectly detected as safe in the most recent preset number of detections is taken as the current false alarm rate.

13. The power large language model output text security detection system according to claim 11, characterized in that, The construction of the current state vector includes: The current state vector is obtained by concatenating the embedding vector to be detected, the current false alarm rate, and the false negative rate.

14. The power large language model output text security detection system according to claim 11, characterized in that, The step of calling the pre-trained benefit evaluation network and risk evaluation network based on the current state vector to obtain the policy probability distribution of the preset policy set, and determining the current policy based on the policy probability distribution includes: The policy probability distribution of the preset policy set can be obtained by the following formula: in, Let be the policy probability of the current policy. Let this be the current state vector. As the current strategy, For benefit value evaluation network, The weight parameters of the benefit value evaluation network, for and The benefit value below, For risk weighting coefficients, For risk value assessment network, The weight parameters for the risk value assessment network. for and The risk value below, For a pre-defined strategy set, For strategy; for and The benefit value below, for and The risk value below; Based on the policy probability distribution, the policy with the highest policy probability is selected as the target policy.

15. The power large language model output text security detection system according to claim 11, characterized in that, The pre-trained benefit evaluation network and risk evaluation network are obtained in the following manner: Construct an initial benefit value assessment network and an initial risk value assessment network based on neural networks; Obtain a training experience replay pool, and train an initial benefit value evaluation network and an initial risk value evaluation network based on the training experience replay pool to obtain pre-trained benefit value evaluation networks and risk value evaluation networks; wherein, the training experience replay pool includes several training samples, and the training samples are... , For the first Each training output text corresponds to a state vector. for Corresponding target strategy for Corresponding calculation benefits, for Corresponding calculation risk, For the first Each training output text corresponds to a state vector.

16. The power large language model output text security detection system according to claim 15, characterized in that, The Corresponding calculation benefits We obtain it from the following formula: in, for Corresponding false alarm rate To preset weights, for Corresponding false negative rate; The Corresponding calculation risk We obtain it from the following formula: in, For indicator functions, when When satisfied =1, otherwise =0; For the first State vectors Corresponding false negative rate This is the preset threshold for the false negative rate.

17. The power large language model output text security detection system according to claim 15, characterized in that, The process of training the initial benefit value evaluation network and the initial risk value evaluation network based on the training experience replay pool to obtain the pre-trained benefit value evaluation network and risk value evaluation network includes: Sampling from the training experience replay pool The training samples are used, and the weight parameters of the initial benefit value evaluation network and the initial risk value evaluation network are updated according to the following formula: in, The weight parameters of the benefit value evaluation network, The weight parameters for the risk value assessment network. for and The benefit value below, for and The benefit value below, As a discount factor, for and The risk value below, for and The risk value below, For a pre-defined strategy set, For benefit value evaluation network, For risk value assessment network; Repeat the above steps until the false alarm rate and the false alarm rate are both less than the preset threshold, to obtain the pre-trained benefit evaluation network and risk evaluation network.

18. The power large language model output text security detection system according to claim 17, characterized in that, Also includes: Several detection samples based on several detection output texts are obtained and stored in the training experience replay pool to obtain the updated experience replay pool. The pre-trained benefit value evaluation network and risk value evaluation network are updated according to the updated experience replay pool.

19. The power large language model output text security detection system according to claim 11, characterized in that, Both the benefit value assessment network and the risk value assessment network consist of an input layer, two hidden layers, and an output layer connected in sequence; the output safety classifier is constructed using a multilayer perceptron.

20. The power large language model output text security detection system according to claim 11, characterized in that, The process of obtaining the current security score by calling the pre-trained output security classifier based on the embedding vector to be detected, and combining it with the target policy to obtain the security detection result of the current output text of the power big data language model to be detected includes: When the current security score is less than the target policy, the security detection result of the current output text of the power big data language model to be detected is harmful; when the current security score is not less than the target policy, the security detection result of the current output text of the power big data language model to be detected is safe.

21. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the power big language model output text security detection method as described in any one of claims 1 to 10.

22. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the power big language model output text security detection method as described in any one of claims 1 to 10.

Images