Instruction response method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides an instruction response method, electronic equipment, a storage medium and a program product, after a to-be-processed instruction is received, a trained large language model is firstly used for prison break attack detection, and the safety of the instruction is judged. And if the instruction is safe (that is, the instruction does not contain the jail break attack behavior), a corresponding response result is generated by using the same model, so that the processing continuity and efficiency are ensured. And if the jail break attack behavior is detected, the instruction is rejected immediately, so that the potential safety risk is effectively prevented. The synchronous processing mode not only improves the safety of the model, but also ensures the timeliness of model response.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence security technology, and more specifically, to an instruction response method, electronic device, storage medium, and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) such as the GPT series of models have made significant progress in the field of natural language processing. These models are able to generate coherent and natural text and perform well in a variety of tasks, such as dialogue generation, text summarization, and machine translation. However, with the widespread use of large language models, security issues have gradually become prominent. In particular, malicious users may bypass the model security mechanism by entering "jailbreak instructions" to guide the model to generate unsafe or harmful content. This jailbreak attack behavior may not only mislead users, but also may lead to legal risks. Although some security protection measures for large language models have been proposed, there is still a lack of an effective method to enable the model to have the ability to detect jailbreak attacks and achieve simultaneous processing of jailbreak attack detection and model reasoning. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a command response method, an electronic device, a storage medium and a program product to achieve the technical effect of synchronously processing jailbreak attack detection and reasoning.

[0004] A first aspect of an embodiment of the present application provides a command response method, the method comprising:

[0005] Get pending instructions;

[0006] Inputting the instruction to be processed into a trained large language model, and using the trained large language model to perform jailbreak attack detection on the instruction to be processed to obtain a detection result;

[0007] If the detection result indicates that the instruction to be processed does not contain a jailbreak attack behavior, using the trained large language model to generate a response result for the instruction to be processed;

[0008] If the detection result indicates that the instruction to be processed includes the jailbreak attack behavior, instruct the trained large language model to reject the instruction to be processed.

[0009] In the above implementation process, after receiving the instruction to be processed, the trained large language model is first used to detect jailbreak attacks and determine the safety of the instruction. If the instruction is safe (that is, it does not contain jailbreak attack behavior), the same model is used to generate the corresponding response result, ensuring the consistency and efficiency of the processing. If a jailbreak attack behavior is detected, the instruction is immediately rejected, effectively preventing potential security risks. This synchronous processing method not only improves the security of the model, but also ensures the timeliness of the model response.

[0010] Furthermore, the generating a response result for the instruction to be processed by using the trained large language model includes:

[0011] Calculating an attention matrix corresponding to the instruction to be processed using the trained large language model; the attention matrix is ​​used to characterize the attention allocation relationship between each element in the instruction to be processed;

[0012] Instruct the trained large language model to generate the response result according to the attention matrix.

[0013] In the above implementation process, the attention matrix of the instruction to be processed is calculated by using the trained large language model, and the response result is generated accordingly. This response generation method based on the attention mechanism enables the model to understand the instruction content more accurately, thereby generating a response that is more in line with the instruction intention, effectively improving the accuracy of the instruction response.

[0014] Furthermore, the system memory stores an attention matrix corresponding to the instruction to be processed; the attention matrix is ​​used to characterize the attention allocation relationship between the elements in the instruction to be processed; and the use of the trained large language model to generate a response result for the instruction to be processed includes:

[0015] Obtaining the attention matrix;

[0016] Instruct the trained large language model to generate the response result according to the attention matrix.

[0017] In the above implementation process, the response speed of the model is greatly improved by reusing the previously calculated and stored attention matrix to generate the response result.

[0018] Further, after instructing the trained large language model to reject the instruction to be processed, the method further includes:

[0019] Delete the attention matrix.

[0020] In the above implementation process, after the trained large language model refuses to process the pending instruction, by deleting the attention matrix corresponding to the pending instruction, useless data can be effectively cleaned up and memory usage can be reduced.

[0021] A second aspect of an embodiment of the present application provides a large language model training method, the method comprising:

[0022] Acquire a training data set of a large language model; the training data set includes security data for training the language text processing capability of the large language model, and attack data for training the large language model to identify jailbreak attacks;

[0023] A preset detection level is added to the large language model, and the large language model is trained using the training data set to obtain a trained large language model; wherein the trained large language model is used to process instructions to be processed; and the detection level is used to detect whether the instructions to be processed are jailbreak attacks.

[0024] In the above implementation process, by integrating security data and attack data to build a training data set, and adding a detection layer to the model, a trained large language model that can respond to pending instructions and simultaneously detect jailbreak attacks is trained. This training method not only improves the model's language text processing capabilities, but also enhances the model's security protection capabilities, enabling the model to effectively identify and reject jailbreak attack instructions, ensuring the stable operation of the model.

[0025] Furthermore, the method further comprises:

[0026] Performing preset adapter training on the large language model based on the training data set to obtain a target weight;

[0027] The target weight is merged into the large language model to obtain the trained large language model.

[0028] In the above implementation process, by introducing the adapter training step, the large language model is fine-tuned based on the training data set to obtain the optimized target weights, and these weights are merged into the model to finally obtain the trained large language model. This not only improves the efficiency of model training, but also further enhances the model's ability to handle complex tasks and identify jailbreak attacks.

[0029] Furthermore, the using the training data set to train the large language model includes:

[0030] Inputting training data into the large language model, the training data including security data carrying security labels and attack data carrying attack labels;

[0031] Determine a first loss function based on the classification detection result output by the large language model for the training data and the label corresponding to the training data;

[0032] Determine a second loss function based on response data output by the large language model for the training data and the training data;

[0033] The first loss function and the second loss function are used to update the model parameters of the large language model to obtain the trained large language model.

[0034] By integrating the loss functions of the jailbreak attack classification detection task and the reasoning response task, the parameters of the large language model are comprehensively optimized, which significantly improves the overall performance and practicality of the model.

[0035] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising:

[0036] processor;

[0037] a memory for storing processor-executable instructions;

[0038] Wherein, when the processor calls the executable instruction, any method described in the first aspect and the second aspect is implemented.

[0039] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the steps of any one of the methods described in the first aspect and the second aspect are implemented.

[0040] A fifth aspect of an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method described in any one of the first and second aspects is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flowchart of a command response method provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a data flow processing flow provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of a large language model training method provided in an embodiment of the present application;

[0045] Figure 4A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0047] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0048] In related technologies, large language models do not have the ability to detect jailbreak attacks and do not support the simultaneous detection of jailbreak attacks and reasoning processes. Jailbreak attack detection is often completed by introducing an independent classification model before or after the large language model reasoning, but this processing method greatly affects the response speed of the model.

[0049] In response to any of the above-mentioned problems, the present application embodiment provides a command response method, referring to Figure 1 , Figure 1 A flowchart of a command response method provided in an embodiment of the present application.

[0050] In this embodiment, the method includes:

[0051] Step S10: Obtain instructions to be processed;

[0052] It should be noted that the instruction to be processed carries the instruction input by the user to the large language model, that is, carries the question input by the user.

[0053] Step S20: inputting the instruction to be processed into the trained large language model, and using the trained large language model to perform jailbreak attack detection on the instruction to be processed to obtain a detection result;

[0054] It should be noted that the trained large language model can be a natural language processing model based on deep learning, such as the GPT series, BERT series, etc. The trained large language model has both the ability to understand and reason about language text and the ability to detect jailbreak attacks.

[0055] Jailbreak attacks are attacks that bypass the security restrictions of large language models by inputting malicious instructions, inducing the model to produce outputs that violate its design intent or security guidelines. This attack exploits vulnerabilities in the model's internal processing mechanism and may cause the model to generate harmful, inappropriate or even illegal responses.

[0056] The detection result of the jailbreak attack can be a binary result (specifically, the binary result is: including / not including the jailbreak attack behavior) to guide the subsequent processing flow.

[0057] Specifically, the instructions input by the user enter the model, and a specific detection process is executed inside the model to determine whether the instructions to be processed contain jailbreak attack behavior. This process will output a binary result: if it is detected that it contains jailbreak attack behavior, an abnormal label (for example, label 1) is output; if no jailbreak attack behavior is detected, that is, it does not contain jailbreak attack behavior, then a normal label (for example, label 0) is output.

[0058] Step S30: if the detection result indicates that the instruction to be processed does not contain a jailbreak attack behavior, using the trained large language model to generate a response result for the instruction to be processed;

[0059] It is understandable that after confirming that the pending instruction does not contain jailbreak attack behavior, the model will generate an appropriate response result based on the content and context of the instruction. This response result can be varied, such as:

[0060] Result of executing the instruction: If the instruction is a query or calculation request, the model may directly return the query result or calculation result.

[0061] Parsing instructions: If the instruction carries a specific question, the model will analyze and process it to generate an answer to the question.

[0062] Further questions or suggestions: If the model believes that more information is needed to generate an accurate response, it will ask the user further questions.

[0063] Step S40: If the detection result indicates that the instruction to be processed includes the jailbreak attack behavior, instruct the trained large language model to reject the instruction to be processed.

[0064] Specifically, if the instruction to be processed includes a jailbreak attack, further processing of the instruction is stopped, and a fixed attack prompt response or a rejection response is returned.

[0065] like Figure 2 As shown, Figure 2A schematic diagram of a data flow processing flow provided for an embodiment of the present application. The figure shows the data flow processing flow of a large language model from bottom to top, and the bottom shows the prompt part of the user input, that is, the text entered by the user. Then, the prompt enters "Transform Layer1" to "Transform LayerN" for transformation and processing. It should be understood that the Transformer architecture of the model includes one or more transformation layers (Transformer Layer). The "N" in the figure is a general term, indicating that there are multiple such layers. Afterwards, the prompt processed by the transformation layer is synchronously input into the classification layer for jailbreak attack detection and enters the reasoning response process. While performing jailbreak attack detection, the reasoning response process will calculate the Key and Value matrix corresponding to the indication, or obtain the pre-stored Key and Value matrix related to the indication. When the classification layer completes the jailbreak attack detection and outputs a label, the model will determine whether to advance the subsequent reasoning response process based on the label. If the label indicates that the indication contains a jailbreak instruction, or that the indication is a jailbreak instruction, a fixed response (such as "suspected unsafe instruction") or a rejection response is output, the subsequent reasoning response process is not promoted, and the pre-stored Key and Value matrix related to the indication is deleted; if the label indicates that the indication does not contain a jailbreak instruction, or that the indication is not a jailbreak instruction, the calculated Key and Value matrix, or the pre-stored Key and Value matrix, is used to promote the subsequent reasoning response process.

[0066] In this embodiment, jailbreak attack detection and response to the instructions to be processed are realized through the trained large language model. It can not only effectively identify and reject instructions containing jailbreak attack behaviors to ensure system security, but also use the model to generate accurate response results for the instructions when the instructions are safe.

[0067] Based on any of the above embodiments, the step of using the trained large language model to generate a response result for the instruction to be processed includes:

[0068] Calculating an attention matrix corresponding to the instruction to be processed using the trained large language model; the attention matrix is ​​used to characterize the attention allocation relationship between each element in the instruction to be processed;

[0069] It should be noted that in the attention mechanism, three key components are usually involved: Query, Key, and Value. These three parts work together to achieve the dynamic attention of the model to the input data. Among them, the Query matrix: represents the information or context that currently needs to be paid attention to. The Key matrix: corresponds to the Query, and is used to calculate the correlation or similarity with the Query, so as to determine which Keys each Query should focus on. The Value matrix: according to the calculated correlation or similarity (that is, the result of the Query and Key similarity calculation), the Value is weighted and summed to obtain the final attention output.

[0070] In the implementation of the attention mechanism, Key and Value usually exist in the form of matrices. Each row of these matrices corresponds to the Key or Value representation of an element (such as a word, character, or subsequence) in the input data. The Key matrix is ​​used to perform similarity calculations with the Query to generate an attention matrix. The Value matrix is ​​used to perform weighted summation based on the weights in the attention matrix to obtain the final attention output. In this embodiment and the following embodiments, the attention matrix refers to the Key and Value matrices.

[0071] It should be understood that the generation of the attention matrix is ​​one of the important steps for the model to understand the input command. By calculating the attention relationship between elements, the model can more accurately capture the key information in the command and provide support for the subsequent generation of response results.

[0072] In the specific implementation, the trained large language model first receives the instruction to be processed as input. This instruction may be a sentence, paragraph, or a longer text sequence. The model preprocesses the input, such as word segmentation, word embedding (converting text into vector representation), etc. for subsequent processing. Subsequently, the model constructs two key matrices: the Key matrix and the Value matrix. The Key matrix is ​​used to store the key representations of each element (such as words, characters, or subsequences) in the input instruction. These key representations will be used to calculate the similarity or correlation between elements. The Value matrix stores the value representations corresponding to these keys. These value representations contain important information about the input elements and will be used in the subsequent attention weighting process.

[0073] Instruct the trained large language model to generate the response result according to the attention matrix.

[0074] Specifically, after obtaining the attention matrix, the model will perform a weighted summation of each element in the instruction according to the weights in the matrix. This is actually a weighted representation of the instruction, reflecting the importance of each element in the instruction in generating the response result. The representation obtained by weighted summation will be sent to the decoder part of the model. The decoder will generate the final response result based on this weighted representation. This process may involve a series of complex calculations and reasoning, including vocabulary selection, grammatical construction, and semantic understanding. During the decoding process, the model may use strategies such as greedy search and beam search to find the optimal response result. Finally, the model will output a response result for the instruction to be processed. This result can be a text sequence, a number, a classification label, or any other form of data.

[0075] In this embodiment, a trained large language model is used to generate response results for the instructions to be processed. The attention distribution relationship between elements is accurately represented by calculating the attention matrix corresponding to the instructions, and the model is guided to generate response results accordingly, thereby achieving a deep understanding and efficient processing of the instruction content and improving the accuracy of the response.

[0076] Based on any of the above embodiments, an attention matrix corresponding to the instruction to be processed is stored in the system memory; the attention matrix is ​​used to characterize the attention allocation relationship between each element in the instruction to be processed; and the use of the trained large language model to generate a response result for the instruction to be processed includes:

[0077] Obtaining the attention matrix;

[0078] Instruct the trained large language model to generate the response result according to the attention matrix.

[0079] It should be noted that the attention matrix is ​​not limited to being stored in the system memory, but can also be stored in the cloud and other locations.

[0080] KV Cache technology aims to improve the efficiency and speed of large language models when generating text. This technology achieves effective reuse of computing resources by caching the Key and Value matrices in the previously calculated attention mechanism. Specifically, when the model starts processing a new dialogue or text generation task, it first calculates the attention matrix corresponding to the current input and stores it in KV Cache together with the Key and Value matrices. In the subsequent dialogue or text generation process, when the model needs to generate the next word or character, it checks whether the Key and Value matrices related to the current input already exist in KV Cache. If so, the model can directly use these cached matrices without recalculating, which greatly reduces the amount of calculation and improves the response speed. In addition, KV Cache technology can also accelerate the autoregressive generation process. In autoregressive generation, the model needs to gradually generate each word or character in the text sequence, and each newly generated word or character will be used as the input of the next time step. By reusing the Key and Value matrices stored in KV Cache, the model can process these inputs more efficiently and generate high-quality text output.

[0081] In the specific implementation, KV Cache technology is integrated into the process of generating response results. The system memory stores the Key and Value matrices in the attention mechanism that have been calculated previously. Before starting to process a new instruction to be processed, first check whether there are Key and Value matrices related to the instruction or similar instructions in the memory (these matrices were generated in previous conversations or calculations and stored in KV Cache). In the process of autoregressive generation, the large language model no longer needs to calculate the Key and Value matrices for each time step from scratch. Instead, it directly uses the Key and Value matrices stored in KV Cache to speed up the calculation. The model calculates the output of the current time step based on the input of the current time step (which may be the words or characters generated in the previous time step) and the stored Key and Value matrices. This process is repeated until a complete response result is generated.

[0082] In this embodiment, the attention matrix stored in the system memory, as a representation of the attention distribution relationship between the elements of the instruction to be processed, is obtained and used to guide the trained large language model to generate response results. This not only reduces repeated calculations, but also significantly improves the response speed of the model.

[0083] Based on any of the above embodiments, after instructing the trained large language model to reject the pending instruction, the method further includes:

[0084] Delete the attention matrix.

[0085] It should be noted that storing the attention matrix requires system memory. When the model refuses to execute an instruction, it means that the attention matrix corresponding to the instruction will no longer be used in subsequent operations. Therefore, deleting these useless matrices in time can free up memory space.

[0086] Specifically, first, an event that the model refuses to execute the pending instruction needs to be detected, for example, by checking the output of the model. If it is detected that the model refuses to process the pending instruction, the attention matrix associated with the instruction is located, and the position of the matrix in the memory can be located using an index or identifier, and the located attention matrix is ​​removed from the memory. The attention matrix associated with the instruction can be calculated before processing the instruction.

[0087] As an example, in the same round of conversation, suppose the user enters three different instructions at different time points. When the model processes the first instruction, it calculates the corresponding Key and Value matrices and stores them in KVCache. Subsequently, the model uses these stored Key and Value matrices to generate the response results of the first conversation. If the model detects that the instruction contains a jailbreak attack when processing the second instruction, it will refuse to process this instruction. At the same time, it will delete the Key and Value matrices previously calculated and stored for the first instruction from KV Cache to ensure that these sensitive data will not be reused by subsequent operations. When the third instruction is received, the model will recalculate the new Key and Value matrices for the instruction and store these new matrices in KV Cache again. In this way, the model can use these new matrices to generate the response results of subsequent conversations.

[0088] In this embodiment, after instructing the trained large language model to reject the pending instruction, the system immediately deletes the attention matrix associated with the instruction, effectively avoiding the accumulation of useless data and ensuring the rational allocation and efficient use of system resources.

[0089] In addition, the present application embodiment also provides a large language model training method, referring to Figure 3 , Figure 3 A flowchart of a large language model training method provided in an embodiment of the present application.

[0090] In this embodiment, the method includes:

[0091] Step S01: Acquire a training data set of a large language model; the training data set includes security data for training the language text processing capability of the large language model, and attack data for training the large language model to identify jailbreak attacks;

[0092] It should be understood that security data: This part of data is used to train the model's language and text processing capabilities to ensure that the model can accurately understand and generate text that conforms to language standards. It can include various daily conversations, articles, etc.

[0093] Attack data: This part of the data contains information used to train the model to identify jailbreak attacks. Jailbreak attacks usually refer to the behavior of bypassing system security mechanisms and gaining unauthorized access rights. Attack data may include various forms of malicious instructions, code snippets, or attack patterns.

[0094] For example, in addition to security data corresponding to normal conversations and attack data containing jailbreak instructions, the training data set may also include a third type of data, namely, security-related question-answer data, such as password verification, permission requests, etc. This data is used to train the model to correctly handle sensitive information and ensure that the model complies with security guidelines and privacy policies.

[0095] Step S02: adding a preset detection level to the large language model, and using the training data set to train the large language model to obtain a trained large language model; wherein the trained large language model is used to process the instructions to be processed; and the detection level is used to detect whether the instructions to be processed are jailbreak attacks.

[0096] It should be noted that the preset detection layer can be a binary classification layer. The main task of this layer is to classify the instructions to be processed and determine whether they are jailbreak attacks, thereby ensuring that the model can identify jailbreak instructions. As an example, in a large language model of the Transformer architecture, convolutional layers or self-attention layers are often used to extract features of the input text. These features may contain key information related to jailbreak attacks, so the detection layer can be embedded after these layers. As another example, in an encoder-decoder architecture, the encoder is responsible for processing the input text and generating an intermediate representation. The detection layer can be embedded after an intermediate layer of the encoder to classify the encoded features. As another example, if the large language model is designed to perform multiple tasks (such as text generation, sentiment analysis, etc.), the detection layer can be embedded before the specific task layer.

[0097] It is understood that through training, the model is able to learn and understand the information in the training data set, including the processing rules of language text and the detection methods of jailbreak attacks. The training process usually involves multiple iterations, in each iteration the model processes a portion of the training data and adjusts its internal parameters based on the error between the predicted results and the true labels. This process is repeated until the performance of the model on the validation set reaches a certain predetermined standard. After training, the model will have the ability to process everyday language text and be able to detect and reject potential jailbreak attack instructions.

[0098] In this embodiment, by training the model in combination with security data and attack data, and adding a preset detection level to the model, the effect of accurately detecting and identifying jailbreak attacks while responding to pending instructions is achieved.

[0099] Based on any of the above embodiments, the method further includes:

[0100] Performing preset adapter training on the large language model based on the training data set to obtain a target weight;

[0101] The target weight is merged into the large language model to obtain the trained large language model.

[0102] It should be noted that adapter training is a parameter-efficient transfer learning method that allows the model to adapt to new tasks (i.e., identifying jailbreak attacks) by training a small number of newly added parameters (i.e., adapters) without changing the main structure of the large model, while maintaining the model's performance on the original task (i.e., language text processing). It is understandable that adapter training is used to help the model better identify jailbreak attacks.

[0103] It should be understood that adapters are usually designed as lightweight neural network structures that are inserted into specific locations in the model (such as after certain layers). As an example, the quantized low-rank adapter (QLoRA) technology is used to fine-tune the instructions of the large language model. The adapter is a quantized low-rank adapter, and a quantized low-rank adapter is inserted in each layer of the large language model. During the training process, only the parameters of these adapters are updated, while the rest of the model remains unchanged. The model is trained using a training data set, and the parameters of the adapter are optimized at the same time, which usually involves multiple iterations. In each iteration, the parameters of the adapter are adjusted according to the error between the predicted result and the true label. Through adapter training, a set of optimal adapter parameters, namely target weights, can be obtained. These weights represent the best performance of the model in the task of identifying jailbreak attacks. The trained adapter parameters (i.e., target weights) are merged into the original large language model, which means that the parameter values ​​of the adapter are updated to the corresponding positions in the model, thereby completing the model training process. After merging, the model has the ability to identify jailbreak attacks while still maintaining the performance of language text processing tasks. This model is the trained large language model.

[0104] In this embodiment, by introducing adapter training, efficient parameter transfer learning is achieved in the training process of the large language model. By training a small number of newly added parameters (i.e., adapters), the model can better adapt to new tasks (i.e., identifying jailbreak attacks) while maintaining the performance of the original tasks (such as language text processing). This training method not only improves the security and reliability of the model, but also reduces the training cost and computing resource requirements.

[0105] Based on any of the foregoing embodiments, the step of training the large language model using the training data set includes:

[0106] Inputting training data into the large language model, the training data including security data carrying security labels and attack data carrying attack labels;

[0107] Determine a first loss function based on the classification detection result output by the large language model for the training data and the label corresponding to the training data;

[0108] Determine a second loss function based on response data output by the large language model for the training data and the training data;

[0109] The first loss function and the second loss function are used to update the model parameters of the large language model to obtain the trained large language model.

[0110] It should be noted that the training data may include the data in the training data set, or other data that is independent of the training data set and prepared in advance.

[0111] The safe data carries a security label, indicating that the data is normal and has no attack, while the attack data carries an attack label, indicating that the data is malicious and attempts to jailbreak. The goal of the training process is to update the parameters of the model by minimizing the two loss functions so that the model can accurately distinguish between safe data and attack data.

[0112] A training data set containing security data and attack data is input into a large language model, and the data in the training data set is used to train the model so that the model can recognize and distinguish different types of instructions. After the model receives the training data, it will output a classification detection result, that is, whether the model believes that the input data is security data or attack data. This classification detection result is compared with the label (security label or attack label) corresponding to the training data, and the classification error, that is, the first loss function, is calculated. This loss function measures the performance of the model on the classification task, and the training goal is to minimize this loss function to improve the classification accuracy of the model. In addition to the classification detection result, the model also outputs response data for security data, compares these response data with security data, and calculates the response error, that is, the second loss function. This loss function measures the performance of the model on the response task, and the training goal is also to minimize this loss function to improve the response accuracy of the model. Exemplarily, after obtaining the two loss functions, the backpropagation algorithm is used to update the parameters of the model. The backpropagation algorithm calculates the gradient of the loss function to the model parameters, and then updates the parameters in the opposite direction of the gradient, thereby reducing the value of the loss function. This process will be iterated many times until the value of the loss function converges to a smaller range, at which point the model is considered to have been trained. After the above steps of training, a trained large language model is finally obtained.

[0113] In addition, in order to achieve real-time response and reasoning synchronization of jailbreak attack detection of large language models, improve the response speed of large language models and the accuracy of jailbreak attack detection, the embodiment of the present application also provides a method for detecting jailbreak instructions using large language models (LLMs). The method collects normal corpus and jailbreak instruction corpus extensively, performs preprocessing and constructs a question-answering dataset and marks labels, uses quantized low-rank adapter (QLoRA) technology to fine-tune the model instructions, and optimizes the loss function of the autoregressive task and the jailbreak instruction classification task. In the reasoning stage, KV Cache technology is used to accelerate autoregressive generation, and the jailbreak instruction detection head is used to identify and process potential unsafe instructions to ensure the security and reliability of the model output. The specific implementation steps are as follows: (1) Corpus collection: widely collect the corpus required for model fine-tuning, including normal dialogue corpus, jailbreak instructions and secure alignment corpus. Among them, normal dialogue corpus and secure alignment corpus constitute the question-answering dataset, and jailbreak instructions only contain prompts input to the model, not the response of the model. The collection of jailbreak instructions is done by collecting jailbreak instructions and templates to generate a large-scale jailbreak prompt dataset; the other part is to use the automated jailbreak algorithm to generate a large-model jailbreak prompt dataset. (2) Data preprocessing and label setting: For normal dialogue corpus and security alignment corpus, only the prompts and responses of the first round of dialogue are used as training datasets. This is because: whether the prompts and responses of one round of dialogue are used for training or the prompts and responses of multiple rounds of dialogue are used for training, their training results are the same. For the jailbreak prompt dataset, a unified fixed response is constructed, such as "suspected unsafe instructions", and a question-answering dataset of prompts and responses is formed. Then, according to the model input template, the prompts and responses are respectively placed in the corresponding tags (the format example is as follows: <|im_start|>system\nsystem prompt<|im_end|>\n'<|im_start|>user:input prompt<|im_end|>\n<|im_start|>assistant:response;), and the prompt parts of the normal dialogue corpus, security alignment corpus and jailbreak prompt are extracted, and the corresponding labels are set respectively. The prompt part of the normal dialogue corpus is labeled 0, the jailbreak prompt label is 1, and the security alignment corpus sets the label according to the instruction content. If it contains attack behavior, it is marked as 1, otherwise it is marked as 0 or 1 according to the model response. Finally, write the formatted input and label into the json file. A single data example: {"prompt":'<|im_start|>system\nSystem prompt<|im_end|>\n'<|im_start|>user:Input prompt: How can I be energetic? <|im_end|>\n<|im_start|>assistant:First of all, you must ensure adequate sleep every day....','label':0}.(3) Model adjustment training: Adjust the training of the large language model, including two tasks: instruction fine-tuning and jailbreak instruction classification. First, select a suitable large language model as the basic model, and add a binary classification layer to the output of the last transformer layer of the assistant token to perform the classification task at the same time. Then, use the preprocessed training data and QLoRA technology (i.e., quantized low-rank adapter) to fine-tune the model for instructions. During the fine-tuning process, the model loss is divided into the autoregressive language model loss and the classification loss of jailbreak instruction detection (i.e., the model loss is divided into the loss of the instruction fine-tuning task and the loss of the jailbreak instruction classification task). The two parts of the loss are calculated through the total loss function to update the large language model and obtain the trained large language model. Among them, the loss function is shown in the following formula 1:.

[0114] Loss=Lossautoreg+Lossclass Formula 1;

[0115] In formula 1, L autoreg is the loss function of the autoregressive task, L class is the loss function of the classification detection task. The loss function of the autoregressive task uses the cross entropy loss function, assuming that the probability distribution of the token generated by the model at step t is p(ω t |ω <t ), where ω t is the target word of the tth step, and the cross entropy loss function can be expressed as follows:

[0116]

[0117] In Formula 2, T is the total number of tokens in the output corpus (prompt + response + template), ω t is the target word of step t, ω <t represents all tokens before step t, and k is the number of all tokens in the input corpus up to assistant;

[0118] The loss function of jailbreak instruction detection is the cross entropy loss function of binary classification. Assume that the output of the classification layer is P when T = k. class (yω k ), where y is the binary label, i.e. the label of the prompt. Then, the loss function of the classification task can be expressed as follows:

[0119] Loss class =- ∑ y∈{0,1} log(y | w k ) Formula 3;

[0120] Then, the total loss function for calculating the instruction fine-tuning task loss and the jailbreak instruction classification task loss can be expressed as follows:

[0121]

[0122] (4) Model deployment and application: During the model deployment and application phase, the prompt instructions input by the user are input into the jailbreak instruction detection head (the detection head is equivalent to the binary classification layer) in the trained large language model for security identification. At the same time, the trained large language model records the Key and Value matrices of the reasoning process. If the detection result is 0, that is, the input prompt does not have a jailbreak risk, the cached Key and Value matrices are used to perform the large language model autoregressive task reasoning output to reduce repeated calculations. If the detection result is 1, a fixed response (such as "suspected unsafe instruction") or a rejection response is returned, and the cached Key and Value matrices are deleted to prevent further unsafe reasoning. In this way, jailbreak instructions can be effectively detected and prevented, ensuring the security and reliability of the system.

[0123] In this embodiment, two loss functions are used to guide the training of the large language model. The first loss function measures the performance of the model on the classification task (i.e., jailbreak attack detection task), while the second loss function measures the performance of the model on the response task. By minimizing these two loss functions, a trained large language model that performs well in both classification (i.e., jailbreak attack detection) and response tasks can be obtained. Not only is the accuracy of the response results generated by the model improved, but the model's ability to recognize jailbreak instructions is also enhanced.

[0124] Based on the method described in any of the above embodiments, the present application also provides Figure 4 A schematic diagram of the structure of an electronic device is shown in FIG. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method described in any of the above embodiments.

[0125] Based on the method described in any of the above embodiments, the present application also provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute the method described in any of the above embodiments.

[0126] Based on the method described in any of the above embodiments, the present application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. When the computer program is executed by a processor, the method described in any of the above embodiments is implemented.

[0127] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0129] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0130] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0131] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0132] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A command response method, characterized in that: The method comprises: Get pending instructions; Inputting the instruction to be processed into a trained large language model, and performing jailbreak attack detection on the instruction to be processed using the trained large language model to obtain a detection result; If the detection result indicates that the instruction to be processed does not contain a jailbreak attack behavior, using the trained large language model to generate a response result for the instruction to be processed; If the detection result indicates that the instruction to be processed includes the jailbreak attack behavior, instruct the trained large language model to reject the instruction to be processed.

2. The method according to claim 1, characterized in that The generating a response result for the instruction to be processed by using the trained large language model includes: Calculating an attention matrix corresponding to the instruction to be processed using the trained large language model; the attention matrix is ​​used to characterize the attention allocation relationship between each element in the instruction to be processed; Instruct the trained large language model to generate the response result according to the attention matrix.

3. The method according to claim 1, characterized in that The attention matrix corresponding to the instruction to be processed is stored in the system memory; the attention matrix is ​​used to represent the attention allocation relationship between each element in the instruction to be processed; The generating a response result for the instruction to be processed by using the trained large language model includes: Obtaining the attention matrix; Instruct the trained large language model to generate the response result according to the attention matrix.

4. The method according to claim 2 or 3, characterized in that After instructing the trained large language model to reject the pending instruction, the method further includes: Delete the attention matrix.

5. A large language model training method, characterized in that: The method comprises: Acquire a training data set of a large language model; the training data set includes security data for training the language text processing capability of the large language model, and attack data for training the large language model to identify jailbreak attacks; A preset detection level is added to the large language model, and the large language model is trained using the training data set to obtain a trained large language model; wherein the trained large language model is used to process instructions to be processed; and the detection level is used to detect whether the instructions to be processed are jailbreak attacks.

6. The method according to claim 5, characterized in that The method further comprises: Performing preset adapter training on the large language model based on the training data set to obtain a target weight; The target weight is merged into the large language model to obtain the trained large language model.

7. The method according to claim 5, characterized in that The step of training the large language model using the training data set includes: Inputting training data into the large language model, the training data including security data carrying security labels and attack data carrying attack labels; Determine a first loss function based on the classification detection result output by the large language model for the training data and the label corresponding to the training data; Determine a second loss function based on response data output by the large language model for the training data and the training data; The first loss function and the second loss function are used to update the model parameters of the large language model to obtain the trained large language model.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, when the processor calls the executable instruction, the method described in any one of claims 1-7 is implemented.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the steps of any method described in claims 1-7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images