Sentence text processing method, device, equipment and storage medium

Abstract

The present application provides a sentence text processing method, device, equipment and storage medium, the method comprising: obtaining an input query sentence; inputting the query sentence into a pre-trained semantic representation model to obtain a classification result of whether the query sentence has restricted content; in response to the classification result that the query sentence has restricted content, inputting the to-be-processed text corresponding to the query sentence into a preset large language model, determining the hierarchical information of each text character in the to-be-processed text and the character embedding features corresponding to the to-be-processed text; determining the embedding layer features of the large language model according to the character embedding features and each hierarchical information, and updating the large language model according to the embedding layer features, so that the updated large language model outputs an answer result according to the query sentence. The technical solution of the embodiment of the present application performs early warning and defense of restricted content from both the input side and the model side of the large language model, thereby improving the defense capability of the large language model against input restricted content.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and specifically to a sentence text processing method, apparatus, device and storage medium. Background Art

[0002] With the continuous development of computer technology and artificial intelligence technology, various users have an increasing demand for intelligence in production and life. Among them, technologies such as large language models and natural language processing are favored by people in various fields due to their wide applicability and high work efficiency.

[0003] Currently, using large language models to assist people's production and life has become a common technology. However, the large language models in the existing technology have poor ability to distinguish restricted content of the input, resulting in poor output results of the large language models. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present application aims to provide a sentence text processing method, apparatus, device and storage medium to improve the defense capability of a large language model against input restricted content, so that the output results of the large language model can be more secure and harmless.

[0005] The present application provides a method for processing a sentence text, the method comprising:

[0006] Get the input query statement;

[0007] Input the query sentence into the pre-trained semantic representation model to obtain the classification result of whether the query sentence contains restricted content;

[0008] In response to the classification result that the query sentence has restricted content, the to-be-processed text corresponding to the query sentence is input into a preset large language model to determine the hierarchical information of each text character in the to-be-processed text and the character embedding features corresponding to the to-be-processed text;

[0009] According to the character embedding features and information at each level, the embedding layer features of the large language model are determined, and the large language model is updated according to the embedding layer features, so that the updated large language model outputs an answer result according to the query statement.

[0010] Optionally, in response to the classification result that the query statement contains restricted content, inputting the to-be-processed text corresponding to the query statement into a preset large language model includes:

[0011] In response to the classification result that the query sentence has restricted content, the query sentence is input into a preset prompt text generation model to obtain a prompt guide text of the query sentence, and the prompt guide text is input into the large language model as a text to be processed.

[0012] Optionally, determining the embedding layer features of the large language model according to the character embedding features and the information of each level includes:

[0013] According to the information of each level, determine the hierarchical embedding features of the prompt guidance text;

[0014] Add the character embedding features and the level embedding features to get the embedding layer features.

[0015] Optionally, determining the hierarchical embedding features of the prompt guidance text according to the information of each level includes:

[0016] Multiply the vector of the hierarchical information corresponding to each text character in the prompt guidance text with the pre-constructed prompt segment embedding matrix to obtain the hierarchical embedding feature;

[0017] Among them, the prompt fragment embedding matrix is ​​constructed according to the embedding layer identification value of the large language model and the character embedding dimension of the large language model.

[0018] Optionally, the step of inputting the query statement into a pre-trained semantic representation model to obtain a classification result of whether the query statement contains restrictive content includes:

[0019] Input the query sentence into the pre-trained semantic representation model to obtain the sentence features corresponding to the query sentence;

[0020] Matching at least one sensitive word entity appearing in the query statement in a preset sensitive word library, and determining entity features of each sensitive word entity;

[0021] The enhanced features are obtained by averaging the features of each entity;

[0022] The classification result is determined based on the sentence features and enhanced features.

[0023] Optionally, the sentence vector corresponding to the sentence feature and the enhancement vector corresponding to the enhancement feature have different dimensions, and determining the classification result according to the sentence feature and the enhancement feature includes:

[0024] Concatenate the sentence vector and the enhanced vector to determine the fusion feature;

[0025] Based on the fused features, the classification result is determined.

[0026] Optionally, the method further includes:

[0027] In response to the classification result that the query sentence does not have any restrictive content, the query sentence is input into the large language model as a text to be processed.

[0028] The present application also provides a sentence text processing device, which includes:

[0029] A query statement acquisition module is used to acquire an input query statement;

[0030] A classification result determination module is used to input the query statement into the pre-trained semantic representation model to obtain a classification result of whether the query statement contains restricted content;

[0031] A restricted content processing module is used for inputting the to-be-processed text corresponding to the query sentence into a preset large language model in response to the classification result that the query sentence has restricted content, and determining the hierarchical information of each text character in the to-be-processed text and the character embedding features corresponding to the to-be-processed text;

[0032] The answer result output module is used to determine the embedding layer features of the large language model based on the character embedding features and information at each level, and update the large language model based on the embedding layer features so that the updated large language model outputs the answer result according to the query statement.

[0033] The present application also provides an electronic device, the electronic device comprising:

[0034] Processor and memory;

[0035] The processor is used to execute the steps of the statement text processing method provided in any embodiment of the present application by calling the program or instruction stored in the memory.

[0036] An embodiment of the present application also provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of the sentence text processing method provided by any embodiment of the present application.

[0037] In summary, in the technical solution of the embodiment of the present application, by inputting the query statement into the pre-trained semantic representation model, a classification result of whether the query statement contains restricted content is obtained, and the semantic representation model is used to make a pre-judgment on whether the query statement contains restricted content, so as to help the large language model get an early warning before the query statement is input; after determining that there is restricted content in the query statement, the embedding layer features of the large language model are determined according to the hierarchical features of the text characters and the character embedding features of the text to be processed, so as to update the model, so that the updated large language model can output the answer result according to the query statement, and the restricted content can be processed in the model, so as to prevent the large language model from outputting the answer result with restricted content due to answering the question with restricted content. The technical solution of the embodiment of the present application performs early warning and defense of restricted content from both the input side and the model side of the large language model, improves the overall filtering effect of restricted content, improves the defense capability of the large language model against the input restricted content, and makes the output result of the large language model safer and harmless. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 is a flowchart of a sentence text processing method provided by an embodiment of the present application;

[0040] Figure 2A It is a schematic diagram of a large language model defense restriction content provided by an embodiment of the present application;

[0041] Figure 2B is a schematic diagram of a sensitive word library provided in an embodiment of the present application;

[0042] Figure 2C It is a schematic diagram of matching sensitive words in a query sentence in a sensitive word library provided by an embodiment of the present application;

[0043] Figure 2D is a schematic diagram of feature splicing provided in an embodiment of the present application;

[0044] Figure 2E is a schematic diagram of a large language model embedding layer provided in an embodiment of the present application;

[0045] Figure 2F is a schematic diagram of an embedding layer fusion embedding feature provided in an embodiment of the present application;

[0046] Figure 3 It is a structural schematic diagram of a sentence text processing device provided in an embodiment of the present application;

[0047] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] As mentioned in the background technology, in response to the problems in the prior art, this application proposes a sentence text processing method. Figure 1 is a flowchart of a sentence text processing method provided by an embodiment of the present application. Figure 1 , the sentence text processing method specifically includes:

[0051] S110: Obtain an input query statement.

[0052] Among them, the input query statement can be any sentence that the user wants to input into the large language model, it can be a question that the large language model is expected to answer, or it can be a word that the large language model is expected to explain, etc. The embodiment of the present application does not limit the content and form of the input query statement.

[0053] S120: Input the query statement into the pre-trained semantic representation model to obtain a classification result of whether the query statement contains restrictive content.

[0054] Among them, the pre-trained semantic representation model can be any natural language processing model that can classify query statements according to the semantic information in the query statements. Of course, the embodiment of the present application aims to improve the defense capability of the large language model against the input restricted content. Therefore, before the query statement is input into the large language model for processing, the query statement is first input into the pre-trained semantic representation model to determine whether there is restricted content in the query statement. Then, in an embodiment of the present application, the semantic representation model can have the semantic recognition capability of restricted content. The classification results can correspond to two types: the query statement has restricted content and the query statement does not have restricted content, which can be understood as a binary classification of whether there is restricted content.

[0055] Of course, the pre-trained semantic representation model can use BERT (Bidirectional Encoder Representations from Transformers, a pre-trained semantic representation model based on Transformer). BERT uses a bidirectional attention mechanism and is naturally good at semantic encoding, so it has an advantage in semantic understanding tasks.

[0056] In addition, it should be explained that the restricted content can be any sensitive words or sentences in the query statement, for example, some malicious content, and sensitive words can include but are not limited to those about people's physical and mental health, privacy and property, ethics, discrimination and prejudice, illegal crimes and malicious attacks, etc. It should be solemnly stated that this application aims to improve the defense capability of the large language model against restricted content, and there is the possibility of using sensitive words for model training, but it is by no means to use restricted content. The technical solutions in the embodiments of this application and each implementation method comply with the law and do not violate public order and good customs.

[0057] S130. In response to the classification result that the query statement contains restricted content, the to-be-processed text corresponding to the query statement is input into a preset large language model to determine the hierarchical information of each text character in the to-be-processed text and the character embedding features corresponding to the to-be-processed text.

[0058] The text to be processed may be the input of the large language model. The large language model generates and outputs an answer result for the text to be processed. In the embodiment of the present application, there is no restriction on the large language model (LLM), which may be any type of large language model, and can be set by relevant technical personnel according to actual conditions and / or manual experience.

[0059] The text characters in the text to be processed can be the smallest units into which the text is divided before or during the process of being processed by the model, for example, they can be tokens in a large language model. The formation of these smallest units depends on the vocabulary and word segmentation strategy of the large language model, etc. The embodiments of the present application do not limit the vocabulary and word segmentation strategy of the large language model. The hierarchical information can be information representing the type to which the text characters belong. The types may include but are not limited to user instruction characters, large model reply content characters, and function call characters, etc. The embodiments of the present application do not list them all here. The hierarchical information can be identified by numbers. The hierarchical information corresponding to the user instruction characters is 1; the large model reply content characters are 2; the function call characters are 3, and so on. Each text character in the text to be processed is assigned a corresponding identification number according to its own hierarchical information, which is used for vector calculation in the large language model. This facilitates calculation and recovery of actual semantic information from the vector.

[0060] The character embedding feature can be the feature of the embedding layer that needs to be input into the large language model corresponding to the text to be processed, or it can be the set of vector representations of each character in the text to be processed in the large language model. Since the large language model converts the text into a vector according to the established encoding rules for processing in the process of processing text data, the character embedding feature corresponding to the text to be processed can be in the form of a vector. The text to be processed is divided into multiple characters, each of which is converted into a vector through an established encoding process, and the character embedding feature can be a set of vectors of these characters.

[0061] Specifically, when the classification result identified in the above steps determines that there are restricted contents in the query statement, it is necessary to process the restricted contents of the to-be-processed text corresponding to the query statement to prevent the large model from answering malicious messages to the user. The pre-processing step here is to first determine the hierarchical information of each text character and the character embedding features corresponding to the to-be-processed text, to provide support for preventing the large model from answering malicious messages in the subsequent steps.

[0062] S140: Determine the embedding layer features of the large language model according to the character embedding features and each level of information, and update the large language model according to the embedding layer features, so that the updated large language model outputs an answer result according to the query statement.

[0063] Among them, the embedding layer features of the large language model can be the features that need to be input into the embedding layer of the large language model when the large language model analyzes the processed text and outputs the answer result. The embedding layer helps the large language model understand the user's questions and related information based on these features, so as to provide more accurate answers.

[0064] It is understandable that the character embedding features correspond to the text to be processed, and each level of information represents the character type of each character. The character embedding features and the level information are vectorized to obtain the embedding layer features, thereby updating the embedding layer in the large language model, and using the large language model after the embedding layer is updated to answer the query questions raised by the user. It is understandable that the character type represented by the level information identifies the source and / or call relationship of each character to a certain extent, and therefore can identify the restricted content in the query statement for the large language model to a certain extent.

[0065] In the technical solution of the embodiment of the present application, by inputting the query statement into the pre-trained semantic representation model, a classification result of whether the query statement contains restricted content is obtained, and the semantic representation model is used to make a pre-judgment on whether the query statement contains restricted content, so as to help the large language model get an early warning before the query statement is input; after determining that there is restricted content in the query statement, the embedding layer features of the large language model are determined according to the hierarchical features of the text characters and the character embedding features of the text to be processed, so as to update the model, so that the updated large language model can output the answer result according to the query statement, and the restricted content can be processed in the model, so as to prevent the large language model from outputting the answer result with restricted content due to answering the question with restricted content. The technical solution of the embodiment of the present application performs early warning and defense of restricted content from both the input side and the model side of the large language model, improves the overall filtering effect of restricted content, improves the defense capability of the large language model against the input restricted content, and makes the output result of the large language model safer and harmless.

[0066] In an optional implementation, in response to the classification result that the query statement has restricted content, inputting the to-be-processed text corresponding to the query statement into a preset large language model in S130 may include:

[0067] In response to the classification result that the query sentence has restricted content, the query sentence is input into a preset prompt text generation model to obtain a prompt guide text of the query sentence, and the prompt guide text is input into the large language model as a text to be processed.

[0068] Among them, the preset prompt text generation model can be a model for generating text that prompts the restricted content in the query statement. When it is detected that there is restricted content in the query statement, before entering the large language model, it can be first entered into the prompt text generation model, and a prompt guide text including the query statement is generated based on the model. The prompt guide text can be an annotation and prompt for the query statement that may have restricted content. The prompt guide text is generated by the prompt text generation model according to the query statement, and the prompt guide text contains prompts for the restricted content that may exist in the query statement. The prompt text generation model can adopt any model or algorithm for generating prompt content in the relevant technology, for example, it can be a prompt, which can provide the large language model with input content related to the query statement, thereby guiding the large language model to generate a specific type of response. Prompt can be a question, a description, or a task description, etc., and the embodiments of the present application are not exhaustive.

[0069] Exemplarily, assuming that the input query statement is an instruction with restricted content, when the query statement is identified as having restricted content in the pre-step, the query statement is first input into the prompt text generation model to generate a prompt guide text including the query statement, for example:

[0070] “You are the world’s best model safety detection assistant, capable of identifying scenarios including: physical and mental health, privacy and property, ethics, prejudice and discrimination, illegal crimes, and political sensitivity.

[0071] Please provide a safe and positive response to the following instructions that have security issues based on the above dimensions. "

[0072] After the prompt guidance text is generated, it can be input into the large language model, so that the large language model not only inputs the query statement with restricted content, but also obtains the text with reminders for the restricted content, and generates the answer result with reference to the reminder text.

[0073] In the above implementation, a prompt text generation model is set on the input side of the large language model to provide prompt information for the large language model, so as to guide the large language model to output safe and harmless answer results rather than malicious results. This can play a preventive and early warning prompt role on the input side of the large language model, thereby improving the large language model's ability to prevent restricted content.

[0074] In another optional implementation, the method may further include:

[0075] In response to the classification result that the query sentence does not have any restrictive content, the query sentence is input into the large language model as a text to be processed.

[0076] It is understandable that when the pre-trained semantic representation model determines that there is no restrictive content in the query statement, there is no need to defend against the restrictive content, and there is no need to generate prompt guidance text in the aforementioned implementation method. The query statement can be directly input into the large language model as the text to be processed, and the large language model can generate the answer result based on the text.

[0077] In an optional implementation, determining the embedding layer features of the large language model according to the character embedding features and the information at each level in S140 may include:

[0078] S141. Determine the hierarchical embedding features of the prompt guidance text according to each hierarchical information.

[0079] Among them, the hierarchical embedding feature can be a feature of the model embedding layer corresponding to the prompt guide text. It can be understood that before the prompt guide text is introduced, the large language model only has a token embedding layer for the query statement. In order to better distinguish the hierarchy of the prompt guide text (such as the Prompt instruction), a prompt embedding layer for the prompt guide text can be added to the embedding layer of the large language model. The hierarchical information of the prompt embedding layer is the same as the hierarchical information of the prompt guide text, that is, the number of identification values ​​of the hierarchical information is the same as the number of types of characters in the prompt guide text. For example, the prompt guide text includes four types of system prompt characters (system), user instruction characters (user), large model reply content characters (assistant) and function call characters (function call), then there can be four corresponding hierarchical information, and the value range of the hierarchical information can be {0, 1, 2, 3}, and the four digital identifiers correspond to a character type respectively. Similar to the character embedding feature, the hierarchical information of each character in the prompt guide text is brought into the large language model for vector calculation to obtain the hierarchical embedding feature corresponding to the prompt guide text.

[0080] S142: Add the character embedding feature and the hierarchical embedding feature to obtain the embedding layer feature.

[0081] Since the character embedding features and the hierarchical embedding features belong to the same text to be processed, the dimensions of the character embedding features and the hierarchical embedding features are the same, and the two are added together to obtain the embedding layer features that are finally used for the embedding layer of the large language model.

[0082] It can be understood that by adding hierarchical embedding features on the basis of character embedding features, the hierarchical information identified in the prompt guidance text can be substituted into the embedding layer of the large language model, which can improve the judgment of the large language model on restricted content and further avoid the large language model from outputting malicious answers related to restricted content, thereby building a defense for the large language model on the model side.

[0083] In a further optional implementation, determining the hierarchical embedding features of the prompt guidance text according to each hierarchical information in S141 may include: multiplying the vector of hierarchical information corresponding to each text character in the prompt guidance text with a pre-constructed prompt segment embedding matrix to obtain the hierarchical embedding features. The prompt segment embedding matrix is ​​constructed according to the embedding layer identification value of the large language model and the character embedding dimension of the large language model.

[0084] Among them, the prompt segment embedding matrix can be an embedding layer matrix expression corresponding to the aforementioned hierarchical information in the large language model. In order to provide the hierarchical embedding features corresponding to the hierarchical information for the large language model, it is necessary to pre-construct the prompt segment embedding matrix. The prompt segment embedding matrix is ​​constructed according to the embedding layer identification value of the large language model and the character embedding dimension of the large language model. The embedding layer identification value can be the size of the embedding layer, and the character embedding dimension can be the dimension of the character embedding feature of the text to be processed. Exemplarily, the embedding layer size is S, and the dimension is consistent with the dimension of the character embedding feature, which is D. Pre-construct an S×D size matrix E P Continuing with the previous example, if the hierarchical information includes four types: system, user, assistant, and function call, then S=4. Assume that there are N tokens in the text to be processed X, and each token has corresponding hierarchical information S n , S n The value range is {0, 1, 2, 3}. In this example, it is assumed that X has only three types: system, user, and assistant. Then the level information S of the question X to be processed is n As shown in Table 1.

[0085] Table 1

[0086]

[0087] Among them, the level information of 0 corresponds to the token being of system type, and the level information of 2 corresponds to the token being of assistant type. The rest is the same as the previous example and will not be elaborated here.

[0088] The level information S n With the pre-built cue fragment embedding matrix E P Multiply them together to get the hierarchical embedding features corresponding to the hierarchical information (e 0 P , e 0 P ,…,e 2 P , e 2 P ).

[0089] The above implementation provides a feasible solution for determining the hierarchical embedding features corresponding to the hierarchical information, and provides a calculation method for multiplying the prompt fragment embedding matrix and the hierarchical information. The obtained hierarchical embedding features can be applied to the large language model in the aforementioned implementation, which helps to improve the large language model's defense capabilities against restricted content on the model side.

[0090] In another optional implementation, inputting the query statement into the pre-trained semantic representation model in S120 to obtain a classification result of whether the query statement contains restrictive content may include:

[0091] S121. Input the query statement into the pre-trained semantic representation model to obtain the sentence features corresponding to the query statement.

[0092] The sentence feature may be a semantic feature vector corresponding to the query sentence obtained after the query sentence is input into the semantic representation model, and the sentence feature may represent the semantics of the query sentence. It is understandable that the semantic representation model has the ability to encode the input query sentence, and according to the established encoding logic, the query sentence may be encoded into a corresponding feature vector, i.e., a sentence feature.

[0093] S122: Match at least one sensitive word entity appearing in the query statement in a preset sensitive word library, and determine entity features of each sensitive word entity.

[0094] The sensitive word library may be a collection of pre-collected sensitive words, which may include but not be limited to sensitive words in different scenarios such as physical and mental health, privacy and property, ethics, discrimination and prejudice, illegal crimes, malicious attacks, etc. The sensitive word entity may be a sensitive word that appears in the query statement, and the entity feature is the semantic feature vector corresponding to the sensitive word entity.

[0095] Specifically, through a pre-set matching algorithm, the words in the query sentence are compared with the words in the sensitive word library to determine which words in the query sentence are sensitive words, and these sensitive words are used as sensitive word entities and converted into feature vectors, i.e., entity features, through the encoding of the semantic representation model. Among them, the pre-set matching algorithm can adopt any word matching algorithm in the relevant technology, for example, the multi-mode AC algorithm (Aho-Corasick, AC automaton algorithm) can be adopted, which is a string matching algorithm for multi-mode matching, and each string in the query sentence is verified with the sensitive word library to determine which words in the query sentence are sensitive words.

[0096] S123. Calculate the average value of each entity feature to obtain an enhanced feature.

[0097] There may be more than one sensitive word in the query statement. Each sensitive word entity can generate corresponding entity features through the semantic representation model. Since these entity features are vectors of the same dimension, addition, subtraction, multiplication and division operations can be performed. The average value of the entity features corresponding to all sensitive words appearing in the query statement is calculated and used as the enhanced feature.

[0098] S124. Determine a classification result based on the sentence features and the enhanced features.

[0099] Sentence features can reflect the semantics of query sentences, and enhanced features can improve the recognition of sensitive words in query sentences. Based on these two types of features, it is judged whether there are sensitive words in the query sentence, and the binary classification results are output. Of course, sentence features and enhanced features can be processed by binary classification through linear layers in the semantic representation model, or sentence features and enhanced features can be fused first, and then output classification through the linear layer of the semantic representation model.

[0100] In a further optional implementation, the dimensions of the sentence vector corresponding to the sentence feature and the enhancement vector corresponding to the enhancement feature are different;

[0101] Determining the classification result according to the sentence features and the enhanced features in S124 may include: concatenating the sentence vector and the enhanced vector to determine a fusion feature; and determining the classification result based on the fusion feature.

[0102] Among them, the fused feature can be the concatenation result of the sentence vector and the enhanced vector. For example, BERT is a semantic representation model, and its output layer fuses the entity features of sensitive words before performing binary classification. Assuming that the dimension of the sentence vector determined by the output layer of BERT is 768 dimensions and the entity feature is 64 dimensions, the fused vector obtained by direct concatenation is 832 dimensions. Then the fused feature is output and classified through the linear layer in BERT.

[0103] In the above two implementations, how to classify and judge whether a query statement contains sensitive words and other restricted content in a semantic representation model is explained. By determining the entity features of the sensitive word entity, enhanced features are further obtained. The fusion features after concatenating sentence features and enhanced features are used to improve the accuracy of identifying restricted content, thereby providing assistance for malicious defense of the large language model on the input side.

[0104] Figure 2A It is a schematic diagram of a large language model defense restriction content provided by an embodiment of the present application. Based on the above embodiment and various implementation methods, the embodiment of the present application also provides a preferred example as a comprehensive example and / or supplement to the above implementation methods, as follows.

[0105] When defending the security of large language models, we use input-side defense and model-level structural adjustment defense. Input-side defense mainly uses the BERT classification model to classify and judge the input user instructions. Model-level structural adjustment integrates PSE (Prompt Segment Embedding) hierarchical information to defend against implicit input, jailbreak prompts, indirect injection and other restricted content questioning methods on the model side. The overall defense structure is as follows: Figure 2A The following describes the defenses on the input side and the model side respectively.

[0106] Before judging whether the query statement contains restricted content, build a sensitive word library. According to the large language model security governance specification, the main scenarios involving security are as follows: physical and mental health, privacy and property, ethics, discrimination and prejudice, illegal crimes, malicious attacks, etc. Collect and annotate relevant data according to these six scenarios as positive samples for security detection. At the same time, collect sensitive word libraries related to these six scenarios for subsequent semantic enhancement. Collect user instructions of open source SFT (Supervised Fine-Tuning) data as negative samples for training. There are fewer instructions involving security issues in real usage scenarios, and they usually account for a very small proportion. In order to align the data distribution of real scenarios as much as possible, the ratio of the number of positive and negative samples can be 1:99.

[0107] BERT uses the encoder part of the Transformer structure and the bidirectional attention mechanism, which naturally has strong semantic encoding capabilities and is therefore advantageous in semantic understanding tasks.

[0108] The collected positive and negative security detection samples are mixed as training samples. Therefore, using BERT to determine whether there is restricted content in the user's query statement is essentially a binary classification model. The output label 1 indicates that the query statement is unsafe and has restricted content, and the label 0 indicates that it is safe.

[0109] Encode the sensitive word library constructed above. Assume that the number of sensitive words in the sensitive word library is 2001, numbered 0 to 2000, and the encoding length is 64 dimensions. The size of the word library in the actual usage scenario is subject to the specific situation. Figure 2B shown.

[0110] Based on this collected and annotated sensitive word library, the multi-mode AC matching algorithm is used to extract sensitive word entities in the classified data. For example, a sensitive word in the sentence input by the user is numbered 3 in the sensitive word library. The entity feature of the input can be obtained as e3. Correspondingly, if the AC matching algorithm hits multiple entities in a query statement: W 1 , W 2 …W n, we can get the enhanced features of each entity in the query statement as (e1+e2+e3) / 3, that is, the embedding features of each word are averaged, such as Figure 2C shown.

[0111] BERT has limited encoding capabilities for user input query statements. In order to enhance the encoding of sensitive words, the sensitive word entity features are integrated in the output layer of BERT before classification. For example, the output layer of BERT is 768-dimensional, and the above entity features are 64-dimensional. The two are concatenated into an 832-dimensional vector through the concatenation function (concat). After a linear layer, the output classification is as follows: Figure 2D As shown, where X 1 , X 2 …X n-1 and X n The input text for BERT.

[0112] It should be noted that there is a problem with the data in the sensitive word library, that is, the distribution of positive and negative samples is extremely unbalanced. To solve this problem, the focal loss function is used. Focal loss is a dynamically balanced cross loss. With the help of scaling parameters, it can dynamically reduce the weight of easy-to-separate samples during training, thereby focusing on difficult-to-separate samples. Its formula is as follows:

[0113]

[0114] Among them, FL is the loss of focal loss, α t is the weight coefficient of positive and negative samples, which is used to adjust the balance of government sample data volume, and γ is the balance coefficient of difficult and easy classification samples, which can control the balance of easy and difficult classification samples. t is the probability of a positive sample, 1-p t is the probability of a negative sample. Focal loss is a new loss function for unbalanced datasets, which mainly alleviates the problem of class imbalance by introducing adjustable parameters. In practical applications, focal loss can effectively improve the performance of the model, especially in dealing with class imbalance and difficult-to-classify scenarios.

[0115] In the field of computer science and NLP (Natural Language Processing), prompts refer to input information or instructions provided to a computer program or model. In large language models, prompts are questions or statements given by users to the model to guide the model to generate relevant responses. After receiving the prompt, the model will generate content related to the prompt based on the internal training knowledge and algorithms. Prompt injection is a means of manipulating large models to output toxic or leaked information by using malicious instructions as part of the input prompt. Prompt injection can be roughly divided into direct injection, prompt leakage, jailbreak attack and indirect prompt injection.

[0116] Among them, direct injection is to directly input toxic or harmful content into the instructions to guide the model to reply.

[0117] Cue leakage is the attempt to manipulate the model output to make it leak the original cue word, which may lead to the exposure of sensitive information.

[0118] Jailbreak attack is that the large language model is trained on a large amount of text data, learning various patterns and associations, so that it can generate coherent and logical text, but it does not have the ability to understand and reason in the true sense. Jailbreak is to design prompt words to guide or control the large language model to generate non-compliant content.

[0119] Indirect prompt injection is to hide malicious instructions through documents, web pages or code completion, bypass detection, and implicitly trigger attacks.

[0120] On the input side, there are BERT model pre-measures to detect malicious and toxic inputs of the model, thus achieving a certain degree of defense. However, BERT may miss the detection of obscure inputs.

[0121] When BERT classifies and judges user input, if BERT judges that the user input is safe, no processing is required. In existing methods, if there is a problem with the user input detection, an error is directly returned and it is not directly input to the large model. The method of this solution is that if BERT judges the user input as unsafe input, the prompt word (i.e. the prompt guide text in the above implementation) is used to guide the model to make a safe response instead of directly discarding it.

[0122] The main approach is to generate a prompt guidance text to tell the large language model the unsafe content classification, and then tell the model that the current user input may contain restricted content, and then guide the model to give an evasive answer to the current unsafe input.

[0123] Using the examples of the above embodiments and implementation methods, it is assumed that the query statement input by the user is an instruction with restricted content. When it is identified in the pre-step that the query statement has restricted content, the query statement is first input into the prompt text generation model to generate a prompt guide text including the query statement, for example:

[0124] “You are the world’s best model safety detection assistant, capable of identifying scenarios including: physical and mental health, privacy and property, ethics, prejudice and discrimination, illegal crimes, and political sensitivity.

[0125] Please provide a safe and positive response to the following instructions that have security issues based on the above dimensions. "

[0126] On the large language model side, we borrowed the idea of ​​BERT’s embedding layer and added a PSE embedding information, which encodes the instruction hierarchy directly into the large language model input. This enables the subsequent self-attention layer to more effectively identify and follow instruction priorities, thereby improving security and robustness. The current embedding layer of the large language model only has a token embedding layer, and its structure is as follows: Figure 2E As shown in the figure, token embeddings are text character embeddings, and E0~E8 are the corresponding embedding features of system, user, and assistant in the input text in the embedding layer of the model. System, user, and assistant in the input text represent the system prompt token, user instruction token, and large model reply content token, respectively.

[0127] In order to better distinguish the levels of prompt instructions, a PSE embedding layer is added to the large model. In addition to system, user, and assistant, the current prompt instruction types may have a function call part in some cases. Therefore, the PSE embedding layer is first constructed according to the general scenario. The embedding layer size is S, and the dimension is the same as the token embedding dimension of the large model, which is D.

[0128] First, construct an S×D size matrix E P , let S=4 represent the four character types: system, user, assistant, and function call.

[0129] Assume that the text to be processed X has N tokens, and each token has corresponding hierarchical information S n , the value range is {0,1,2,3}. S n The results are shown in Table 1 above. According to the level information S n and the PSE matrix E PMultiplying the hierarchical embedding features is (e 0 P , e 0 P ,…,e 2 P , e 2 P ).

[0130] The text to be processed X has N tokens, so the character embedding feature of the token of the text to be processed is (e 0 T , e 1 T ,…,e n-1 T , e n T ).

[0131] The hierarchical embedding features of the PSE layer and the character embedding features of the question to be processed are added to obtain the final embedding layer features (e 0 P +e 0 T , e 0 P +e 1 T ,…,e 2 P +e n-1 T , e 2 P +e n T ), the input structure of the fused PSE embedding layer is as follows Figure 2F As shown in the figure, E0~E8 are the corresponding embedded features of system, user and assistant in the input text in each embedding layer of the model. By using the embedding layer features, the large language model has enhanced its defense capability against restricted content on the model side, improving the security and robustness of the large language model, effectively solving the problem of poor defense capability in traditional large model security defense methods, and improving the security specification capability of the large model.

[0132] Figure 3 300 includes a query sentence acquisition module 310, a classification result determination module 320, a restricted content processing module 330 and an answer result output module 340, as follows.

[0133] A query statement acquisition module 310 is used to acquire an input query statement;

[0134] A classification result determination module 320 is used to input the query statement into the pre-trained semantic representation model to obtain a classification result of whether the query statement contains restricted content;

[0135] The restricted content processing module 330 is used for inputting the to-be-processed text corresponding to the query sentence into a preset large language model in response to the classification result that the query sentence has restricted content, and determining the hierarchical information of each text character in the to-be-processed text and the character embedding features corresponding to the to-be-processed text;

[0136] The answer result output module 340 is used to determine the embedding layer features of the large language model according to the character embedding features and each level of information, and update the large language model according to the embedding layer features, so that the updated large language model outputs the answer result according to the query statement.

[0137] In the technical solution of the embodiment of the present application, by inputting the query statement into the pre-trained semantic representation model, a classification result of whether the query statement contains restricted content is obtained, and the semantic representation model is used to make a pre-judgment on whether the query statement contains restricted content, so as to help the large language model get an early warning before the query statement is input; after determining that there is restricted content in the query statement, the embedding layer features of the large language model are determined according to the hierarchical features of the text characters and the character embedding features of the text to be processed, so as to update the model, so that the updated large language model can output the answer result according to the query statement, and the restricted content can be processed in the model, so as to prevent the large language model from outputting the answer result with restricted content due to answering the question with restricted content. The technical solution of the embodiment of the present application performs early warning and defense of restricted content from both the input side and the model side of the large language model, improves the overall filtering effect of restricted content, improves the defense capability of the large language model against the input restricted content, and makes the output result of the large language model safer and harmless.

[0138] Based on the above implementations, optionally, the restricted content processing module 330 may include:

[0139] The text input unit is used to input the query statement into a preset prompt text generation model in response to the classification result that the query statement has restricted content, obtain the prompt guide text of the query statement, and input the prompt guide text as the text to be processed into the large language model.

[0140] Based on the above implementations, optionally, the answer result output module 340 may include:

[0141] A hierarchical embedding feature determination unit, used to determine the hierarchical embedding features of the prompt guidance text according to each hierarchical information;

[0142] The embedding layer feature determination unit is used to add the character embedding feature and the hierarchical embedding feature to obtain the embedding layer feature.

[0143] Based on the above implementations, optionally, the hierarchical embedding feature determination unit may be specifically used to:

[0144] Multiply the vector of the hierarchical information corresponding to each text character in the prompt guidance text with the pre-constructed prompt segment embedding matrix to obtain the hierarchical embedding feature;

[0145] Among them, the prompt fragment embedding matrix is ​​constructed according to the embedding layer identification value of the large language model and the character embedding dimension of the large language model.

[0146] Based on the above implementations, optionally, the classification result determination module 320 may include:

[0147] A sentence feature determination unit, used for inputting a query sentence into a pre-trained semantic representation model to obtain a sentence feature corresponding to the query sentence;

[0148] An entity feature determination unit, used to match at least one sensitive word entity appearing in the query sentence in a preset sensitive word library, and determine the entity features of each sensitive word entity;

[0149] An enhanced feature determination unit, used for averaging the features of each entity to obtain an enhanced feature;

[0150] The classification result determination unit is used to determine the classification result according to the sentence features and the enhanced features.

[0151] Based on the above implementations, optionally, the dimensions of the sentence vector corresponding to the sentence feature and the enhancement vector corresponding to the enhancement feature are different, and the classification result determination unit may include:

[0152] The feature concatenation subunit is used to concatenate the sentence vector and the enhancement vector to determine the fusion feature;

[0153] The fusion feature classification subunit is used to determine the classification result based on the fusion feature.

[0154] Based on the above embodiments, optionally, the device 300 may further include:

[0155] The non-restricted content processing module is used to input the query sentence as a text to be processed into the large language model in response to the classification result that the query sentence does not have any restricted content.

[0156] The sentence text processing device provided in the embodiment of the present application can execute the steps in the sentence text processing method provided in the method embodiment of the present application, and has the ability to execute each step and its beneficial effects, which will not be repeated here.

[0157] Figure 4 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0158] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0159] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the sentence text processing method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0160] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 403 may include, for example, a keyboard, a mouse, etc. The output device 404 may output various information to the outside, including early warning prompt information, braking force, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0161] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 400 may also include any other appropriate components.

[0162] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the sentence text processing method provided by any embodiment of the present application.

[0163] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0164] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the sentence text processing method provided by any embodiment of the present application.

[0165] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0166] It should be noted that the terms used in this application are only for describing specific embodiments, rather than limiting the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0167] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0168] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make several improvements, modifications or changes without departing from the principles of this application, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the invention to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A sentence text processing method, characterized in that: The method comprises: Get the input query statement; Inputting the query statement into a pre-trained semantic representation model to obtain a classification result of whether the query statement contains restrictive content; In response to the classification result that the query statement has restricted content, inputting the to-be-processed text corresponding to the query statement into a preset large language model to determine the hierarchical information of each text character in the to-be-processed text and the character embedding features corresponding to the to-be-processed text; Determining an embedding layer feature of the large language model according to the character embedding feature and each of the hierarchical information, and updating the large language model according to the embedding layer feature, so that the updated large language model outputs an answer result according to the query statement; In response to the classification result that the query statement has restricted content, inputting the to-be-processed text corresponding to the query statement into a preset large language model includes: In response to the classification result that the query statement has restricted content, the query statement is input into a preset prompt text generation model to obtain a prompt guide text for the query statement, and the prompt guide text is input into the large language model as a text to be processed.

2. The method according to claim 1, characterized in that Determining the embedding layer features of the large language model according to the character embedding features and each of the hierarchical information includes: Determining the hierarchical embedding features of the prompt guiding text according to each of the hierarchical information; The character embedding feature and the level embedding feature are added to obtain the embedding layer feature.

3. The method according to claim 2, characterized in that Determining the hierarchical embedding features of the prompt guiding text according to each of the hierarchical information includes: Multiplying the vector of the hierarchical information corresponding to each text character in the prompt guiding text with the pre-constructed prompt segment embedding matrix to obtain the hierarchical embedding feature; The prompt segment embedding matrix is ​​constructed according to the embedding layer identification value of the large language model and the character embedding dimension of the large language model.

4. The method according to any one of claims 1 to 3, characterized in that: The step of inputting the query statement into a pre-trained semantic representation model to obtain a classification result of whether the query statement contains restrictive content includes: Inputting the query statement into a pre-trained semantic representation model to obtain a statement feature corresponding to the query statement; Matching at least one sensitive word entity appearing in the query statement in a preset sensitive word library, and determining entity features of each of the sensitive word entities; averaging the entity features to obtain enhanced features; The classification result is determined according to the sentence feature and the enhanced feature.

5. The method according to claim 4, characterized in that The sentence vector corresponding to the sentence feature and the enhancement vector corresponding to the enhancement feature have different dimensions, and determining the classification result according to the sentence feature and the enhancement feature includes: Concatenate the sentence vector and the enhanced vector to determine a fusion feature; Based on the fused features, the classification result is determined.

6. The method according to claim 1, characterized in that The method further comprises: In response to the classification result that the query statement does not have any restrictive content, the query statement is input into the large language model as a text to be processed.

7. A sentence text processing device, characterized in that: The device comprises: A query statement acquisition module is used to acquire an input query statement; A classification result determination module, used for inputting the query statement into a pre-trained semantic representation model to obtain a classification result of whether the query statement contains restrictive content; A restricted content processing module, for, in response to the classification result indicating that the query statement has restricted content, inputting the query statement into a preset prompt text generation model to obtain a prompt guide text of the query statement, inputting the prompt guide text as a text to be processed into a large language model, and determining the hierarchical information of each text character in the text to be processed and the character embedding features corresponding to the text to be processed; The answer result output module is used to determine the embedding layer features of the large language model according to the character embedding features and each of the hierarchical information, and update the large language model according to the embedding layer features, so that the updated large language model outputs the answer result according to the query statement.

8. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the steps of the statement text processing method according to any one of claims 1 to 6 by calling the program or instruction stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the sentence text processing method according to any one of claims 1 to 6.

Images