Method and device for large model security defense and electronic equipment

By classifying and security evaluation of the input content of the large model, and combining filtering and evaluation models to adjust its processing capabilities, the problem of insufficient security in the face of harmful inputs is solved, and a safer output is achieved.

CN118940276APending Publication Date: 2024-11-12CHINA ACADEMY OF INFORMATION & COMM
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410952369.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Large models lack security reinforcement capabilities when facing improper or harmful inputs, may output or generate responses containing harmful content as they are, and lack the ability to process harmful information.

Method used

By classifying and preprocessing the input content, using a filtering model for text feature recognition and classification, combining the security evaluation model to evaluate the input and output pairs, and adjusting the large model based on the unsafe input and output pairs, introducing low-rank matrix and external knowledge library to improve its processing power.

Benefits of technology

It enhances the processing capability of large models for harmful information, reduces the situation of unsafe output, improves the processing process of unsafe inputs, and ensures that the output is safer.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940276A_ABST
    Figure CN118940276A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and discloses a method and device for large model security defense and electronic equipment, and the method comprises the steps: classifying input contents, and obtaining target input; obtaining a target output according to the target input through the target large model, and obtaining a target input-output pair according to the target input and the target output; performing safety evaluation on the target input and output pair to obtain an unsafe input and output pair; and adjusting the target large model according to the unsafe input and output pair to obtain the adjusted target large model. According to the technical scheme of the invention, security evaluation is carried out on the target input and output pairs, unsafe input and output pairs in the target input and output pairs can be determined, and the large model is adjusted according to the unsafe input and output pairs, so that the processing process of the large model on unsafe input can be improved, and the situation of outputting unsafe output can be reduced. Therefore, the harmful information processing capability of the large model is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and for example, to a method and device, and electronic equipment for large model security defense. Background Art

[0002] Currently, with the rapid development of artificial intelligence technology, large models have become an important tool for processing and analyzing large amounts of data. These models are usually based on deep learning algorithms and are able to perform complex tasks such as natural language processing, image recognition, and predictive analysis. However, although large models have demonstrated excellent performance in many fields, they have some inherent flaws in terms of security.

[0003] In related technologies, large models are designed to learn and simulate patterns in data, but lack the ability to conduct ethical and legal review of their inputs.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] Large models lack security reinforcement capabilities when faced with many inappropriate or harmful content. After receiving harmful input, they will output it as is or generate a response containing harmful content, and lack the ability to process harmful information.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a method, device, and electronic device for large model security defense to enhance the large model's ability to process harmful information.

[0009] In some embodiments, a method for large model security defense includes: classifying input content to obtain a target input; obtaining a target output based on the target input through a target large model, and obtaining a target input-output pair based on the target input and the target output; performing a security assessment on the target input-output pair to obtain an unsafe input-output pair; adjusting the target large model based on the unsafe input-output pair to obtain an adjusted target large model.

[0010] Optionally, classifying the input content includes: preprocessing the input content; performing text feature recognition on the preprocessed input content based on a trained filtering model, and classifying the input content according to the text feature.

[0011] Optionally, the filtering model is trained in the following manner: the filtering model is iteratively trained one or more times; the iterative training includes: calculating the gradient of the loss function with respect to the parameters of the filtering model based on the training data; updating the parameters of the filtering model based on the gradient; and performing a performance evaluation on the filtering model after the parameter update based on the verification data; wherein the convergence conditions of the iterative training include: the number of iterative training times reaches a preset number or the performance evaluation result meets a preset condition.

[0012] Optionally, performing a security assessment on the target input-output pair includes: obtaining an assessment model for security assessment; setting prompt words and examples for the assessment model; and scoring the target input-output pair based on the prompt words and examples based on the assessment model.

[0013] Optionally, the evaluation model is trained according to the following method: embedding evaluation criteria in the prompt words of the evaluation model; the evaluation criteria include security evaluation standards for input-output pairs, and the format in which the evaluation model receives inputs and generates outputs; obtaining manually annotated scores for training input-output pairs under the evaluation criteria; training the evaluation model based on the training input-output pairs and manually annotated scores to obtain a trained evaluation model.

[0014] Optionally, the target large model is adjusted according to the unsafe input-output pairs, including: rewriting the unsafe input-output pairs to obtain input-output pair samples; introducing a trainable low-rank matrix into the target large model to modify the weights of the target large model; training the low-rank matrix according to the input-output pair samples to adjust the weights of the target large model to obtain the adjusted target large model.

[0015] Optionally, the target output is obtained according to the target input through the target large model, including: obtaining the initial output of the target large model according to the target input; collecting security knowledge through an external knowledge base to establish a security knowledge base; based on the security knowledge base, adjusting the initial output to obtain the target output.

[0016] Optionally, the initial output is adjusted to obtain a target output, including: obtaining target security knowledge matching the initial output in a security knowledge base; and performing fusion reasoning based on the target security knowledge and the initial output to obtain the target output.

[0017] In some embodiments, the apparatus for large model security defense includes a processor and a memory storing program instructions, and the processor is configured to execute the method for large model security defense as described above when running the program instructions.

[0018] In some embodiments, the electronic device includes: an electronic device body, which is provided with a large model; and a device for large model security defense as described above, which is installed on the electronic device body.

[0019] The method, device, and electronic device for large model security defense provided by the embodiments of the present disclosure can achieve the following technical effects:

[0020] In the disclosed embodiment, the input content of the large model is classified to obtain the target input, the target input is processed by the target large model to obtain the target output, and a target input-output pair is formed. By performing a security assessment on the target input-output pair, an unsafe input-output pair in the target input-output pair can be determined. The unsafe input-output pair reflects that the target large model may still output an unsafe output when processing the unsafe input. Therefore, according to the unsafe input-output pair, the large model can be adjusted to improve the large model's processing process of unsafe inputs, reduce the situation of outputting unsafe outputs, and thus enhance the large model's ability to handle harmful information.

[0021] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0023] Figure 1 is a schematic diagram of a method for large model security defense provided by an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of a framework of a large model provided by an embodiment of the present disclosure;

[0025] Figure 3 It is a schematic diagram of a data processing flow of a large model provided by an embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram of a LoRA fine-tuning method provided by an embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of another method for large model security defense provided by an embodiment of the present disclosure;

[0028] Figure 6 It is a schematic diagram of a device for large model security defense provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0030] The terms "first", "second", etc. in the technical solutions described in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the disclosed embodiments described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0031] Unless otherwise stated, the term "plurality" means two or more.

[0032] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0033] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0034] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0035] Combination Figure 1 As shown, the embodiment of the present disclosure provides a method for large model security defense, the execution subject of the method may be a processor, and the method includes:

[0036] S001, the processor classifies the input content to obtain the target input.

[0037] S002, the processor obtains a target output according to a target input through a target large model, and obtains a target input-output pair according to the target input and the target output.

[0038] S003, the processor performs a security assessment on the target input-output pair to obtain an unsafe input-output pair.

[0039] S004, the processor adjusts the target large model according to the unsafe input-output pair to obtain an adjusted target large model.

[0040] In the disclosed embodiment, the input content of the large model is classified to obtain the target input, the target input is processed by the target large model to obtain the target output, and a target input-output pair is formed. By performing a security assessment on the target input-output pair, an unsafe input-output pair in the target input-output pair can be determined. The unsafe input-output pair reflects that the target large model may still output an unsafe output when processing the unsafe input. Therefore, according to the unsafe input-output pair, the large model can be adjusted to improve the large model's processing process of unsafe inputs, reduce the situation of outputting unsafe outputs, and thus enhance the large model's ability to handle harmful information.

[0041] Combination Figure 2 As shown, the embodiment of the present disclosure provides a framework of a large model, including a filtering model, an initial large model, a target large model and an evaluation model. Figure 3 As shown, the processor can filter and classify the input content of the large model through the filtering model, and divide the input content into safe content and harmful content by judging the harmfulness of the input text. Among them, safe content includes text, images, videos or other forms of data that do not violate laws and regulations, do not contain harmful information, do not have a negative impact on society, and are suitable for public dissemination; harmful content includes information that may cause harm to personal, social or national security. Harmful content may violate laws and regulations, spread false information, contain discrimination or hate speech, infringe personal privacy, or otherwise threaten social order and public interests. The initial large model processes the safe content and generates a safe output. The target large model is obtained by fine-tuning the initial model. The target large model can be used to process harmful content and generate a target output. The security of the target output is evaluated by the evaluation model, and the unsafe input-output pair can be determined according to the evaluation result. According to the unsafe input-output pair, the target large model can be adjusted multiple times to further enhance the target large model's ability to handle unsafe inputs. In addition, when processing harmful information through the target big model, you can also borrow external knowledge bases to carry out more stringent defense reinforcement of the target big model based on real-time updated laws and regulations, new security incidents and other information, so that the generated output is more accurate and secure.

[0042] Optionally, classifying the input content includes: preprocessing the input content; performing text feature recognition on the preprocessed input content based on a trained filtering model, and classifying the input content according to the text feature.

[0043] In this embodiment, the model is classified, and the input content needs to be pre-processed by data cleaning, word segmentation, standardization, etc. The pre-processed input content is sent to the filtering model to perform the forward propagation process. During the forward propagation process, the input content will be processed by multiple layers of the filtering model in sequence. These layers each perform specific computing tasks, including the product operation of weights and input content, the addition of bias, and the application of nonlinear activation functions, so as to gradually extract and transform the features of the input content. As the input content flows from the input layer to the output layer, the model gradually builds a high-level understanding of the input content based on the feature weights and patterns learned during training. Finally, the output features are binary classified through the linear layer, accurately divided into two categories of "harmful" or "harmless", and the harmful content is identified as the target input. This intelligent classification process is crucial to maintaining the security and health of the network environment, helping to automatically identify and prevent the spread of potential bad information.

[0044] Optionally, the filtering model is trained in the following manner: the filtering model is iteratively trained one or more times; the iterative training includes: calculating the gradient of the loss function with respect to the parameters of the filtering model based on the training data; updating the parameters of the filtering model based on the gradient; and performing a performance evaluation on the filtering model after the parameter update based on the verification data; wherein the convergence conditions of the iterative training include: the number of iterative training times reaches a preset number or the performance evaluation result meets a preset condition.

[0045] In this embodiment, the training data is the input text used for iterative training, and the purpose of iterative training is to enable the filtering model to classify the input text into safe content and harmful content by training the filtering model. The input text can be expressed as:

[0046] S = {w1, w2, ..., w n}

[0047] Among them, S represents the input text, w i Represents the i-th word in the input text.

[0048] The training process of the filtering model first needs to be based on BERT (Bidirectional Encoder Representations from Transformers), output text features according to S, and divide the input text into safe content and harmful content through a linear layer according to the text features. d is a d-dimensional linear layer, which can be obtained:

[0049] z=Wx+b

[0050] Among them, W represents the weight matrix, b represents the bias term, and z represents the original output of the filtering model. In order to obtain a binary classification decision from the output of the linear layer, the activation function Sigmoid function is used to map any real number to the interval (0, 1) to solve the binary classification problem, which can be expressed as:

[0051]

[0052] Among them, σ represents the Sigmoid function, represents the predicted probability that the input x belongs to class 1.

[0053] By calculating the gradient of the loss function with respect to the model parameters, and using the gradient to update the model parameters. In this process, the calculation of the gradient involves calculating the partial derivative of the loss function to determine the update direction and magnitude of each parameter. The back propagation algorithm is used to efficiently calculate these gradients. The loss function is:

[0054]

[0055] Among them, y represents the true label.

[0056] The gradient of the loss function with respect to the weights and biases is then calculated, and the model parameters are updated:

[0057]

[0058]

[0059] Among them, η represents the learning rate.

[0060] During the training process of the filter model, the parameters of the filter model are updated through multiple training cycles. Each iteration uses the training data to calculate the loss and gradient, and then updates the weights and biases. Finally, the performance of the filter model is evaluated on independent validation data, including using accuracy, precision, recall, and F1 score.

[0061] Optionally, performing a security assessment on the target input-output pair includes: obtaining an assessment model for security assessment; setting prompt words and examples for the assessment model; and scoring the target input-output pair based on the prompt words and examples based on the assessment model.

[0062] Prompt words are a set of keywords or phrases used to guide the assessment model in performing security assessments. Possible prompt words include: comply with laws and regulations, avoid discriminatory speech, ensure the authenticity of information, protect personal privacy, or prevent copyright infringement. Examples are specific input-output pairs that are used to show how to apply prompt words and assessment criteria. Examples can help the assessment model understand the assessment process and provide specific application scenarios for the assessment criteria. Examples should clearly show the data structure of the input-output pair, as well as the output form of the score and the reason for the score. Guided by the prompt words and examples, the assessment model will automatically score each input-output pair. The scoring criteria are based on the preset assessment criteria. If the assessment model determines that the input-output pair has a security risk, the score is 1; if it determines that there is no risk, the score is 0.

[0063] Optionally, the evaluation model is trained according to the following method: embedding evaluation criteria in the prompt words of the evaluation model; the evaluation criteria include security evaluation standards for input-output pairs, and the format in which the evaluation model receives inputs and generates outputs; obtaining manually annotated scores for training input-output pairs under the evaluation criteria; training the evaluation model based on the training input-output pairs and manually annotated scores to obtain a trained evaluation model.

[0064] Under the guidance of the evaluation criteria, the input-output pairs are manually labeled and scored. The input-output pairs are used as new training data input, and the labeled scores are used as training data output to train the evaluation model and achieve fine-tuning of the evaluation model.

[0065] Optionally, the target large model is adjusted according to the unsafe input-output pairs, including: rewriting the unsafe input-output pairs to obtain input-output pair samples; introducing a trainable low-rank matrix into the target large model to modify the weights of the target large model; training the low-rank matrix according to the input-output pair samples to adjust the weights of the target large model to obtain the adjusted target large model.

[0066] In this embodiment, the unsafe input-output pair is rewritten to obtain the rewritten unsafe input and the forward-guided safe output, and the target large model is adjusted according to the rewritten unsafe input and the safe output, so that the target large model can output a safe output when processing the unsafe input. The training process of the target large model is based on the LoRA (Low-Rank Adaptation) fine-tuning method, and a trainable low-rank matrix is ​​introduced into the target large model. By training the low-rank matrix, the weight of the target large model can be adjusted.

[0067] To rewrite the unsafe input-output pairs, we need to first determine the input content with a score of 0 as an unsafe input-output pair based on the output of the evaluation model, and represent the unsafe input-output pair as S ′= [IO], where S ′ Represents an unsafe input-output pair, I represents unsafe input, and O represents unsafe output. Under the premise of ensuring semantic consistency, the unsafe input is rewritten using the model to obtain N groups of input samples in, represents the rewritten unsafe input sample set, I i Represents the rewritten i-th group of unsafe input samples. Then the unsafe output is rewritten into M groups of safe output samples through the template in, represents the rewritten set of secure output samples, O i Represents the i-th group of secure output samples after rewriting. By combining the rewritten output samples with the input samples, N×M groups of input-output pair samples can be obtained.

[0068] Based on the above-mentioned integrated N×M groups of input-output pair samples, the LoRA fine-tuning method can be used to adjust the target large model in an efficient and low-cost manner without retraining the parameters of the target large model to enhance its security response capability. The LoRA fine-tuning method allows the target large model to adapt to specific tasks by optimizing a small number of parameters while keeping the pre-trained weights basically unchanged, significantly reducing the required storage space and computing resources. LoRA improves training efficiency by simplifying the model adjustment process, while reducing resource consumption during model deployment and runtime, making it more efficient and practical when processing high-dimensional data and complex tasks. LoRA indirectly trains some dense layers in the neural network by optimizing the rank decomposition matrix that changes during the adaptation process of the dense layer, while keeping the pre-trained weights unchanged.

[0069] Combination Figure 4 As shown, in LoRA, given the pre-trained weight matrix W0, by introducing the low-rank matrix A = N(0, σ 2 ) and B = 0 to update the weights of the target large model. The forward propagation process of LoRA in the neural network layer can be expressed as:

[0070] h=W0x+ΔWx=W0x+BAx

[0071] Among them, ΔW = BA, which represents a low-rank update to W0, and A and B are matrices with smaller dimensions. The input x is transformed through the original weights and the updated weights to produce the output h. Therefore, the objective function of training using the LoRA method is expressed as:

[0072]

[0073] Instead of updating all parameters Φ, by adjusting a small number of parameters Θ, these parameters indirectly affect Φ through ΔΦ, Φ0 represents the parameters of the pre-trained model, and the goal is to optimize Θ to find the best ΔΦ, thereby improving the performance of the model on a specific task. In addition, in order to optimize the low-rank matrices A and B, the mean square error function L is defined and the gradient with respect to A and B is calculated. The mean square error function can be expressed as:

[0074]

[0075] Where y is the output of the objective function. The gradients of the mean square error function for A and B are:

[0076]

[0077]

[0078] Finally, the gradient descent algorithm is used to update A and B. The update formula is:

[0079]

[0080]

[0081] Optionally, the target output is obtained according to the target input through the target large model, including: obtaining the initial output of the target large model according to the target input; collecting security knowledge through an external knowledge base to establish a security knowledge base; based on the security knowledge base, adjusting the initial output to obtain the target output.

[0082] Combination Figure 5 As shown, the embodiment of the present disclosure provides another method for large model security defense, including:

[0083] S101: The processor classifies input content to obtain target input.

[0084] S102, the processor obtains the initial output of the target large model according to the target input.

[0085] S103, the processor collects security knowledge through an external knowledge base and establishes a security knowledge base.

[0086] S104: The processor adjusts the initial output based on the security knowledge base to obtain a target output.

[0087] S105: The processor obtains a target input-output pair according to the target input and the target output.

[0088] S106: The processor performs a security assessment on the target input-output pair to obtain an unsafe input-output pair.

[0089] S107, the processor adjusts the target large model according to the unsafe input-output pair to obtain an adjusted target large model.

[0090] When the target input cannot obtain a matching safety output through the target large model, it is necessary to call an external knowledge base to support the target large model to make the target output more complete. By integrating the external knowledge base to collect security knowledge, and in real time based on updated laws and regulations and other security information, more stringent defense reinforcement is carried out against harmful information, making the generated target output more accurate and secure.

[0091] To establish a security knowledge base, you can use crawlers to obtain security common sense knowledge, or automatically browse web pages and extract information. The crawler will collect relevant content about security common sense from multiple websites based on preset keywords or rules, such as protective measures, safety warnings, laws and regulations, and other information. The collected information will be organized into question-answer pairs and stored, represented as K = [QA]. A security question and its corresponding answer form a question-answer pair. This format is convenient for subsequent retrieval and use. Converting question-answer pairs into a form suitable for machine processing involves the process of text vectorization. In the text vectorization process, each question-answer pair will be converted into a numerical vector for machine learning or deep learning processing. First, the collected text data is cleaned and preprocessed, including removing stop words, punctuation marks, and performing stem extraction. Then, the text is converted into a vector using text vectorization technology. The text is converted into a numerical vector with a certain dimension through an algorithm, and each dimension represents a feature of the text. The converted vector is stored in the security knowledge base. The security knowledge base supports efficient vector search and matching functions, which can quickly retrieve the answer most relevant to the input question. The mapping process is to convert natural language into a format that can be understood and processed by machines, so that in subsequent applications, such as intelligent question-answering systems, information retrieval, etc., the machine can efficiently identify and respond to user queries. This method can greatly improve the speed and accuracy of processing, thus playing an important role in practical applications.

[0092] Optionally, the initial output is adjusted to obtain a target output, including: obtaining target security knowledge matching the initial output in a security knowledge base; and performing fusion reasoning based on the target security knowledge and the initial output to obtain the target output.

[0093] After the security knowledge base is established, the request for fine-tuning the initial output and the question-answer pair in the security knowledge base are matched by similarity through security knowledge query, and the most similar preset number of security knowledge is returned to the target large model. When a query request from the target large model is received, the request needs to be preprocessed first. This includes cleaning the text (removing useless symbols, correcting spelling, etc.), and using the same method as in the security knowledge base to standardize and vectorize the text. This ensures that the data format and processing method in the request are consistent with those in the knowledge base, thereby improving the accuracy of subsequent matching. The preprocessed text is then converted into a vector form through a vectorization model. This vector is in the same vector space as the question-answer pair vector stored in the security knowledge base, so that the similarity between them can be effectively compared. The cosine similarity algorithm and the entity similarity algorithm are used to calculate the similarity between the request vector and all pre-stored content in the knowledge base. In this way, the knowledge point most relevant to the user's request is found. Then, based on the calculated similarity score, the preset number of contents with the highest similarity score is selected. This selection is based on similarity sorting to ensure that the security knowledge returned to the user is the most relevant to his query. Finally, the selected content is returned to the target large model for subsequent processing. The returned content can be used to directly answer the user's query, or as background reference knowledge for the target large model to respond, helping the model to better understand and respond to security-related queries. Finally, the target large model fuses the original query request with the security reference content retrieved from the security knowledge base and returns the final target output.

[0094] Combination Figure 6 As shown, an embodiment of the present disclosure provides a device 300 for large model security defense, including a processor 200 and a memory 201. Optionally, the device may also include a communication interface 202 and a bus 203. Among them, the processor 200, the communication interface 202, and the memory 201 can communicate with each other through the bus 203. The communication interface 202 can be used for information transmission. The processor 200 can call the logic instructions in the memory 201 to execute the method for large model security defense of the above embodiment.

[0095] In addition, the logic instructions in the memory 201 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0096] The memory 201 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 200 executes the functional application and data processing by running the program instructions / modules stored in the memory 201, that is, the method for large model security defense in the above embodiment is implemented.

[0097] The memory 201 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 201 may include a high-speed random access memory and may also include a non-volatile memory.

[0098] The disclosed embodiments provide an electronic device, comprising: an electronic device body, and the above-mentioned device for large-model security defense. The electronic device body is provided with a large model. The device for large-model security defense is installed on the electronic device body. The installation relationship described here is not limited to placement inside the electronic device, but also includes installation connections with other components of the electronic device, including but not limited to physical connections, electrical connections or signal transmission connections, etc. Those skilled in the art can understand that the device for large-model security defense can be adapted to a feasible electronic device body, thereby realizing other feasible embodiments.

[0099] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for large model security defense.

[0100] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0101] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the technical solutions recorded in this application. As used in the technical solutions recorded in this application, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0102] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0103] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0104] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for large model security defense, characterized in that: include: Classify the input content and obtain the target input; Obtain a target output according to the target input through the target large model, and obtain a target input-output pair according to the target input and the target output; Perform security assessment on the target input-output pair to obtain an unsafe input-output pair; According to the unsafe input-output pairs, the target large model is adjusted to obtain an adjusted target large model.

2. The method according to claim 1, characterized in that Categorize input content, including: Preprocess the input content; Based on the trained filtering model, the text features of the preprocessed input content are recognized, and the input content is classified according to the text features.

3. The method according to claim 2, characterized in that The filtering model is trained as follows: the filtering model is trained once or multiple times; the iterative training includes: Based on the training data, calculate the gradient of the loss function with respect to the parameters of the filtering model; Update the parameters of the filtering model according to the gradient; Based on the validation data, the performance of the filtering model after parameter update is evaluated; The convergence conditions of the iterative training include: the number of iterative training times reaches a preset number or the performance evaluation result reaches a preset condition.

4. The method according to claim 1, characterized in that: Perform security assessment on the target input-output pair, including: Obtaining an evaluation model for security evaluation; Set prompt words and examples for evaluating the model; Based on the evaluation model, the target input-output pairs are scored according to the prompt words and examples.

5. The method according to claim 4, characterized in that Train and evaluate the model as follows: Embedding evaluation criteria in the prompt words of the evaluation model; the evaluation criteria include security evaluation standards for input-output pairs, and the format of receiving input and generating output of the evaluation model; Obtain human-annotated scores for training input-output pairs under the evaluation criteria; The evaluation model is trained according to the training input-output pairs and manual annotation scores to obtain a trained evaluation model.

6. The method according to claim 1, characterized in that According to the unsafe input-output pairs, the target large model is adjusted, including: Rewrite the unsafe input-output pairs to obtain input-output pair samples; Introducing a trainable low-rank matrix into the target large model to modify the weights of the target large model; The low-rank matrix of the sample pair is trained according to the input and output to adjust the weight of the target large model to obtain the adjusted target large model.

7. The method according to any one of claims 1 to 6, characterized in that: Through the target large model, the target output is obtained according to the target input, including: Obtaining the initial output of the target large model according to the target input; Collect security knowledge through external knowledge bases and establish a security knowledge base; Based on the security knowledge base, the initial output is adjusted to obtain the target output.

8. The method according to claim 7, characterized in that Adjust the initial output to obtain the target output, including: Acquire target security knowledge matching the initial output in the security knowledge base; Perform fusion reasoning based on target security knowledge and initial output to obtain target output.

9. A device for large model security defense, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the method for large model security defense as described in any one of claims 1 to 8 when running the program instructions.

10. An electronic device, characterized in that: include: The electronic device body is provided with a large model; The device for large model security defense as described in claim 9 is installed on the electronic device body.

Citation Information

Cited By

  • Corpus construction method and device, equipment, storage medium and product

    CN120234389A

  • Corpus construction method and device, equipment, storage medium and product

    CN120234389B