Large model safety prevention and control method and device

Abstract

The embodiment of the invention provides a large model safety prevention and control method and device. The problem that a large model is insufficient in safety in a specified field range is solved. The large model safety prevention and control method comprises the steps of receiving label data, and performing automatic processing on the label data; filtering the label data after the automatic processing; training and merging the large model based on the filtered label data; receiving questions proposed by the user, and performing security identification and filtering on the questions to judge whether to answer or not; if the questions can be answered, reasoning and answering the questions through a big model after training and merging the questions, and obtaining question and answer pairs; and performing vectorization storage on the question and answer pairs, and providing later annotations.

Description

Technical Field

[0001] The present invention relates to the field of model safety technology, and in particular to a large model safety control method and device. Background Art

[0002] While large models provide convenience and innovation, they also bring a series of security issues. For example: 1. Unexplainability. Large models are often very complex and lack transparency in their decision-making process, which may make the model's behavior difficult to explain and understand. 2. Dependence: Over-reliance on large models may lead to reduced human judgment in the decision-making process, especially in high-risk areas. 3. Manipulation and abuse. 4. Transparency and explainability: Large models are often "black boxes" and lack transparency in their decision-making process, which makes it difficult to explain the reasons for their output results. 5. Difficulty of control: For some generative large models, the output content may be difficult to fully control and predict, and may produce inappropriate or harmful content.

[0003] At present, in order to deal with these security issues, generally only the following measures are taken: 1. Data protection: ensure the security of training data and desensitize sensitive information. 2. Content verification: establish a mechanism to verify the authenticity of the content generated by the model. 3. Access control: strictly control and manage the access and use of the model. 4. Laws and regulations: formulate relevant laws and regulations to regulate the use of large models and crack down on illegal activities. 5. Ethical review: conduct ethical review during model development and deployment to ensure that it complies with social ethical standards. Through these measures, you can enjoy the convenience brought by large models while minimizing their potential risks and negative impacts. However, this process requires a lot of manual annotation and content restrictions. For example, during the content verification process, the seed is often unrecognizable due to the "black box" of the large model, especially for the results that cannot be interrupted during the streaming output of the large model. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a large model security control method and device, which solves the problem of insufficient security of large models within a specified field.

[0005] In the first aspect, an embodiment of the present invention provides a large model security control method, including: receiving label data, and automatically processing the label data; filtering the automatically processed label data; training and merging the large model based on the filtered label data; receiving questions raised by users, and performing security identification and filtering on the questions to determine whether to answer them; if the questions can be answered, reasoning and answering the questions through the trained and merged large model to obtain question-answer pairs; vectorizing and storing the question-answer pairs, and providing them for later annotation.

[0006] In one embodiment, the multi-label data includes at least one of: web crawled data, manually processed data, and benchmark label data.

[0007] In one embodiment, the automated processing of the label data includes: processing the label data based on the setting of the large module to generate accurate label questions and answers; processing the label data based on the setting of the large module to generate reverse label questions and answers.

[0008] In one embodiment, the filtering of the automatically processed label data includes: inputting the precise label questions and answers and the reverse label questions and answers into the large model, and formatting and filtering the precise label questions and answers and the reverse label questions and answers.

[0009] In one embodiment, the training and merging of the large model based on the filtered label data includes: fine-tuning the large model based on the precise label question and answer; performing multi-label training on the large model based on the precise label question and answer and the reverse label question and answer to generate and save label vector parameters; and merging the large model after fine-tuning training and the large model after multi-label training.

[0010] In one embodiment, after the problems are trained and merged into a large model, the method further includes: if the model output involves irrelevant problems, the problems are again subjected to security identification filtering.

[0011] In the second aspect, an embodiment of the present invention provides a large-model security control device, including: a data receiving unit for receiving label data; an automatic processing unit for automatically processing the label data; a data filtering unit for filtering the label data after automatic processing; a training merging unit for training and merging the large model based on the filtered label data; a question receiving unit for receiving questions raised by users; a question filtering unit for performing security identification and filtering on questions to determine whether to answer them; an inference answering unit for performing inference answers to questions and obtaining question-answer pairs; a storage unit for vectorizing and storing question-answer pairs and providing them for later annotation.

[0012] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned method.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method are implemented.

[0014] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the above-mentioned method when executed by a processor.

[0015] An embodiment of the present invention provides a large model security control method and device, which receives label data and automatically processes the label data; filters the automatically processed label data; trains and merges the large model based on the filtered label data; receives questions raised by users, and performs security identification and filtering on the questions to determine whether to answer them; if they can be answered, the questions are answered through the trained and merged large model to obtain question-answer pairs; the question-answer pairs are vectorized and stored, and provided for later annotation. By utilizing the method or device of the present invention, when a user inputs a question or a large model outputs a question, the large model can perform security interception of the content before returning it to the user, and the security layer is an embedded layer of the large model, and the interception will be returned quickly, thereby reducing the waste of resources for reasoning after the data enters the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of a large model safety control method provided by one embodiment of the present invention.

[0017] Figure 2 Shown is a schematic structural diagram of a large-model safety control device provided by an embodiment of the present invention.

[0018] Figure 3 Shown is a schematic structural diagram of a large model safety control system provided by an embodiment of the present invention.

[0019] Figure 4 The figure is a flow chart of a control process based on a large model safety control system provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0022] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0023] This embodiment provides a large model security control method, such as Figure 1 As shown, the large model safety prevention and control method includes:

[0024] Step 01: Receive label data and automatically process the label data. This method first requires the user to submit data with multiple labels, and the large model security prevention and control system receives these label data. The label data provides a basis for automatic data generation. Optionally, the label data is multi-label question and answer data. The label data can accurately match the label.

[0025] The multi-label data includes at least one of web crawled data, manually processed data and benchmark label data.

[0026] Web crawling data: Since some reverse label data is required for data training, the workload of manual labeling is too large, so you can first crawl public question and answer data from the Internet to reduce manual operations.

[0027] Manually processed data: This part of the data is sampled and labeled manually from the question-answering data after reasoning, so the sample data can be continuously improved;

[0028] Benchmark label data: Benchmark label data is label data entered by users. The backend administrator provides benchmark label data, which is provided in question-and-answer mode, and each data can have multiple labels.

[0029] The step of automatically processing the label data includes:

[0030] The label data is processed based on the setting of the large model to generate accurate label questions and answers; it can be understood that according to the setting of the large model module, positive label-related questions are generated in batches to provide more data sets for model training.

[0031] Based on the setting of the large model, the label data is processed to generate reverse label questions and answers; it can be understood that according to the setting of the large model module, questions opposite to the label are generated to provide more data sets for model training

[0032] Step 02: Filter the label data after automatic processing.

[0033] To ensure the quality of label data, including data integrity, accuracy, and consistency, the precise label questions and answers and reverse label questions and answers obtained in the above steps are put into the big model for format judgment. The format includes length, Chinese and English, etc.

[0034] Step 03: Train and merge the large model based on the filtered labeled data.

[0035] Model training, the induction or learning process, is a core step in model fine-tuning and involves training the model using labeled data.

[0036] The training and merging of the large model based on the filtered label data includes:

[0037] Step 031: Model fine-tuning training, that is, fine-tuning the large model based on the precise label question and answer; preferably, Lora fine-tuning is performed on the precise label question and answer.

[0038] Step 032: Multi-label training of the model, multi-label training of the large model is performed based on the precise label question and answer and the reverse label question and answer, and label vector parameters are generated and saved; preferably, after step 031 is completed, Lora fine-tuning multi-label training is performed on the precise label question and answer and the reverse label question and answer, and label vector parameters are generated and saved.

[0039] Step 033: Model merging, merging the large model after fine-tuning training and the large model after multi-label training.

[0040] Step 04: Receive the question raised by the user and perform security identification filtering on the question to determine whether to answer it. Preferably, performing security identification filtering on the question includes performing multi-label security identification filtering on the question input by the user.

[0041] Step 05: If the question can be answered, the question is trained and merged into a large model to perform reasoning and answer the question and obtain a question-answer pair. If the model output involves irrelevant questions, the question is filtered for security identification again.

[0042] Step 06: Vectorize and store the question and answer pairs and provide them for later annotation. In addition, system maintenance will regularly sample data labels from the question and answer storage to provide higher quality answers for the next iteration and upgrade.

[0043] To sum up, the large model security control method provided by the present invention, after determining whether the question raised by the user can be answered, also includes step 07: if no answer is required, directly reject it, and provide a unified rejection answer discourse based on system maintenance, which is directly returned to the user and ends the reasoning.

[0044] The large model security control method provided by the present invention provides a label-based rapid question-and-answer security filtering, realizes automated batch self-generation of multi-label data, realizes the filtering and screening of large amounts of network data, and can perform security control on data before large model reasoning and during the reasoning process.

[0045] This embodiment provides a large model safety control device 100, such as Figure 2 As shown, the large model safety control device 100 includes: a data receiving unit 10, an automatic processing unit 20, a data filtering unit 30, a training merging unit 40, a question receiving unit 50, a question filtering unit 60, an inference answering unit 70 and a storage unit 80. Among them,

[0046] The data receiving unit 10 is used to receive label data; wherein the multi-label data includes at least one of web crawling data, manually processed data and benchmark label data.

[0047] The automatic processing unit 20 is used to automatically process the label data;

[0048] The data filtering unit 30 is used to filter the label data after the automatic processing;

[0049] The training merging unit 40 is used to train and merge the large model based on the filtered label data;

[0050] The question receiving unit 50 is used to receive questions raised by users;

[0051] The question filtering unit 60 is used to perform security identification filtering on the question to determine whether to answer it;

[0052] The reasoning answering unit 70 is used to perform reasoning answers to questions and obtain question-answer pairs;

[0053] The storage unit 80 is used to vectorize and store the question-answer pairs and provide them for later annotation.

[0054] If the model output involves irrelevant issues, the issues are filtered for security identification again.

[0055] Furthermore, the automated processing unit 20 is also used to process the label data based on the setting of the large module to generate accurate label questions and answers; and to process the label data based on the setting of the large module to generate reverse label questions and answers.

[0056] Furthermore, the data filtering unit 30 is also used to input the precise label question and answer and the reverse label question and answer into the large model, and perform formatting and filtering on the precise label question and answer and the reverse label question and answer.

[0057] Furthermore, the training merging unit 40 is also used to perform fine-tuning training on the large model based on the precise label question and answer; perform multi-label training on the large model based on the precise label question and answer and the reverse label question and answer, generate and save label vector parameters; and merge the large model after fine-tuning training and the large model after multi-label training.

[0058] The big model is trained with data based on the needs of specific fields, that is, users can let the big model only answer questions within the specified business field.

[0059] This embodiment provides a large model security control system, such as Figure 3 As shown in the figure, the main modules of the safety control system include:

[0060] Initial training set (manual) module H, the system first requires users to submit multi-label data, which provides the basis for automatic data generation. The label data refers to multi-label question and answer data, including:

[0061] H1: Web data crawling. Some reverse label data is required for data training. The workload of manual labeling is too large. Therefore, we first crawl public question and answer data from the Internet to reduce manual operations.

[0062] H2: Manual post-processing of questions and answers. This part of the data is sampled and labeled manually from the questions and answers after reasoning, which can continuously improve the sample data;

[0063] H3: User label data entry. The backend administrator provides benchmark label data. The data is provided in a question-and-answer mode, and each data can have multiple labels.

[0064] User question and answer module H4: provides users with a question and answer access interface.

[0065] Automated data processing module D: performs automated data processing based on the administrator's input data. Including:

[0066] L1: Generate accurate label questions and answers. According to the settings of the initial training set (manual) module H, batch generate positive, label-related questions to provide more data sets for model training;

[0067] L2: Generate reverse label questions and answers. According to the settings of the initial training set (manual) module H, generate questions that are opposite to the labels to provide more data sets for model training;

[0068] Data filtering module V1: ensures data quality, including data integrity, accuracy, consistency, etc. The module puts L1 and L2 data into the model for format judgment, including length, Chinese and English, etc.

[0069] Model training M: Model training is the induction or learning process and is a core step in model fine-tuning. It involves training the model using labeled data (module D):

[0070] M1: Model fine-tuning training, performing Lora fine-tuning training on L1 data.

[0071] M2: Model multi-label training. After M1 is completed, Lora fine-tuning multi-label training is performed on L1 and L2 data to generate and save label vector parameters.

[0072] M3: Model merging, merging the trained M1 and M2, and performing Lora merging on M1.

[0073] Multi-label vector storage module M4: extracts the vector layer of the M3 model and stores it separately.

[0074] Model security reasoning R: After the module is combined with M training, security control is performed on questions and answers.

[0075] Question filtering module R1: performs M4 multi-label security identification filtering on the user's input questions;

[0076] Model reasoning module R2: security filtering issues are put into M3 for normal reasoning;

[0077] Reasoning filtering output module R3: If the model output involves irrelevant issues, it will be filtered again to avoid problems that are missed when filtering in R1 later.

[0078] Question and answer process storage module R4: The question and answer pairs are saved so that the administrator can conduct security sampling checks on the data in the future.

[0079] This embodiment provides a prevention and control process based on a large model security prevention and control system, such as Figure 4 As shown in the figure, the prevention and control process of the large model-based safety prevention and control system includes:

[0080] 0. Initialize label data entry:

[0081] A certain amount of labeled data is manually given, which is an accurate match of the label.

[0082] 1. Web data crawling

[0083] The label data needs to be reverse identified, so this module collects reverse labels, and the data also needs to be judged by the module to confirm whether it is completely opposite.

[0084] 2. Data Filtering

[0085] The data needs to be formatted and filtered, and the text length, Chinese and English, and encoding must be filtered.

[0086] 3. Multi-label model training

[0087] The generated data and training are used for fine-tuning training of the model and multi-label layer training.

[0088] 4. User Q&A Mode

[0089] Users raise questions, and this part contains a historical memory function.

[0090] 5. Question safety filtering

[0091] According to the multi-label layer of M4 described in the above embodiment, fast vectorized filtering is performed instead of reasoning the content, and True or False is directly returned. If False, the answer is rejected in step 6; otherwise, normal reasoning is performed in step 7.

[0092] 6. Unified refusal to answer

[0093] According to system maintenance, the uniform rejection answer utterance provided is directly returned to the user, and the reasoning ends.

[0094] 7. Problem reasoning

[0095] For normal questions, the large model after training and merging is used to perform reasoning and answer the questions.

[0096] 8. Question and answer storage

[0097] The question-answer pairs are vectorized and stored for later annotation.

[0098] 9. Manual annotation of questions and answers

[0099] System maintenance will regularly sample data labels from the question and answer storage to provide higher quality answers for the next iteration and upgrade.

[0100] 10. Execution is completed.

[0101] On the basis of the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0102] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0103] In some implementations of this embodiment, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0104] The processor may include, but is not limited to, one or more processors or microprocessors, etc. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0105] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the computer-readable storage medium may include but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0106] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0107] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0108] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0109] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0110] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0112] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the technical field to which the present disclosure belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A large model safety control method, characterized in that: include: receiving label data, and automatically processing the label data; Filter the label data after automatic processing; Merge the training of large models based on filtered label data; Receive questions from users and perform security identification and filtering on the questions to determine whether to answer them; If the question can be answered, the large model after training and merging will be used to reason and answer the question and obtain the question-answer pair; The question-answer pairs are vectorized and stored, and provided for later annotation.

2. The large model safety control method according to claim 1 is characterized in that: The multi-label data includes at least one of web crawled data, manually processed data and benchmark label data.

3. The large model safety control method according to claim 1 is characterized in that: The automatic processing of the label data includes: The label data is processed based on the setting of the large module to generate accurate label questions and answers; The label data is processed based on the setting of the large module to generate reverse label questions and answers.

4. The large model safety control method according to claim 3 is characterized in that: The filtering of the automatically processed label data includes: inputting the precise label questions and answers and the reverse label questions and answers into the large model, and formatting and filtering the precise label questions and answers and the reverse label questions and answers.

5. The large model safety control method according to claim 5 is characterized in that: The training and merging of the large model based on the filtered label data includes: Fine-tune the large model based on the precise label question and answer; Based on the precise label question and answer and the reverse label question and answer, multi-label training is performed on the large model to generate and save label vector parameters; Merge the large model trained by fine-tuning and the large model trained by multi-label training.

6. The large model safety control method according to claim 1 is characterized in that: After the problems are trained and merged into a large model, the method further includes: if the model output involves irrelevant problems, the problems are again subjected to security identification and filtering.

7. A large model safety control device, characterized in that: include: A data receiving unit, used for receiving tag data; An automated processing unit, used for automatically processing the label data; A data filtering unit, used to filter the label data after automatic processing; A training merging unit, used to train and merge large models based on filtered label data; A question receiving unit, used for receiving questions raised by users; A question filtering unit, used to perform security identification and filtering on questions to determine whether to answer them; The reasoning answering unit is used to provide reasoning answers to questions and obtain question-answer pairs; The storage unit is used to vectorize and store question-answer pairs and provide them for later annotation.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images