Content security capability construction method and device of industry large model, equipment and medium

Abstract

The invention discloses a content security capability construction method and device of an industry large model, equipment and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: pulling user interaction data in a production environment of the industry large model based on a preset data pulling frequency; calling a preset super-large-scale model to carry out risk discrimination on the to-be-discriminated question data so as to determine whether misjudged target question data exists or not according to a discrimination result; if the target question data exists, correcting a data label of the target question data, and expanding the original data set based on the corrected data and a preset super-large-scale model so as to perform iterative training on the security model by using the expanded data set, and migrating knowledge of the preset super-large-scale model into the security model; and constructing a preference pair data set based on the user feedback data and a preset super-large-scale model, and performing value alignment training on the industry large model to control the generation content of the industry large model to meet a preset value concept.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to methods, devices, equipment, and media for building content security capabilities for industry large models. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the application of industry big models has penetrated into multiple fields, providing efficient and intelligent services. However, as the text generation capabilities of industry big models become increasingly powerful, content security issues have gradually become prominent. While the content security modules of existing industry big models can block inappropriate content to a certain extent, due to the uneven distribution of training data and the limitations of model capabilities, misjudgments inevitably occur, misidentifying legitimate user questions as risky content. These misjudgments not only affect the user experience but may also restrict access for legitimate users. Furthermore, because traditional security technologies focus primarily on reviewing static content, their detection and protection mechanisms for information generated in real time by big models are insufficient. Furthermore, the complexity of the big model structure and the diversity of generated content further increase the difficulty of content security detection.

[0003] To sum up, how to improve the interception accuracy of security models and ensure the security of content generated by large industry models is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment, and medium for building content security capabilities for industry large models, which can improve the interception accuracy of the security model and ensure the security of the content generated by the industry large model. The specific solution is as follows:

[0005] First, this application provides a method for building content security capabilities for an industry-wide model, including:

[0006] Pulling user interaction data from a production environment of a large industry model based on a preset data pulling frequency, and cleaning the user interaction data to store the cleaned user interaction data in a database to be processed;

[0007] Calling a preset ultra-large-scale model to perform risk assessment on the question data to be assessed in the database to be processed, so as to determine whether there is misjudged target question data in the question data to be assessed based on the assessment result; the question data to be assessed is the user question data intercepted by the security model in the user interaction data; the security model is a model for risk assessment of user questions input into the industry large model; the preset ultra-large-scale model is a model whose model scale meets the preset ultra-large-scale assessment conditions;

[0008] If the target question data exists, the data label of the target question data is corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so that the security model is iteratively trained using the obtained expanded data set, and the knowledge of the preset ultra-large-scale model is transferred to the security model through a preset knowledge distillation technology, and the security model in the production environment is replaced with the obtained target security model;

[0009] A preference pair dataset is constructed based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and the industry large model is trained for value alignment using a preset reinforcement learning algorithm and the preference pair dataset to control the generated content of the industry large model to meet the preset value concept.

[0010] Optionally, calling a preset ultra-large-scale model to perform risk identification on the question data to be identified in the database to be processed, so as to determine whether there is misidentified target question data in the question data to be identified based on the identification result, includes:

[0011] Calling a preset ultra-large-scale model through a preset interface to perform risk discrimination on the question data to be discriminated in the database to be processed, so as to obtain a discrimination result output by the preset ultra-large-scale model;

[0012] Determining whether the confidence level of the judgment result is higher than a preset confidence threshold;

[0013] If the confidence level in the discrimination result is higher than a preset confidence threshold, and the discrimination result indicates that the question data to be discriminated is risk-free data, then the question data to be discriminated is determined to be misjudged target question data.

[0014] Optionally, the expanding the original data set based on the corrected data and the preset ultra-large-scale model includes:

[0015] The preset ultra-large-scale model is called to generate a number of newly added user question data; the newly added user question data is data whose semantic similarity with the corrected data meets a preset similarity condition and is risk-free;

[0016] The corrected data and the newly added user question data are added to the original data set to expand the original data set.

[0017] Optionally, the iterative training of the security model using the obtained expanded data set includes:

[0018] Cleaning the obtained expanded data set, and dividing the cleaned expanded data set into a training data set, a validation data set, and a test data set according to a preset distribution ratio;

[0019] Introducing a preset low-rank matrix in the target layer of the security model, and initializing the preset low-rank matrix using randomly generated initial values ​​or weights in the security model; the preset low-rank matrix is ​​a matrix whose rank satisfies a preset low-rank determination condition;

[0020] Iteratively training the security model with all parameters frozen except for the preset low-rank matrix using the training data set, and after each round of training, fine-tuning the security model using the validation data set;

[0021] A performance evaluation is performed on the security model after iterative training based on the test data set to obtain the security model whose performance after iterative training meets preset performance conditions.

[0022] Optionally, migrating the knowledge of the preset ultra-large-scale model to the security model using a preset knowledge distillation technology, and replacing the security model in the production environment with the obtained target security model, includes:

[0023] Calling the preset ultra-large-scale model to generate corresponding soft labels for the user question data in the training data set;

[0024] Performing distillation training on the security model using the soft labels, and controlling the smoothness of the soft labels using a preset temperature parameter during the distillation training process, so as to transfer knowledge of the preset ultra-large-scale model to the security model;

[0025] The security model after distillation training is evaluated using the verification data set and the test data set to obtain a target security model whose performance meets the preset performance conditions, and the target security model is used to replace the security model in the production environment.

[0026] Optionally, constructing a preference pair dataset based on the user feedback data in the database to be processed and the preset ultra-large-scale model includes:

[0027] Obtain user feedback data from the database to be processed;

[0028] Generating, by the preset ultra-large-scale model, several new question contents whose contents meet preset similarity conditions with the target question contents of the user feedback data;

[0029] Using the preset ultra-large-scale model, a plurality of answers corresponding to the target question content and the plurality of newly added question contents are generated respectively, and a first target answer and a second target answer are determined from the plurality of answers through a preset scoring mechanism; the first target answer is the answer with the highest score among the plurality of answers; and the second target answer is the answer with the lowest score among the plurality of answers;

[0030] A preference pair data set is constructed based on the target question content, the first target answer and the second target answer corresponding to the target question content, the several newly added question contents, and the first target answers and the second target answers corresponding to the several newly added question contents.

[0031] Optionally, the performing value alignment training on the industry model using a preset reinforcement learning algorithm and the preference pair dataset includes:

[0032] Using a direct preference optimization algorithm, the industry model is trained on value alignment using a preset likelihood function and the preference pair dataset;

[0033] Alternatively, the preference is used to train a preset reward model for the data set, and the industry large model is trained for value alignment based on the proximal strategy optimization algorithm. During the value alignment training process, the value alignment training strategy is adjusted according to the reward score output by the preset reward model.

[0034] In a second aspect, the present application provides a device for building content security capabilities for a large industry model, including:

[0035] A data cleaning module is used to pull user interaction data in the production environment of the industry large model based on a preset data pulling frequency, and clean the user interaction data to store the cleaned user interaction data in the database to be processed;

[0036] A risk identification module is configured to call a preset ultra-large-scale model to perform risk identification on the question data to be identified in the database to be processed, so as to determine whether there is misidentified target question data in the question data to be identified based on the identification result; the question data to be identified is the user question data intercepted by the security model in the user interaction data; the security model is a model for performing risk identification on user questions input into the industry large model; the preset ultra-large-scale model is a model whose model scale meets the preset ultra-large-scale identification conditions;

[0037] a data set expansion module, configured to, if the target question data exists, correct the data labels of the target question data, and expand the original data set based on the corrected data and the preset ultra-large-scale model, so as to iteratively train the security model using the obtained expanded data set, migrate the knowledge of the preset ultra-large-scale model to the security model through a preset knowledge distillation technique, and replace the security model in the production environment with the obtained target security model;

[0038] The model training module is used to construct a preference pair dataset based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and use the preset reinforcement learning algorithm and the preference pair dataset to perform value alignment training on the industry large model to control the generated content of the industry large model to meet the preset value concept.

[0039] In a third aspect, the present application provides an electronic device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor is used to execute the computer program to implement the aforementioned method for building content security capabilities of the industry big model.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method for building content security capabilities of the industry big model is implemented.

[0043] In this embodiment, user interaction data in the production environment of the industry large model is pulled based on the preset data pulling frequency, and the user interaction data is cleaned to store the cleaned user interaction data in the database to be processed; the preset ultra-large-scale model is called to perform risk judgment on the question data to be judged in the database to be processed, so as to determine whether there is misjudged target question data in the question data to be judged based on the obtained judgment result; the question data to be judged is the user question data in the user interaction data that is intercepted by the security model; the security model is a model for performing risk judgment on user questions input into the industry large model; if the target question data exists, the target question data is judged. The data labels of the question data are corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so that the security model can be iteratively trained with the obtained expanded data set, and the knowledge of the preset ultra-large-scale model is transferred to the security model through the preset knowledge distillation technology, and the security model in the production environment is replaced with the obtained target security model; a preference pair data set is constructed based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and the industry large model is value-aligned with the preset reinforcement learning algorithm and the preference pair data set to control the generated content of the industry large model to meet the preset value concept. As can be seen from the above, the present application pulls user interaction data in the industry large-scale model production environment based on the preset data pulling frequency, and stores it in the database to be processed after cleaning, so as to call the preset ultra-large-scale model to perform risk judgment on the question data to be judged in the database to be processed, and determine whether there is misjudged target question data based on the obtained judgment result. If so, the data label of the target question data is corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so as to use the obtained expanded data set to iteratively train the security model, and migrate the knowledge of the preset ultra-large-scale model to the security model through the preset knowledge distillation technology, and replace the security model in the production environment with the obtained target security model. At the same time, a preference pair data set is constructed based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and the preset reinforcement learning algorithm and the preference pair data set are used to perform value alignment training on the industry large model to control the generated content of the industry large model to meet the preset value concept.In this way, through the above process of this application, user interaction data from the production environment of the industry big model is regularly pulled, and the security model is iteratively trained based on the question data to be judged in the user interaction data and data enhancement technology, which can improve the accuracy and robustness of the security model. At the same time, the industry big model is value-aligned through user feedback data to ensure that even if the small security model fails to accurately identify risk issues, the industry big model will not generate content that does not conform to the preset values, thereby improving the interception accuracy of the security model and ensuring the security of the content generated by the industry big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of a method for building content security capabilities for a large industry model disclosed in this application;

[0046] Figure 2 A schematic diagram of the overall technical architecture of a method for building content security capabilities for a large industry model disclosed in this application;

[0047] Figure 3 A schematic diagram of the structure of a device for building content security capabilities for a large industry model disclosed in this application;

[0048] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] While existing industry-wide content security modules can block inappropriate content to a certain extent, due to the uneven distribution of training data and limitations in model capabilities, they inevitably suffer from misjudgments, misidentifying legitimate user questions as risky content. These misjudgments not only impact the user experience but can also restrict access for legitimate users. Furthermore, because traditional security technologies primarily focus on reviewing static content, their detection and protection mechanisms for the real-time output of large models are insufficient. Furthermore, the structural complexity of large models and the diversity of generated content further complicate content security detection.

[0051] In order to overcome the above technical problems, this application provides a method for building content security capabilities of an industry large model to improve the interception accuracy of the security model and ensure the security of the content generated by the industry large model.

[0052] See also Figure 1 As shown, an embodiment of the present invention discloses a method for building content security capabilities of an industry large model, including:

[0053] Step S11: Pull user interaction data from the production environment of the industry large model based on a preset data pulling frequency, and clean the user interaction data to store the cleaned user interaction data in a database to be processed.

[0054] In this embodiment, the corresponding user interaction data is pulled from the production environment of the industry big model through the log collector based on the preset data pulling frequency, and the user interaction data is cleaned to remove duplicate, invalid or noisy data to ensure the cleanliness and consistency of the data, and the cleaned user interaction data is stored in the database to be processed. Among them, the industry big model is a vertical field generative big model that has been deeply customized and optimized for specific industry needs and application scenarios, and is used to dynamically generate answers to user questions; the specific industry field can be medical, manufacturing process, finance and other industry fields; the production environment refers to the environment in which external services are formally provided during the software development process; the user interaction data includes user question data intercepted in the production environment, answers generated by the industry big model for the user question data, and user feedback data on the model-generated content; the cleaning process includes removing redundant spaces, punctuation marks, special characters, etc.; the database to be processed is a distributed database such as MongoDB (a database based on distributed file storage) or Elasticsearch (an open source distributed search engine and data analysis engine), which is convenient for subsequent batch processing and querying.

[0055] It is understandable that if Figure 2The figure shows the overall technical architecture of a method for building content security capabilities for an industry-wide model, as provided in this application. To correct potential misjudgments in existing security models when identifying user input, this application introduces an automated security model iteration method. By regularly pulling user data, this method leverages the powerful discriminative capabilities of a pre-set, ultra-large-scale model to correct misjudgments made by smaller security models in complex scenarios, thereby improving the accuracy and robustness of the overall system. Furthermore, to ensure responses consistent with human values ​​while ensuring security, a value alignment mechanism for the industry-wide model (RLHF) is introduced. By incorporating human feedback, the generation behavior of the industry-wide model is dynamically optimized, ensuring that even if the security model fails to accurately identify certain risk issues, subsequent industry-wide models will not generate content that is inconsistent with pre-set values. It should be noted that the data pulling operation based on the pre-set data pulling frequency can be executed through a scheduled task scheduling system, such as Celery (a distributed task queue) or Airflow (an open source scheduling tool), while also regularly performing risk identification, label correction, and data storage operations. Integrate steps such as data preprocessing, model training, evaluation, and deployment into an automated pipeline, and implement automated management through tools such as Jenkins (an extensible automation server) or GitLab CI / CD (an integrated continuous integration and continuous delivery tool), reducing manual intervention and improving the maintainability and scalability of the system. Specifically, the preset data pulling frequency can be adjusted according to actual needs and is usually set to be executed once a day or once a week. In addition, during the automated processing process, this embodiment can also introduce an exception handling mechanism to ensure that the system can automatically retry or record error logs when encountering abnormal situations such as network fluctuations and API (Application Programming Interface) call failures, thereby avoiding data loss or processing interruptions. In this way, this embodiment pulls user interaction data from the production environment of the industry big model based on the preset data pulling frequency, and saves it to the database to be processed after cleaning, so that subsequent misjudgment correction and value alignment processing can be performed based on the user interaction data, thereby improving the content security of the industry big model, effectively identifying and intercepting risky inputs, and optimizing the user experience, ensuring that responses that conform to human values ​​are provided under the premise of ensuring safety; in the processing flow of security capability construction, an exception handling mechanism is introduced to ensure automatic retry or error log recording when encountering abnormal situations, to avoid data loss or processing interruption.

[0056] Step S12: Call the preset ultra-large-scale model to perform risk assessment on the question data to be assessed in the database to be processed, so as to determine whether there is misjudged target question data in the question data to be assessed based on the assessment results; the question data to be assessed is the user question data intercepted by the security model in the user interaction data; the security model is a model for performing risk assessment on user questions input into the industry large model; the preset ultra-large-scale model is a model whose model scale meets the preset ultra-large-scale assessment conditions.

[0057] In this embodiment, a preset ultra-large-scale model is called to perform risk assessment on the question data to be judged in the database to be processed, so as to determine whether there is misjudged target question data in the question data to be judged based on the obtained assessment results. The preset ultra-large-scale model can be an ultra-large-scale model such as GPT-4 (a large-scale multimodal model) or PaLM (a path language model), which has strong semantic understanding and reasoning capabilities and can accurately judge the risk of user questions. The question data to be judged is user question data in the user interaction data that is intercepted by the security model in the production environment, including but not limited to information such as the text entered by the user, the reason for the interception, and the time of the interception. The security model is a model for performing risk assessment on user questions input into the industry large model, and can be a security model based on Qwen1.5-1.8B-Intruct (an open source large language model that has been fine-tuned with instructions). It has been pre-trained in the content security field and has certain risk identification capabilities.

[0058] Specifically, a preset ultra-large-scale model is called through a preset interface to perform risk assessment on the question data to be assessed in the database to be processed, thereby obtaining a assessment result output by the preset ultra-large-scale model; a determination is made as to whether the confidence level in the assessment result is higher than a preset confidence threshold; if the confidence level in the assessment result is higher than the preset confidence threshold, and the assessment result indicates that the question data to be assessed is risk-free data, then the question data to be assessed is determined to be misjudged target question data. The preset interface is a preset API interface; the assessment results are divided into two categories: high-risk and risk-free. High-risk refers to questions involving sensitive topics, illegal content, or content that does not conform to preset values, while risk-free refers to normal user input that does not involve any risky content. That is, the preset ultra-large-scale model is called through a preset interface, and the question data to be assessed in the database to be processed is used as input. Risk assessment is performed on the data based on its semantic content to obtain a assessment result output by the preset ultra-large-scale model. Furthermore, to ensure accuracy in discrimination, this embodiment can also set a preset confidence threshold, such as 0.9. Only when the confidence level of the preset ultra-large-scale model's discrimination result for a question exceeds this threshold, i.e., 0.9, is the discrimination result considered reliable. If it does not exceed this threshold, it is discarded as junk data. Specifically, after obtaining the discrimination result, a determination is made as to whether the confidence level in the discrimination result exceeds the preset confidence threshold. If the confidence level in the discrimination result exceeds the preset confidence threshold and indicates that the question data to be discriminated is risk-free, the question data to be discriminated is determined to be the target question data that was misjudged.

[0059] It should be noted that, during the API call process, this embodiment can adopt an asynchronous processing mechanism to improve the concurrent processing capability of the system. At the same time, in order to improve processing efficiency, this embodiment can also adopt batch processing technology to package multiple user question data and send them to the preset ultra-large-scale model for identification at one time, which can significantly reduce the overhead of API calls and improve the throughput of the system. In addition, in order to reduce the frequent calls to the preset ultra-large-scale model, this embodiment can introduce a cache mechanism to store the user question data and its results that have been identified in a cache such as Redis (a memory data structure storage), so that when the same question appears again, the result can be directly obtained from the cache to avoid repeated calls. In this way, this embodiment uses a preset ultra-large-scale model to re-identify the risks of user question data intercepted in the production environment to screen out the misjudgment data, so as to subsequently correct the misjudgment data and solve the misjudgment problem of the security model in complex scenarios, thereby improving the accuracy and robustness of the model; the concept of confidence is introduced in the judgment results output by the preset ultra-large-scale model to ensure the accuracy of the judgment; the system's concurrent processing capability and processing efficiency are improved through asynchronous processing mechanisms and batch processing technologies, which can significantly reduce the overhead of API calls and improve the system's throughput; the machine results of the user question data that have been judged are stored in the cache, so that when the same question appears again, the results can be directly obtained from the cache, reducing the frequent calls to the preset ultra-large-scale model.

[0060] Step S13: If the target question data exists, the data label of the target question data is corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so as to iteratively train the security model using the obtained expanded data set, and migrate the knowledge of the preset ultra-large-scale model to the security model through the preset knowledge distillation technology, and replace the security model in the production environment with the obtained target security model.

[0061] In this embodiment, if the discrimination results determine that the target question data contains misclassified data, the data label of the target question data is corrected and relabeled as "risk-free." Based on the corrected data and the preset ultra-large-scale model, the original dataset is augmented using a preset data augmentation technique. The security model is then iteratively trained using the augmented dataset. Knowledge from the preset ultra-large-scale model is transferred using a preset knowledge distillation technique, and the security model in the production environment is replaced with the resulting target security model. The augmented dataset includes the corrected data relabeled as "risk-free" and similar examples generated through data augmentation. It is understood that to ensure the efficiency and accuracy of label correction, this embodiment can implement the data label correction operation through an automated script. Furthermore, during the iterative training of the security model, techniques such as gradient accumulation and mixed precision training can be employed to optimize graphics memory usage, ensuring that a larger model can be trained within limited hardware resources, thereby accelerating computation and reducing graphics memory usage. In addition, the model fine-tuning task is regularly executed through the scheduled task scheduling system so that the latest expanded dataset is used for training in each iteration, ensuring that the model can be continuously optimized and adapted to the new data distribution.

[0062] It should be noted that in order to further enhance the generalization capability of the security model, this embodiment can also introduce a preset data augmentation technology to expand the original dataset. Specifically, the process of expanding the original dataset based on the corrected data and the preset ultra-large-scale model is as follows: calling the preset ultra-large-scale model to generate a number of newly added user question data; the newly added user question data is data whose semantic similarity with the corrected data meets a preset similarity condition and is risk-free; adding the corrected data and the newly added user question data to the original dataset to expand the original dataset. That is, the preset ultra-large-scale model is called through the preset interface to generate a number of newly added user question data that are semantically similar to the corrected data but are risk-free, and the corrected data and the newly added user question data are added to the original dataset to expand the original dataset. For example, if a question is mistakenly judged as high-risk, the preset ultra-large-scale model can generate multiple semantically similar but risk-free questions and add them to the training set. It is understandable that this embodiment can store the corrected data and the several newly added user question data in an efficient vector database that stores the original data set, such as ChromaDB (an open source vector database), which supports fast retrieval of high-dimensional vectors and can quickly find the closest knowledge items in a large-scale data set for subsequent automatic training iterations.

[0063] It should be further pointed out that the processing flow of iteratively training the security model using the obtained expanded data set is as follows: the obtained expanded data set is cleaned, and the cleaned expanded data set is divided into a training data set, a verification data set and a test data set according to a preset distribution ratio; a preset low-rank matrix is ​​introduced into the target layer of the security model, and the preset low-rank matrix is ​​initialized using randomly generated initial values ​​or weights in the security model; the preset low-rank matrix is ​​a matrix whose rank meets the preset low-rank judgment condition; the security model with other parameters frozen except the preset low-rank matrix is ​​iteratively trained using the training data set, and after each round of training, the security model is tuned using the verification data set; the performance of the iteratively trained security model is evaluated based on the test data set to obtain the security model whose performance meets the preset performance conditions after iterative training. The preset allocation ratio may be 70% training set, 15% validation set, and 15% test set; the validation dataset is used for model tuning; the test dataset is used for final performance evaluation; the optimization objective for iterative training of the security model may be minimizing the cross-entropy loss function; and evaluation metrics include, but are not limited to, accuracy, recall, and F1 score (a metric used to measure the accuracy of binary classification models). Specifically, the resulting augmented dataset is cleaned to remove noisy data and inconsistent labels. During the cleaning process, duplicate data is removed, text formatting is standardized (e.g., capitalization is unified, special characters are removed), and the accuracy of data labels is ensured. The cleaned augmented dataset is then divided into training, validation, and test datasets according to the preset allocation ratios. Low-Rank Adaptation (LoRA, a low-rank adapter algorithm) is then used to fine-tune the security model, updating only a small number of parameters. This significantly reduces training time and memory requirements. Specifically, a preset low-rank matrix is ​​introduced into the target layer of the security model, and the preset low-rank matrix is ​​initialized using randomly generated initial values ​​or weights extracted from the security model. Subsequently, most of the parameters in the security model except the preset low-rank matrix are frozen, and the security model is iteratively trained using the training data set. After each round of training, the security model is tuned using the verification data set, and the performance of the security model after iterative training is evaluated based on the test data set to obtain the security model whose performance after iterative training meets the preset performance conditions.It is understandable that in order to accelerate the training process, this embodiment can adopt a distributed training framework such as DeepSpeed ​​(a tool for distributed training of large-scale models) or Horovod (a distributed training framework) to distribute the training tasks to multiple GPUs (Graphics Processing Units) or nodes for parallel execution, which can significantly reduce the training time and improve the system throughput.

[0064] It should be noted that in this embodiment, the process for optimizing the security model using a preset knowledge distillation technique and deploying the resulting target security model is as follows: The preset ultra-large-scale model is invoked to generate corresponding soft labels for the user question data in the training dataset; the security model is distilled using the soft labels, and during the distillation training process, the smoothness of the soft labels is controlled using a preset temperature parameter to transfer the knowledge of the preset ultra-large-scale model to the security model; the distilled security model is evaluated using the validation dataset and the test dataset to obtain a target security model whose performance meets the preset performance requirements, and the target security model is used to replace the security model in the production environment. The soft labels are probability distributions output by the model, reflecting the model's confidence in each category. For example, for a binary classification problem (risk / no risk), the preset ultra-large-scale model outputs a probability distribution representing the probability of the question being "risky" or "no risk." That is, the preset ultra-large-scale model generates corresponding soft labels for the user question data in the training dataset, which contain rich semantic information. By learning these soft labels, the security model can better understand complex semantics. Subsequently, the soft labels are used to perform distillation training on the security model. During the distillation training process, since a higher temperature parameter can make the soft labels smoother, it helps the security model better learn the knowledge of the preset ultra-large-scale model. Therefore, the preset temperature parameter is used to control the smoothness of the soft labels to transfer the knowledge of the preset ultra-large-scale model to the security model. In addition, after each fine-tuning and distillation training, the validation dataset and the test dataset are used for evaluation to determine whether the target security model has achieved the performance that meets the preset performance requirements based on the evaluation results.

[0065] It is understandable that during the knowledge distillation process, this embodiment can use gradient clipping and learning rate scheduling techniques to prevent model overfitting and ensure training stability. At the same time, when using the target security model to replace the security model in the production environment, A / B testing (a web page optimization method) or shadow mode can be used for verification to ensure that the performance of the new model in actual applications meets expectations. In addition, after the target security model is deployed, this embodiment can continuously monitor its performance in the actual environment, collect user feedback and intercept logs, and promptly discover and correct model misjudgment problems to ensure the stability and reliability of the system. In this way, this embodiment uses high-quality data generated by the ultra-large-scale model to distill and train the small security model, so that it learns the discrimination ability of the ultra-large-scale model, can better understand complex semantics, improve the discrimination ability of the security model, and reduce the occurrence of misjudgments; by expanding the original data set through data enhancement technology, the generalization ability of the security model can be further improved.

[0066] Step S14: construct a preference pair dataset based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and use the preset reinforcement learning algorithm and the preference pair dataset to perform value alignment training on the industry big model to control the generated content of the industry big model to meet the preset value concept.

[0067] In this embodiment, a high-quality preference pair dataset is constructed based on the user feedback data in the database to be processed and the pre-set ultra-large-scale model, combining manual annotation and automatic scoring mechanisms. The industry-wide model is then trained on value alignment using a pre-set reinforcement learning algorithm and the preference pair dataset to ensure that the generated content of the industry-wide model meets pre-set value concepts. The user feedback data includes, but is not limited to, likes, dislikes, and modification suggestions, and can directly reflect user preferences for model-generated content. The pre-set reinforcement learning algorithm includes, but is not limited to, Direct Preference Optimization (DPO) or Proximal Policy Optimization (PPO).

[0068] It should be pointed out that the processing flow of constructing a preference pair dataset based on the user feedback data in the database to be processed and the preset ultra-large-scale model is as follows: obtain the user feedback data in the database to be processed; generate several new question contents whose contents meet the preset similarity conditions with the target question contents of the user feedback data through the preset ultra-large-scale model; use the preset ultra-large-scale model to generate several answers corresponding to the target question contents and the several new question contents, and determine the first target answer and the second target answer from the several answers through the preset scoring mechanism; the first target answer is the answer with the highest score among the several answers; the second target answer is the answer with the lowest score among the several answers; construct a preference pair dataset based on the target question content, the first target answer and the second target answer corresponding to the target question content, the several new question contents, and the first target answer and the second target answer corresponding to the several new question contents. Among them, the preset scoring mechanism includes a manual scoring mechanism and an automatic scoring mechanism; the manual scoring mechanism is a mechanism for scoring by a professional annotation team according to preset value standards to ensure the accuracy and authority of the preference pair data; the preference pair data format in the preference pair dataset is a triple,<prompt, chosen, rejected> ; prompt refers to user question data; chosen is the first target answer; rejected is the second target answer. That is, obtain the user feedback data pulled from the production environment and stored in the database to be processed, and generate several new question contents whose contents meet the preset similarity conditions with the target question contents of the user feedback data through the preset ultra-large-scale model, as well as several answers corresponding to the target question contents and the several new question contents, and then form preference pair data from the several answers with the first target answer with the highest score and the second target answer with the lowest score according to the preset scoring mechanism, and then construct a preference pair data set based on the formed preference pair data. It can be understood that after constructing the preference pair data set, this embodiment can also regularly pull new user feedback data from the production environment, combine manual labeling and automatic scoring mechanisms, dynamically update the preference pair data set, and continuously optimize the model to ensure that the industry large model can adapt to the latest user needs and preset values.

[0069] It should be further noted that the process for using the preset reinforcement learning algorithm and the preference pair dataset to perform value alignment training on the industry model is as follows: using a direct preference optimization algorithm to perform value alignment training on the industry model using a preset likelihood function and the preference pair dataset; or using the preference pair dataset to train a preset reward model, performing value alignment training on the industry model based on a proximal policy optimization algorithm, and adjusting the value alignment training strategy during the value alignment training process based on the reward score output by the preset reward model. The preset likelihood function can be a function that maximizes the likelihood of preference pairs; and the preset reward model can score model-generated content based on human feedback. In other words, this embodiment can use a direct preference optimization algorithm to perform value alignment training on the industry model using a preset likelihood function and the preference pair dataset, avoiding the need to train an independent reward model. By directly optimizing the model's generated content preferences, the generated content can be ensured to conform to the preset value concepts. For example, the learning rate can be set to 1 / 10 of the learning rate used during pre-training (e.g., 7e-5), the temperature parameter can be set between 0.1 and 1.0, and the training rounds can be set to 3-5 epochs. This embodiment can also use the preference pair data set to train a preset reward model, and perform value alignment training on the industry large model based on the proximal policy optimization algorithm, and maximize the reward by optimizing the policy function. For example, a clipping objective function is introduced to prevent the policy from being updated too much, ensuring the stability of the training so that the content generated by it can obtain a higher reward. For example, the learning rate can be set to 5e-5, the clipping parameter can be set to 0.2, and the training rounds can be set to 3-5 epochs. In the process of value alignment training, the value alignment training strategy is adjusted according to the reward score output by the preset reward model. In this way, this embodiment constructs a preference pair data set and uses a preset reinforcement learning algorithm to perform value alignment training on the industry large model. It can effectively prevent the generation of inappropriate or harmful content through appropriate handling and positive guidance of sensitive topics, help create a healthy and harmonious network environment, promote the healthy development of artificial intelligence technology, and protect users from the influence of bad content.

[0070] As can be seen from the above, the embodiment of the present application pulls user interaction data in the industry large-scale model production environment based on the preset data pulling frequency, and stores it in the database to be processed after cleaning, so as to call the preset ultra-large-scale model to perform risk judgment on the question data to be judged in the database to be processed, and determine whether there is misjudged target question data based on the obtained judgment result. If so, the data label of the target question data is corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so as to use the obtained expanded data set to iteratively train the security model, and migrate the knowledge of the preset ultra-large-scale model to the security model through the preset knowledge distillation technology, and replace the security model in the production environment with the obtained target security model. At the same time, a preference pair data set is constructed based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and the preset reinforcement learning algorithm and the preference pair data set are used to perform value alignment training on the industry large model to control the generated content of the industry large model to meet the preset value concept.In this way, through the above process of the embodiment of the present application, on the one hand, user interaction data is pulled from the production environment of the industry large model based on the preset data pulling frequency, and saved to the database to be processed after cleaning, so that subsequent misjudgment correction and value alignment processing can be performed based on the user interaction data, thereby improving the content security of the industry large model, effectively identifying and intercepting risky inputs, and optimizing the user experience to ensure that responses that conform to human values ​​are provided under the premise of ensuring security; on the one hand, an exception handling mechanism is introduced in the processing flow of security capability construction to ensure automatic retry or error log recording when encountering abnormal situations to avoid data loss or processing interruption; on the one hand, the preset ultra-large-scale model is used to re-identify the risk of user question data intercepted in the production environment to filter out the misjudgment data, so that the misjudgment data can be corrected later to solve the misjudgment problem of the security model in complex scenarios, thereby improving the accuracy and robustness of the model; on the one hand, the concept of confidence is introduced in the judgment result output by the preset ultra-large-scale model to ensure the accuracy of the judgment; on the one hand, asynchronous processing mechanism and batch processing technology are used to improve the accuracy of the judgment. The high concurrent processing capability and processing efficiency of the system can significantly reduce the overhead of API calls and improve the throughput of the system. On the one hand, the machine results of the user question data that have been judged are stored in the cache, so that when the same question appears again, the results can be directly obtained from the cache, reducing the frequent calls to the preset ultra-large-scale model. On the other hand, the high-quality data generated by the ultra-large-scale model is used to perform distillation training on the small security model, so that it can learn the discrimination ability of the ultra-large-scale model, better understand complex semantics, improve the discrimination ability of the security model, and reduce the occurrence of misjudgment. On the one hand, the original data set is expanded through data enhancement technology to further enhance the generalization ability of the security model. On the other hand, by constructing a preference pair data set and using the preset reinforcement learning algorithm to perform value alignment training on the industry large model, it can effectively prevent the generation of inappropriate or harmful content through proper handling and positive guidance of sensitive topics, help create a healthy and harmonious network environment, promote the healthy development of artificial intelligence technology, protect users from the influence of bad content, and thus improve the interception accuracy of the security model and ensure the security of the content generated by the industry large model.

[0071] Accordingly, see Figure 3 As shown, the embodiment of the present application also provides a content security capability building device for an industry large model, including:

[0072] A data cleaning module 11 is configured to pull user interaction data from a production environment of an industry large model based on a preset data pulling frequency, and clean the user interaction data to store the cleaned user interaction data in a database to be processed;

[0073] The risk identification module 12 is configured to call a preset ultra-large-scale model to perform risk identification on the question data to be identified in the database to be processed, so as to determine whether there is any misidentified target question data in the question data to be identified based on the identification result; the question data to be identified is the user question data intercepted by the security model in the user interaction data; the security model is a model for performing risk identification on user questions input into the industry large model; the preset ultra-large-scale model is a model whose model scale meets the preset ultra-large-scale identification conditions;

[0074] a data set expansion module 13 configured to, if the target question data exists, correct the data labels of the target question data, and expand the original data set based on the corrected data and the preset ultra-large-scale model, so as to iteratively train the security model using the obtained expanded data set, migrate the knowledge of the preset ultra-large-scale model to the security model using a preset knowledge distillation technique, and replace the security model in the production environment with the obtained target security model;

[0075] The model training module 14 is used to construct a preference pair data set based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and use the preset reinforcement learning algorithm and the preference pair data set to perform value alignment training on the industry large model to control the generated content of the industry large model to meet the preset value concept.

[0076] As can be seen from the above, the embodiment of the present application pulls user interaction data in the industry large-scale model production environment based on the preset data pulling frequency, and stores it in the database to be processed after cleaning, so as to call the preset ultra-large-scale model to perform risk judgment on the question data to be judged in the database to be processed, and determine whether there is misjudged target question data based on the obtained judgment result. If so, the data label of the target question data is corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so as to use the obtained expanded data set to iteratively train the security model, and migrate the knowledge of the preset ultra-large-scale model to the security model through the preset knowledge distillation technology, and replace the security model in the production environment with the obtained target security model. At the same time, a preference pair data set is constructed based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and the preset reinforcement learning algorithm and the preference pair data set are used to perform value alignment training on the industry large model to control the generated content of the industry large model to meet the preset value concept. In this way, through the above process of the embodiment of the present application, user interaction data in the production environment of the industry big model is regularly pulled, and the security model is iteratively trained based on the question data to be judged in the user interaction data and data enhancement technology, which can improve the accuracy and robustness of the security model. At the same time, the industry big model is value-aligned through user feedback data to ensure that even if the security small model fails to accurately identify the risk problem, the industry big model will not generate content that does not conform to the preset values, thereby improving the interception accuracy of the security model and ensuring the security of the content generated by the industry big model.

[0077] In some specific implementations, the risk identification module 12 may specifically include:

[0078] A risk identification unit, configured to perform risk identification on the question data to be identified in the database to be processed by calling a preset ultra-large-scale model through a preset interface, so as to obtain an identification result output by the preset ultra-large-scale model;

[0079] A condition judgment unit, configured to judge whether the confidence level in the judgment result is higher than a preset confidence threshold;

[0080] The data determining unit is configured to determine that the question data to be determined is misjudged target question data if the confidence level in the discrimination result is higher than a preset confidence threshold and the discrimination result indicates that the question data to be determined is risk-free data.

[0081] In some specific implementations, the data set expansion module 13 may specifically include:

[0082] A data generation unit is configured to call the preset ultra-large-scale model to generate a plurality of newly added user question data; the newly added user question data is risk-free data whose semantic similarity with the corrected data satisfies a preset similarity condition;

[0083] The data adding unit is used to add the corrected data and the plurality of newly added user question data to the original data set to expand the original data set.

[0084] In some specific implementations, the data set expansion module 13 may specifically include:

[0085] A data partitioning unit is used to clean the obtained expanded data set and divide the cleaned expanded data set into a training data set, a validation data set and a test data set according to a preset distribution ratio;

[0086] A weight initialization unit, configured to introduce a preset low-rank matrix into the target layer of the security model and initialize the preset low-rank matrix using randomly generated initial values ​​or weights in the security model; the preset low-rank matrix is ​​a matrix whose rank satisfies a preset low-rank determination condition;

[0087] a model training unit, configured to iteratively train the security model with all parameters frozen except for the preset low-rank matrix using the training data set, and to fine-tune the security model using the validation data set after each round of training;

[0088] A performance evaluation unit is used to perform performance evaluation on the security model after iterative training based on the test data set to obtain the security model whose performance after iterative training meets preset performance conditions.

[0089] In some specific implementations, the data set expansion module 13 may specifically include:

[0090] A label generation unit, configured to call the preset ultra-large-scale model to generate corresponding soft labels for the user question data in the training data set;

[0091] a distillation training unit, configured to perform distillation training on the security model using the soft labels, and to control the smoothness of the soft labels using a preset temperature parameter during the distillation training process, so as to transfer knowledge of the preset ultra-large-scale model to the security model;

[0092] A model evaluation unit is used to evaluate the security model after distillation training using the verification data set and the test data set to obtain a target security model whose performance meets the preset performance conditions, and to replace the security model in the production environment with the target security model.

[0093] In some specific implementations, the model training module 14 may specifically include:

[0094] A data acquisition unit, configured to acquire user feedback data from the database to be processed;

[0095] A content generating unit, configured to generate, through the preset ultra-large-scale model, a plurality of new question contents whose contents meet a preset similarity condition with the target question contents of the user feedback data;

[0096] an answer determination unit, configured to generate, using the preset ultra-large-scale model, a plurality of answers corresponding to the target question content and the plurality of newly added question contents, respectively, and determine, from the plurality of answers, a first target answer and a second target answer using a preset scoring mechanism; the first target answer being the answer with the highest score among the plurality of answers; and the second target answer being the answer with the lowest score among the plurality of answers;

[0097] A data set construction unit is used to construct a preference pair data set based on the target question content, the first target answer and the second target answer corresponding to the target question content, the several new question contents, and the first target answers and the second target answers corresponding to the several new question contents.

[0098] In some specific implementations, the model training module 14 may specifically include:

[0099] A first alignment training unit is configured to perform value alignment training on the industry model using a direct preference optimization algorithm by using a preset likelihood function and the preference pair dataset;

[0100] Or, a second alignment training unit is used to use the preference to train a preset reward model for the data set, perform value alignment training on the industry large model based on the proximal strategy optimization algorithm, and during the value alignment training process, adjust the value alignment training strategy according to the reward score output by the preset reward model.

[0101] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the content security capability construction method of the industry big model disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment can specifically be an electronic computer.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0103] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to implement the content security capability building method of the industry large model executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to perform other specific tasks.

[0105] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for building content security capabilities for a large industry model. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0107] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0110] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for building content security capabilities for a large industry model, characterized by: include: Pulling user interaction data from a production environment of a large industry model based on a preset data pulling frequency, and cleaning the user interaction data to store the cleaned user interaction data in a database to be processed; Calling a preset ultra-large-scale model to perform risk discrimination on the question data to be judged in the database to be processed, so as to determine whether there is any misjudged target question data in the question data to be judged according to the obtained discrimination result; The question data to be determined is the user question data intercepted by the security model in the user interaction data; The security model is a model for performing risk assessment on user questions input into the industry large model; the preset ultra-large-scale model is a model whose scale satisfies the preset ultra-large-scale assessment condition; If the target question data exists, the data label of the target question data is corrected, and the original data set is expanded based on the corrected data and the preset ultra-large-scale model, so that the security model is iteratively trained using the obtained expanded data set, and the knowledge of the preset ultra-large-scale model is transferred to the security model through a preset knowledge distillation technology, and the security model in the production environment is replaced with the obtained target security model; A preference pair dataset is constructed based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and the industry large model is trained for value alignment using a preset reinforcement learning algorithm and the preference pair dataset to control the generated content of the industry large model to meet the preset value concept.

2. The method for building content security capabilities of an industry large model according to claim 1 is characterized in that: The calling of the preset ultra-large-scale model to perform risk identification on the question data to be identified in the database to be processed, so as to determine whether there is misidentified target question data in the question data to be identified according to the identification result, includes: Calling a preset ultra-large-scale model through a preset interface to perform risk discrimination on the question data to be discriminated in the database to be processed, so as to obtain a discrimination result output by the preset ultra-large-scale model; Determining whether the confidence level of the judgment result is higher than a preset confidence threshold; If the confidence level in the discrimination result is higher than a preset confidence threshold, and the discrimination result indicates that the question data to be discriminated is risk-free data, then the question data to be discriminated is determined to be misjudged target question data.

3. The method for building content security capabilities of an industry large model according to claim 1 is characterized in that: The expanding the original data set based on the corrected data and the preset ultra-large-scale model includes: The preset ultra-large-scale model is called to generate a number of newly added user question data; the newly added user question data is data whose semantic similarity with the corrected data meets a preset similarity condition and is risk-free; The corrected data and the newly added user question data are added to the original data set to expand the original data set.

4. The method for building content security capabilities of an industry large model according to claim 1 is characterized in that: The iterative training of the security model using the obtained expanded data set includes: Cleaning the obtained expanded data set, and dividing the cleaned expanded data set into a training data set, a validation data set, and a test data set according to a preset distribution ratio; Introducing a preset low-rank matrix in the target layer of the security model, and initializing the preset low-rank matrix using randomly generated initial values ​​or weights in the security model; the preset low-rank matrix is ​​a matrix whose rank satisfies a preset low-rank determination condition; Iteratively training the security model with all parameters frozen except for the preset low-rank matrix using the training data set, and after each round of training, fine-tuning the security model using the validation data set; A performance evaluation is performed on the security model after iterative training based on the test data set to obtain the security model whose performance after iterative training meets preset performance conditions.

5. The method for building content security capabilities of an industry large model according to claim 4 is characterized in that: The step of migrating the knowledge of the preset ultra-large-scale model to the security model by using the preset knowledge distillation technology, and replacing the security model in the production environment with the obtained target security model, includes: Calling the preset ultra-large-scale model to generate corresponding soft labels for the user question data in the training data set; Performing distillation training on the security model using the soft labels, and controlling the smoothness of the soft labels using a preset temperature parameter during the distillation training process, so as to transfer knowledge of the preset ultra-large-scale model to the security model; The security model after distillation training is evaluated using the verification data set and the test data set to obtain a target security model whose performance meets the preset performance conditions, and the target security model is used to replace the security model in the production environment.

6. The method for building content security capabilities of an industry large model according to claim 1 is characterized in that: The step of constructing a preference pair dataset based on the user feedback data in the to-be-processed database and the preset ultra-large-scale model includes: Obtain user feedback data from the database to be processed; Generating, by the preset ultra-large-scale model, several new question contents whose contents meet preset similarity conditions with the target question contents of the user feedback data; Using the preset ultra-large-scale model, a plurality of answers corresponding to the target question content and the plurality of newly added question contents are generated respectively, and a first target answer and a second target answer are determined from the plurality of answers through a preset scoring mechanism; the first target answer is the answer with the highest score among the plurality of answers; and the second target answer is the answer with the lowest score among the plurality of answers; A preference pair data set is constructed based on the target question content, the first target answer and the second target answer corresponding to the target question content, the several newly added question contents, and the first target answers and the second target answers corresponding to the several newly added question contents.

7. The method for building content security capabilities of an industry large model according to any one of claims 1 to 6, characterized in that: The use of a preset reinforcement learning algorithm and the preference pair dataset to perform value alignment training on the industry model includes: Using a direct preference optimization algorithm, the industry model is trained on value alignment using a preset likelihood function and the preference pair dataset; Alternatively, the preference is used to train a preset reward model for the data set, and the industry large model is trained for value alignment based on the proximal strategy optimization algorithm. During the value alignment training process, the value alignment training strategy is adjusted according to the reward score output by the preset reward model.

8. A content security capability building device for a large industry model, characterized in that: include: A data cleaning module is used to pull user interaction data in the production environment of the industry large model based on a preset data pulling frequency, and clean the user interaction data to store the cleaned user interaction data in the database to be processed; a risk identification module, configured to call a preset ultra-large-scale model to perform risk identification on the question data to be identified in the database to be processed, so as to determine whether there is any misidentified target question data in the question data to be identified based on the identification result; The question data to be determined is the user question data intercepted by the security model in the user interaction data; The security model is a model for performing risk assessment on user questions input into the industry large model; the preset ultra-large-scale model is a model whose scale satisfies the preset ultra-large-scale assessment condition; a data set expansion module, configured to, if the target question data exists, correct the data labels of the target question data, and expand the original data set based on the corrected data and the preset ultra-large-scale model, so as to iteratively train the security model using the obtained expanded data set, migrate the knowledge of the preset ultra-large-scale model to the security model through a preset knowledge distillation technique, and replace the security model in the production environment with the obtained target security model; The model training module is used to construct a preference pair dataset based on the user feedback data in the database to be processed and the preset ultra-large-scale model, and use the preset reinforcement learning algorithm and the preference pair dataset to perform value alignment training on the industry large model to control the generated content of the industry large model to meet the preset value concept.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the content security capability building method of the industry big model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, it implements the content security capability building method of the industry big model as described in any one of claims 1 to 7.

Images