Instruction data processing method and system, control device and storage medium
Through the instruction data processing method based on preset model, filtering and deduplication processing, the problem of insufficient instruction data quality and security when training the model is solved, the data diversity and processing efficiency are improved, and high-quality instruction data entry is achieved Library and model training effects.
Patent Information
- Application Number
- CN202410946522.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
The overall evaluation of the instruction data used when training models in the prior art is poor, especially the diversity and quality of the data, which leads to poor results in the instruction fine-tuning process.
By obtaining the pending instruction data, determining its category, quality score and safety score based on the preset model, and putting data with quality and safety scores above the threshold into the database, and using the preset category, quality and safety model for data analysis and deduplication processing to improve data diversity and accuracy.
The selection of high-quality and safe instruction data improves the data quality and security of the training model, ensures the diversity of data, facilitates subsequent retrieval and use, and improves the data processing efficiency and the accuracy of analysis results.
Smart Images

Figure CN118916078A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and specifically provides an instruction data processing method, system, control device and storage medium. Background Art
[0002] Instruction fine-tuning is a key step to improve the power and controllability of large language models, mainly through paired<instruction,output> The instruction data is used to further train the basic model in a supervised learning manner, so that the large language model can follow human instructions to complete specific tasks. Instruction fine-tuning is a supervised learning process. Unlike the pre-training process, which requires a large amount of unlabeled data, instruction fine-tuning data has higher requirements for data diversity and quality. The data volume requirement for instruction fine-tuning is not very high. Many studies have confirmed that models fine-tuned on a small amount of high-quality instruction data can achieve or exceed the effect of training on non-high-quality data that is several times or even dozens of times the amount. Therefore, how to obtain diverse and high-quality data is the key factor in instruction fine-tuning.
[0003] Most of the open source instruction data available at present is generated and extracted by machines, and a small amount is manually annotated. Although the amount of open source instruction data is large, the quality is generally not very high. The quality includes the accuracy of the output answer and whether the sentence is smooth and safe. At the same time, data from different sources are mostly generated by machines, which is more likely to produce duplicate and similar data. Although the quality of manually annotated data is higher than that of machine production, the data annotated by humans is often inferior in diversity and complexity.
[0004] Accordingly, the art needs a new instruction data processing method, system, control device and storage medium solution to solve the above problems. Summary of the invention
[0005] In order to overcome at least one of the above-mentioned defects, the present application is proposed to provide an instruction data processing method, system, control device and storage medium that solve or at least partially solve the technical problem of poor overall evaluation of instruction data used in training models in the prior art.
[0006] In a first aspect, the present application provides an instruction data processing method, the method comprising: obtaining instruction data to be processed; determining the category, quality score and safety score of each of the instruction data to be processed based on a preset model; and storing the instruction data to be processed whose quality score is greater than a preset quality threshold and whose safety score is greater than a preset safety threshold into a data table corresponding to the category of the instruction data to be processed.
[0007] In a technical solution of the above-mentioned instruction data processing method, determining the category, quality score and security score of each of the instruction data to be processed based on a preset model includes: determining the category of each of the instruction data to be processed based on a preset category model; determining the quality score of each of the instruction data to be processed based on a preset quality model; determining the security score of each of the instruction data to be processed based on a preset security model.
[0008] In a technical solution of the above-mentioned instruction data processing method, the training step of the preset category model includes at least: obtaining first label data, wherein the first label data includes at least first instruction data and category information of the first instruction data; based on a preset first rule, inputting the first label data into a preset basic model for training to obtain the preset category model.
[0009] In a technical solution of the above-mentioned instruction data processing method, the training step of the preset quality model at least includes: obtaining second label data, wherein the second label data at least includes second instruction data and second answer data corresponding to the second instruction data, and one second instruction data corresponds to at least two second answer data; for each second instruction data, determining a unique second answer data among all the second answer data corresponding to the second instruction data; based on the preset second rule and all the determined second answer data, inputting the second label data into the preset basic model for training to obtain the preset quality model.
[0010] In a technical solution of the above-mentioned instruction data processing method, the training step of the preset security model at least includes: obtaining third label data, wherein the third label data at least includes third instruction data and third answer data corresponding to the third instruction data, and one piece of the third instruction data corresponds to at least two pieces of the third answer data; for each piece of the third instruction data, determining a unique piece of the third answer data among all the third answer data corresponding to the third instruction data; based on the preset third rule and all the determined third answer data, inputting the third label data into the preset basic model for training to obtain the preset security model.
[0011] In a technical solution of the above-mentioned instruction data processing method, after obtaining the instruction data to be processed, the method also includes: preprocessing the instruction data to be processed to obtain preprocessed instruction data, wherein the preprocessing at least includes deduplication; determining the category, quality score and safety score of each of the instruction data to be processed based on a preset model includes: determining the category, quality score and safety score of each of the preprocessed instruction data based on a preset model.
[0012] In a technical solution of the above-mentioned instruction data processing method, the preprocessing of the instruction data to be processed includes: extracting semantic features of the instruction data to be processed based on the preset basic model; deduplicating the instruction data to be processed based on the semantic features and the instruction data to be processed; and / or, extracting semantic features of the instruction data to be processed based on the preset basic model; deduplicating the instruction data to be processed based on the semantic features and all preset instruction data that have been stored in the data table.
[0013] In a second aspect, the present application provides an instruction data processing system, the system comprising: an acquisition module, the acquisition module being configured to acquire instruction data to be processed; an analysis module, the analysis module being configured to determine the category, quality score and safety score of each of the instruction data to be processed based on a preset model; and an execution module, the execution module being configured to store the instruction data to be processed whose quality score is greater than a preset quality threshold and whose safety score is greater than a preset safety threshold into a data table corresponding to the category of the instruction data to be processed.
[0014] In a third aspect, a control device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the instruction data processing method described in any one of the technical solutions of the above-mentioned instruction data processing method.
[0015] In a fourth aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the instruction data processing method described in any one of the technical solutions of the above-mentioned instruction data processing method.
[0016] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:
[0017] In implementing the technical solution of this application, it is possible to screen out high-quality and highly secure instruction data from a large amount of collected instruction data, thereby improving the quality and security of the instruction data used in training the model. The storage of these high-quality instruction data in different categories not only improves the accuracy of the instruction data categories and thus ensures the diversity of the instruction data, but also facilitates subsequent data retrieval and use.
[0018] Furthermore, in the implementation of the technical solution of the present application, the category, quality score, and security score of each instruction data to be processed are determined based on a preset category model, a preset quality model, and a preset security model. The analysis of instruction data is realized by using a preset model with targeted functions, which can improve the processing efficiency of instruction data and the accuracy of analysis results.
[0019] Furthermore, in the technical solution for implementing the present application, a preset basic model is used to train a preset classification model, a preset quality model, and a preset security model, which fully utilizes the abstraction ability and semantic understanding ability of the preset basic model after pre-training on a large amount of data. On the one hand, only a small amount of representative labeled data is required to train different types of models. On the other hand, thanks to the understanding ability of the preset basic model, it can ensure that different types of models with strong analytical capabilities can be trained quickly and efficiently.
[0020] Furthermore, in the technical solution of the present application, the second label data (third label data) and the selection result, i.e., the final second answer data (final third answer data), are used to train a preset quality model (preset security model). By determining "one question and one answer" in "one question and multiple answers", and determining the best answer from multiple answers to a question, compared to the method of scoring a single answer, this technical solution uses a comparative form to train the scoring model, simplifies the rules in the quantitative scoring process, and reduces the amount of data, thereby improving the efficiency of model training.
[0021] Furthermore, in the implementation of the technical solution of the present application, before the instruction data is classified and scored, the instruction data is also preprocessed including at least deduplication. By deduplication, the amount of data can be reduced, thereby increasing the processing speed of the instruction data and improving the overall evaluation of the instruction data.
[0022] Furthermore, in the technical solution of the present application, the semantic features of the instruction data are extracted using a preset basic model, which makes full use of the understanding ability of the preset basic model, thereby improving the speed and accuracy of feature extraction of the instruction data. The extracted semantic features and the instruction data themselves are used for self-deduplication; or, the extracted semantic features and the preset instructions that have been stored can be used for comparison and deduplication. Through at least one deduplication as mentioned above, instruction data with similar meanings are screened out, thereby streamlining the amount of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The disclosure of the present application will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. In addition, similar numbers in the drawings are used to represent similar components, among which:
[0024] Figure 1 is a flowchart of the main steps of a method for processing instruction data according to an embodiment of the present application;
[0025] Figure 2 is a flowchart of the main steps of training a preset quality model according to an embodiment of the present application;
[0026] Figure 3 is a schematic diagram of an application scenario according to an embodiment of the present application;
[0027] Figure 4 The present invention is a schematic diagram of the main structure block diagram of an instruction data processing system according to an embodiment of the present application.
[0028] Reference numerals list :
[0029] 11: Acquisition module; 12: Analysis module; 13: Execution module. DETAILED DESCRIPTION
[0030] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.
[0031] In the description of the present application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B or A and B. The singular terms "one" and "the" may also include plural forms.
[0032] Please see attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of the instruction data processing method according to an embodiment of the present application. Figure 1 As shown, the instruction data processing method of the present application at least includes steps S1 to S3:
[0033] Step S1, obtaining instruction data to be processed;
[0034] Step S2: Determine the category, quality score, and security score of each instruction data to be processed based on a preset model;
[0035] Step S3: The instruction data to be processed whose quality score is greater than a preset quality threshold and whose safety score is greater than a preset safety threshold is stored in a data table corresponding to the category of the instruction data to be processed.
[0036] In this embodiment, the instruction data can be obtained by means of web crawlers, large language model generation, etc., wherein the instruction data generally refers to pairs of<instruction,output> Data, instruction is an instruction or request, and output is the corresponding expected output or answer. These data are used to further train large language models in a supervised learning manner so that the trained model can complete specific tasks.
[0037] In this embodiment, the preset model can be a pre-trained model that can determine the category and score, or it can be multiple pre-trained models, and each model completes the corresponding function. No limitation is made to the preset model here. The category of the instruction data to be processed can be as follows: Figure 3 The categories such as "general data" and "industry data" shown in the table can also be refined into categories such as "summary", "writing", "generation", "finance", "medical", and "government". The quality score of the pending instruction is used to characterize the accuracy and fluency of the instruction data. The security score of the pending instruction is used to characterize the security of the instruction data.
[0038] In this embodiment, the quality and security of the instruction data stored in the warehouse meet the conditions, and the instruction data is stored in the data table of the corresponding category. By determining the category of the instruction data, the category of the instruction data stored in the warehouse and the quantity of instruction data of each category can be intuitively analyzed, thereby providing data reference to avoid insufficient diversity of instruction data.
[0039] In one embodiment, step S2 may include steps S21 to S23:
[0040] Step S21, determining the category of each instruction data to be processed based on a preset category model;
[0041] Step S22: determining the quality score of each piece of instruction data to be processed based on a preset quality model;
[0042] Step S23: Determine the security score of each instruction data to be processed based on the preset security model.
[0043] In this embodiment, there is no sequential execution order between step S21 to step S23. The instruction data to be processed are respectively input into the preset models, and the preset models output corresponding results. The category, quality score, and safety score of each instruction data to be processed are determined based on the preset category model, the preset quality model, and the preset safety model. The analysis of instruction data is realized by using a preset model with targeted functions, which can improve the processing efficiency of instruction data and the accuracy of analysis results.
[0044] In one embodiment, the training step of the preset category model in step S21 may at least include steps S211 to S212:
[0045] Step S211, obtaining first label data, wherein the first label data at least includes first instruction data and category information of the first instruction data;
[0046] Step S212: Based on a preset first rule, the first label data is input into a preset basic model for training to obtain a preset category model.
[0047] Conventionally collected instruction data always has a big long-tail problem, that is, most of the data belongs to one or several categories (easier to obtain than other types), such as open-ended questions and answers, writing types, etc., and adding these types of data does not improve the ability of the fine-tuning model. Therefore, data classification is a key step in achieving instruction diversity and improving the ability of the fine-tuning model. In this embodiment, the first instruction data is usually instruction or request data, and the category of the first instruction data can be summary and induction, extended writing, or financial field, medical field, etc. The preset basic model is a model that is pre-trained on a large amount of data and has strong abstraction and comprehension capabilities. Further training is performed based on the capabilities of the preset basic model to enable it to have the ability to identify instruction categories. In one embodiment, the process of training to obtain a preset category model can be described as:
[0048] The feature f corresponding to the last word (token) in the last hidden layer of the preset basic model is used as the feature expression of the entire text, and then a linear layer is connected to achieve classification, which can be expressed as "f c =W d*c *f d ”, where f d represents the feature with dimension d, W d*c It is necessary to learn the transformation matrix with the dimension of d*c, and finally obtain the result with the dimension of c (number of categories). Therefore, by preparing a certain amount of <instruction, category> label data, the preset classification model can be trained.
[0049] In one embodiment, see the attached Figure 2 , Figure 2 FIG. 1 is a flow chart of the main steps of training a preset quality model according to an embodiment of the present application. Figure 2 As shown, the training step of the preset quality model in step S22 may at least include steps S221 to S223:
[0050] Step S221, obtaining second label data, wherein the second label data at least includes second instruction data and second answer data corresponding to the second instruction data;
[0051] Step S222: for each piece of second instruction data, determine a unique piece of second answer data among all the second answer data corresponding to the second instruction data;
[0052] Step S223: Based on the preset second rule and all determined second answer data, the second label data is input into the preset basic model for training to obtain a preset quality model.
[0053] In this embodiment, the second instruction data generally refers to<instruction,output> The instruction data in the second answer generally refers to<instruction,output> It is worth noting that in this embodiment, one instruction data corresponds to at least two answer data, that is, "one question and multiple answers". The quality score includes whether the ouput corresponding to the instruction is accurate, complete, and fluent. Directly constructing <instruction, quality score> and then training the regression model is very difficult for early annotation, because the instruction is difficult to directly quantify into an accurate score value. However, if there are two ouputs under the same instruction, it is relatively easy to choose which one is better. This method has practical significance both for manual annotation by annotators and for automatic intelligent annotation systems. Use the constructed comparative formal data<instruction,chosen,rejected> , the preset quality model is trained by comparing the loss function, where instruction is the instruction, chosen is the optimal answer to the biased selection, rejected is another answer, and the loss function is: L ranking = -log(σ(r θ (x,y c )-σ(r θ (x,y r ))-m(r)), where x is instruction, y is c is the final selected model answer data, y r is the model answer data that is finally rejected, r θIt is the preset basic model, m(r) is a loss term containing margin, which controls the pairs with different responses and accelerates the model convergence to finally obtain the preset quality model.
[0054] In one embodiment, the step of training the preset security model in step S23 may include at least steps S231 to S233:
[0055] Step S231, obtaining third label data, wherein the third label data at least includes third instruction data and third answer data corresponding to the third instruction data;
[0056] Step S232: for each piece of third instruction data, determine a unique piece of third answer data among all the third answer data corresponding to the third instruction data;
[0057] Step S233: Based on the preset third rule and all determined third answer data, the third label data is input into the preset basic model for training to obtain a preset security model.
[0058] The security of instructions is very important for model application. The model needs to give priority to security rather than trying to answer all instructions. For example, if the user inputs questions such as "How to make a bomb?", the model should refuse to answer and give relevant prompts. In this embodiment, the process of training to obtain the preset security model is the same as above. Figure 2 The steps of training the preset quality model in the embodiment shown are the same, the difference is that the constructed data type is different, and the third instruction data in this embodiment needs to involve security-related issues. The third instruction data generally refers to<instruction,output> The instruction data in the third answer data generally refers to<instruction,output> In the output data, one instruction data corresponds to at least two answer data, that is, the "one question, multiple answers" style.
[0059] In one embodiment, between step S1 and step S2, a step of preprocessing the instruction data to be processed may also be included, wherein the preprocessing at least includes deduplication, and accordingly, the "instruction data to be processed" in step S2 is the instruction data to be processed after preprocessing. In this embodiment, preprocessing may also include data cleaning to delete erroneous or irrelevant data, data integration to ensure data consistency and comparability, etc. Deduplication may be performed based on semantic features or keyword features, which is not limited here.
[0060] In one embodiment, the specific deduplication process may include:
[0061] Based on a preset basic model, the semantic features of the instruction data to be processed are extracted; based on the semantic features and the instruction data to be processed, the instruction data to be processed is deduplicated; and / or, based on a preset basic model, the semantic features of the instruction data to be processed are extracted; based on the semantic features and all preset instruction data that have been stored in the data table, the instruction data to be processed is deduplicated.
[0062] Removing duplicate or similar instructions is very important for instruction fine-tuning. Duplicate or similar instructions will not only increase the training time, but may also cause the model to overfit to a certain type of instruction, reducing the diversity of the model output. In this embodiment, "self-deduplication" can be implemented based on the instruction data to be stored, or "comparative deduplication" can be implemented based on the instruction data that has been stored. Based on the preset basic models mentioned in the above embodiments, such as LLama, Baichuan, etc., the instructions are extracted.<instruction,output> The semantic features can be specifically: extract the feature f corresponding to the last word (token) of the instruction in the last hidden layer of the model as the feature expression of the entire text. Because LLM currently generally adopts the decoder structure, the last token can see the entire previous sentence, and the extracted features can represent the semantic information of the entire instruction. When performing self-deduplication, a clustering algorithm can be used to implement its own deduplication operation. Among them, the clustering algorithm can use unsupervised clustering algorithms such as DBSCAN, because there is no need to enter a specific number of clusters, and then obtain the clustering results. Randomly extract a certain number of points (instructions) from each cluster of the cluster to complete the instruction's own deduplication. When performing comparative deduplication, you can directly calculate the cosine similarity cosine between the features. <f 新指令 ,f 库中指令 >To determine the degree of similarity and achieve deduplication.
[0063] As attached Figure 3 The schematic diagram of an application scenario of an embodiment of the present application is shown. After the instruction data is stored in the database, relevant high-quality instructions can be retrieved separately through multiple conditions such as category, quality score sorting, etc., and combined into fine-tuning training data to complete model iterative optimization, which is both convenient and efficient.
[0064] As described above, the technical solution of the present application is described. It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order, and they can be performed simultaneously (in parallel) or in other orders, and these changes are within the scope of protection of the present application.
[0065] Furthermore, the present application also provides an instruction data processing system.
[0066] See attached Figure 4 , Figure 4 FIG. 1 is a main structural block diagram of an instruction data processing system according to an embodiment of the present application. Figure 4 As shown, the instruction data processing system in the embodiment of the present application mainly includes an acquisition module 11, an analysis module 12 and an execution module 13. In some embodiments, one or more of the acquisition module 11, the analysis module 12 and the execution module 13 can be combined into one module. In some embodiments, the acquisition module 11 can be configured to acquire instruction data to be processed. The analysis module 12 can be configured to determine the category, quality score and security score of each of the instruction data to be processed based on a preset model. The execution module 13 can be configured to store the instruction data to be processed whose quality score is greater than a preset quality threshold and whose security score is greater than a preset security threshold into a data table corresponding to the category of the instruction data to be processed. Among them, the analysis module 12 can also be configured to train a preset basic model to obtain a preset classification model, a preset quality model and a preset security model respectively.
[0067] The above instruction data processing system is used to execute Figure 1 The instruction data processing method embodiment shown in the figure has similar technical principles, technical problems solved and technical effects produced. Technical personnel in this technical field can clearly understand that for the convenience and conciseness of description, the specific working process and related instructions of the instruction data processing system can refer to the contents described in the embodiment of the instruction data processing method, which will not be repeated here.
[0068] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.
[0069] Furthermore, the present application also provides a control device. In a control device embodiment according to the present application, the control device includes a processor and a storage device, the storage device can be configured to store a program for executing the instruction data processing method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the instruction data processing method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The control device can be a control device device formed by various electronic devices.
[0070] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for executing the instruction data processing method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned instruction data processing method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.
[0071] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present application, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.
[0072] It is understood by those skilled in the art that each module in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present application, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present application.
[0073] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.
Claims
1. A method for processing instruction data, characterized in that: The method comprises: Get the instruction data to be processed; Based on the preset model, determine the category, quality score and security score of each of the to-be-processed instruction data; The instruction data to be processed whose quality score is greater than a preset quality threshold and whose safety score is greater than a preset safety threshold are stored in a data table corresponding to the category of the instruction data to be processed.
2. The instruction data processing method according to claim 1, characterized in that: Determining the category, quality score, and safety score of each of the to-be-processed instruction data based on the preset model includes: Determine the category of each of the to-be-processed instruction data based on a preset category model; Determine the quality score of each of the to-be-processed instruction data based on a preset quality model; Based on a preset security model, a security score of each piece of instruction data to be processed is determined.
3. The instruction data processing method according to claim 2, characterized in that: The training step of the preset category model at least includes: Acquire first label data, wherein the first label data at least includes first instruction data and category information of the first instruction data; Based on a preset first rule, the first label data is input into a preset basic model for training to obtain the preset category model.
4. The instruction data processing method according to claim 3, characterized in that: The training step of the preset quality model at least includes: Acquire second label data, wherein the second label data at least includes second instruction data and second answer data corresponding to the second instruction data, and one piece of the second instruction data corresponds to at least two pieces of the second answer data; For each piece of the second instruction data, determine a unique piece of the second answer data among all the second answer data corresponding to the second instruction data; Based on the preset second rule and all the determined second answer data, the second label data is input into the preset basic model for training to obtain the preset quality model.
5. The instruction data processing method according to claim 3, characterized in that: The step of training the preset security model at least includes: Acquire third label data, wherein the third label data at least includes third instruction data and third answer data corresponding to the third instruction data, and one piece of the third instruction data corresponds to at least two pieces of the third answer data; For each piece of the third instruction data, determine a unique piece of the third answer data among all the third answer data corresponding to the third instruction data; Based on the preset third rule and all the determined third answer data, the third label data is input into the preset basic model for training to obtain the preset security model.
6. The instruction data processing method according to any one of claims 3 to 5, characterized in that: After obtaining the instruction data to be processed, the method further includes: Preprocessing the instruction data to be processed to obtain preprocessed instruction data, wherein the preprocessing at least includes deduplication; Determining the category, quality score and safety score of each of the to-be-processed instruction data based on a preset model includes: determining the category, quality score and safety score of each of the pre-processed instruction data based on a preset model.
7. The instruction data processing method according to claim 6, characterized in that: The preprocessing of the instruction data to be processed comprises: Based on the preset basic model, extracting semantic features of the instruction data to be processed; Based on the semantic features and the instruction data to be processed, deduplication of the instruction data to be processed; and / or, Based on the preset basic model, extracting semantic features of the instruction data to be processed; Based on the semantic features and all preset instruction data stored in the data table, the to-be-processed instruction data is deduplicated.
8. An instruction data processing system, characterized in that: The system comprises: An acquisition module, wherein the acquisition module is configured to acquire instruction data to be processed; An analysis module, wherein the analysis module is configured to determine the category, quality score, and safety score of each of the to-be-processed instruction data based on a preset model; An execution module is configured to store the pending instruction data whose quality score is greater than a preset quality threshold and whose safety score is greater than a preset safety threshold into a data table corresponding to the category of the pending instruction data.
9. A control device, comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the instruction data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the instruction data processing method according to any one of claims 1 to 7.
Citation Information
Cited By
Method and device for generating control information of energy consumption resources and related equipment
CN120408152A
Multi-modal data processing method and system, electronic equipment and program product
CN121093241A
