Data processing method and apparatus, and model optimization method and apparatus

Abstract

Provided are a data processing method and apparatus, and a model optimization method and apparatus. The data processing method comprises: respectively sampling in a plurality of data sets so as to determine target sample data on the basis of sampling results; inputting the target sample data into a plurality of language models for respectively processing to obtain target prediction data respectively outputted by the plurality of language models; using a preset text annotation model to perform text annotation processing on each piece of target prediction data to obtain text annotation information corresponding to each piece of target prediction data; and constructing a sample data group on the basis of the target sample data, the target prediction data, and the text annotation information, wherein the sample data group is used for executing a model training task and a model verification task.

Description

Data processing methods and apparatus, model optimization methods and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411212294.4, filed on August 30, 2024, entitled "Data Processing Method and Apparatus, Model Optimization Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the field of machine learning technology, and in particular to data processing methods and apparatus, and model optimization methods and apparatus. Background Technology

[0003] With the development of computer and internet technologies, large language models (LLMs) have made significant progress in the field of natural language processing. Well-trained LLMs can support personalized question-and-answer sessions, thus meeting users' needs in different scenarios. However, LLMs that rely solely on imitation learning have significant problems, namely, they may generate seemingly reasonable but inaccurate or unsafe content. Current technologies often use human feedback to optimize the model to address this issue. For example, they collect a large amount of user feedback data and use a reward model to optimize the model, making its output more aligned with user needs. However, this optimization method relies excessively on user feedback, and the data scale is small and lacks diversity. Therefore, an effective solution is urgently needed to address these problems. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a model optimization method, a model optimization apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0006] Sampling is performed on multiple datasets, and the target sample data is determined based on the sampling results.

[0007] The target sample data is input into multiple language models for processing to obtain target prediction data output by the multiple language models respectively.

[0008] The text annotation process is performed on each target prediction data using a pre-defined text annotation model to obtain the text annotation information corresponding to each target prediction data.

[0009] A sample data set is constructed based on the target sample data, the target prediction data, and the text annotation information, wherein the sample data set is used to perform model training tasks and model validation tasks.

[0010] Optionally, the step of sampling from multiple datasets and determining the target sample data based on the sampling results includes:

[0011] Identify multiple datasets, where the data types of the sample data contained in each dataset are different;

[0012] Samples are taken from each dataset to obtain multiple initial sample data, and an initial sample dataset is constructed based on the multiple initial sample data;

[0013] Select the target sample data from the initial sample dataset.

[0014] Optionally, before the step of inputting the target sample data into multiple language models for processing to obtain the target prediction data output by the multiple language models, the method further includes:

[0015] For the target sample data, select multiple language models and determine the corresponding prompt information for each language model;

[0016] The processing of the target sample data by any one of the plurality of language models includes:

[0017] The target sample data is updated using the prompt information corresponding to the language model, and model input context information is generated based on the update result;

[0018] The model input context information is input into the language model for processing to obtain target prediction data.

[0019] Optionally, determining the text annotation information of any one of the target prediction data points corresponding to the multiple language models includes:

[0020] The target sample data and the target prediction data are input into a preset text annotation model. The text annotation model is used to process the target prediction data in at least two annotation dimensions. Based on the processing results, the annotation text and the prediction scores corresponding to the at least two annotation dimensions are determined.

[0021] Based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions, text annotation information corresponding to the target prediction data is generated.

[0022] Optionally, it also includes:

[0023] A preset candidate text annotation model is determined, and the target prediction data is processed using the candidate text annotation model to obtain the candidate annotation text corresponding to the target prediction data;

[0024] Calculate the text similarity between the candidate annotation text and the annotation text;

[0025] If the text similarity is greater than a preset similarity threshold, the step of generating text annotation information corresponding to the target prediction data is performed based on the prediction scores corresponding to the annotation text and the at least two annotation dimensions.

[0026] Optionally, after the step of constructing a sample data set based on the target sample data, the target prediction data, and the text annotation information is performed, the method further includes:

[0027] Add the sample data group to the intermediate sample dataset;

[0028] If the intermediate sample dataset meets the dataset construction conditions, data cleaning and format conversion are performed on the intermediate sample dataset;

[0029] The target sample dataset is generated based on the data cleaning and format conversion results. The target sample data group contained in the target sample dataset is used to perform the model training task and the model validation task.

[0030] Optionally, it also includes:

[0031] Determine the reward model and criticism model corresponding to the language model to be optimized;

[0032] Based on the target sample data group contained in the target sample dataset, the reward model and the criticism model are trained, and the target reward ability and the target criticism model are determined according to the training results.

[0033] The target reward model and the target criticism model are used to optimize the language model to be optimized until a target language model that meets the optimization stopping condition is obtained.

[0034] According to a second aspect of the embodiments of this specification, a model optimization method is provided, comprising:

[0035] Determine the reward model and criticism model corresponding to the language model to be optimized;

[0036] The reward model and the criticism model are trained based on the target sample data set contained in the target sample dataset. The target reward ability and the target criticism model are determined based on the training results. The target sample data set contained in the target sample dataset is obtained by the above method.

[0037] The target reward model and the target criticism model are used to optimize the language model to be optimized until a target language model that meets the optimization stopping condition is obtained.

[0038] According to a third aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0039] The determination module is configured to sample from multiple datasets and determine the target sample data based on the sampling results.

[0040] The processing module is configured to input the target sample data into multiple language models for processing, and obtain the target prediction data output by the multiple language models respectively;

[0041] The annotation module is configured to use a preset text annotation model to perform text annotation processing on each target prediction data to obtain the text annotation information corresponding to each target prediction data.

[0042] The construction module is configured to construct a sample data set based on the target sample data, the target prediction data, and the text annotation information, wherein the sample data set is used to perform model training tasks and model validation tasks.

[0043] According to a fourth aspect of the embodiments of this specification, a model optimization apparatus is provided, comprising:

[0044] The model determination module is configured to determine the reward model and criticism model corresponding to the language model to be optimized.

[0045] The training model module is configured to train the reward model and the criticism model based on the target sample data set contained in the target sample dataset, and to determine the target reward ability and the target criticism model based on the training results, wherein the target sample data set contained in the target sample dataset is obtained by the above method;

[0046] The model optimization module is configured to optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

[0047] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0048] Memory and processor;

[0049] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method or model optimization method.

[0050] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described data processing method or model optimization method.

[0051] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method or model optimization method.

[0052] The data processing method provided in this embodiment aims to provide richer and more diverse feedback data for language models to optimize. It constructs feedback data by combining a text annotation model and multiple language models. Specifically, sampling can be performed on multiple datasets to determine target sample data based on the sampling results, thus increasing sample diversity. The target sample data can then be input into multiple language models for processing, obtaining target prediction data output by each model. Based on this, a pre-defined text annotation model can be used to annotate each target prediction data, obtaining corresponding text annotation information. To ensure quality and data scale when constructing text annotation information, multiple target prediction information outputs from multiple language models can be combined, effectively guaranteeing data richness. Then, a sample data set can be constructed based on the target sample data, each target prediction data, and the corresponding text annotation information. In the application phase, model training and validation tasks can be performed based on the sample data set, thereby enhancing the performance and alignment of the language model. Attached Figure Description

[0053] Figure 1 is a schematic diagram of a data processing method provided in one embodiment of this specification;

[0054] Figure 2 is a flowchart of a data processing method provided in one embodiment of this specification;

[0055] Figure 3 is a flowchart of a model optimization method provided in one embodiment of this specification;

[0056] Figure 4 is a flowchart of a data processing method provided in one embodiment of this specification;

[0057] Figure 5 is a schematic diagram of the structure of a data processing device provided in one embodiment of this specification;

[0058] Figure 6 is a schematic diagram of a model optimization device provided in one embodiment of this specification;

[0059] Figure 7 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0060] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0061] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0062] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0063] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0064] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0065] LLMs, or Large Language Models, are a class of deep learning models in the field of Natural Language Processing (NLP). These models are renowned for their massive parameter size and powerful language understanding and generation capabilities. Trained on large-scale text datasets, LLMs learn complex patterns and rules of language, enabling them to perform various NLP tasks, including but not limited to text generation, text classification, sentiment analysis, and question answering systems.

[0066] This specification provides a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a model optimization method, a model optimization apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0067] In practical applications, although language models can be optimized using user feedback data, this data is limited by time and budget constraints, making it difficult to scale up significantly. Furthermore, the data lacks diversity, and different users have different subjective opinions, leading to inconsistent annotation standards for the feedback data. Therefore, an effective solution is urgently needed to address these issues.

[0068] Referring to the schematic diagram in Figure 1, the data processing method provided in this embodiment, in order to provide richer and more diverse feedback data for language models to optimize, can combine a text annotation model and multiple language models to construct feedback data. Specifically, sampling can be performed on multiple datasets to determine the target sample data based on the sampling results; multiple datasets increase sample diversity. Then, the target sample data can be input into multiple language models for processing to obtain target prediction data output by each language model. Based on this, a preset text annotation model can be used to perform text annotation processing on each target prediction data to obtain text annotation information corresponding to each target prediction data. To ensure quality and data scale when constructing text annotation information, multiple target prediction information can be output by multiple language models, thereby effectively ensuring data richness. Then, a sample data set can be constructed based on the target sample data, each target prediction data, and the text annotation information corresponding to each target prediction data. In the application phase, model training and model validation tasks can be performed based on the sample data set to enhance the performance and alignment effect of the language model.

[0069] Referring to Figure 2, which shows a flowchart of a data processing method according to an embodiment of this specification, the method includes the following steps.

[0070] Step S202: Sample data is collected from multiple datasets and the target sample data is determined based on the sampling results.

[0071] The data processing method provided in this embodiment can be applied to any scenario of constructing feedback data corresponding to a large language model under any business project. The large language model that needs to be optimized for the constructed sample data group can be an LLM model, or it can be other large language models with natural language processing capabilities. This embodiment does not make any limitations here.

[0072] Specifically, multiple datasets refer to collections of sample data provided by different data sources. Each dataset contains samples of the same data type, but the types of samples differ between datasets, thus increasing the diversity of sample data group construction. Correspondingly, target sample data refers to any single sample data that needs to be processed at the current time after sampling.

[0073] Therefore, to provide richer and more diverse feedback data for language models to optimize, feedback data can be constructed by combining text annotation models and multiple language models. Specifically, sampling can be performed on multiple datasets to determine target sample data based on the sampling results; multiple datasets increase sample diversity. Then, the target sample data can be input into multiple language models for processing, obtaining target prediction data output by each language model. Based on this, a pre-defined text annotation model can be used to annotate each target prediction data, obtaining corresponding text annotation information. To ensure quality and data scale when constructing text annotation information, multiple target prediction information outputs from multiple language models can be combined, effectively guaranteeing data richness. Then, a sample data set can be constructed based on the target sample data, each target prediction data, and the corresponding text annotation information. In the application phase, model training and validation tasks can be performed based on the sample data set.

[0074] Furthermore, in order to construct richer feedback data and improve the coverage of the optimized model, a stratified sampling method can be used to complete the sample sampling process. In this embodiment, the specific implementation is as follows:

[0075] Multiple datasets are identified, each containing sample data of different data types; sampling is performed in each dataset to obtain multiple initial sample data, and an initial sample dataset is constructed based on the multiple initial sample data; target sample data is selected from the initial sample dataset.

[0076] Based on this, when sampling, multiple datasets can be determined first, and the data types of the sample data contained in each dataset are different. Then, sampling can be performed separately in each dataset. During the process, a stratified sampling strategy can be used to complete the sampling, thereby improving the richness of the samples. At this time, multiple initial sample data will be obtained, and then an initial sample dataset can be constructed based on the multiple initial sample data. Subsequently, any sample data in the initial sample dataset can be selected as the target sample data to construct the feedback data.

[0077] In practical applications, multiple datasets can be selected using TruthfulQA, FalseQA, Evol-Instruct, UltraChat, ShareGPT, etc., and this embodiment does not impose any limitations. When sampling samples, stratified sampling can be used from multiple data sources to obtain different types of sample data, ensuring that the sample data covers various tasks and scenarios. Furthermore, excessively long or unsuitable sample data can be removed to ensure the sample data is concise and clear.

[0078] For example, when constructing feedback data for optimizing LLMs models, stratified sampling can be performed from multiple datasets such as TruthfulQA, FalseQA, Evol-Instruct, UltraChat, and ShareGPT. Based on the sampling results, n sample data will be obtained. Then, by filtering the n sample data for low-quality sample data, m target sample data will be obtained. Feedback data can then be constructed for each of the m target sample data to achieve the goal of optimizing the LLMs model.

[0079] In practical applications, the target sample data can be the problem text, images, videos, etc. input into the model for processing; this embodiment does not impose any limitations on this.

[0080] In summary, by using stratified sampling to construct the initial sample dataset, the richness of the samples can be effectively improved, thereby covering more task scenarios and enhancing the model's generalization ability.

[0081] Step S204: Input the target sample data into multiple language models for processing to obtain target prediction data output by the multiple language models respectively.

[0082] Specifically, after obtaining the target sample data, in order to ensure the diversity of the feedback data, multiple language models can be combined to process the target sample data, thereby obtaining the target prediction data output by each language model, so as to combine multiple target prediction data to complete the construction of feedback data.

[0083] In this context, "multiple language models" specifically refers to language models capable of processing target sample data and outputting prediction results. Examples include GPT-4, gpt-3.5-turbo, and Bard. This embodiment does not impose any limitations on these models, and the number of selected language models during the prediction data construction phase can be set according to actual needs; this embodiment does not impose any limitations on this either. Correspondingly, the target prediction data is the prediction result obtained by the model after processing the target sample data. For example, if the target sample data is question text, then the target prediction data is the answer text output by the language model. Furthermore, since different models have different capabilities, multiple different prediction data can be obtained.

[0084] Furthermore, in order to output different prediction information through multiple language models and thus obtain different text annotation information in the subsequent process, thereby improving the richness of the feedback data, different prompts corresponding to different language models can be used to update the sample data before generating prediction information. In this embodiment, the specific implementation is as follows:

[0085] Multiple language models are selected for the target sample data, and prompt information corresponding to each language model is determined; wherein, the processing of the target sample data by any one of the multiple language models includes: updating the target sample data using the prompt information corresponding to the language model, generating model input context information based on the update result; and inputting the model input context information into the language model for processing to obtain target prediction data.

[0086] Specifically, the prompt information refers to the prompt words corresponding to different language models. Correspondingly, the model input context information refers to the text information obtained after updating the target sample data using the prompt information, which is used to input the model for generating prediction data.

[0087] Based on this, when selecting multiple language models to generate prediction data, multiple language models can be selected according to the target sample data, and the corresponding prompt information for each language model needs to be determined. This allows for updating the target sample data with different prompt information, resulting in different prediction data when inputting into different language models, facilitating the subsequent construction of diverse feedback data. For processing the target sample data using any one of the multiple language models, the target sample data can first be updated using the prompt information corresponding to the language model, generating model input context information based on the update result. Then, the model input context information can be input into the language model for processing, thereby obtaining the target prediction data output by that language model. Similarly, any language model can be processed in this way to obtain multiple prediction data for subsequent use.

[0088] In practical applications, to avoid the problem of inconsistent text style and quality in the predicted data generated from sample data, different multilingual models can be randomly selected to output the target predicted data corresponding to the target sample data. Furthermore, different principles and prompt words can be introduced to update the sample data, so as to induce the language model to output target predicted data with different styles and qualities.

[0089] Following the previous example, after obtaining m target sample data (in this embodiment, m=2), two language models can be selected for target sample data 1 and target sample data 2, namely language model a and language model b. Then, the prompt words corresponding to language models a and b can be determined, and target sample data 1 is updated to target sample data 1a and 1b, and target sample data 2 is updated to target sample data 2a and 2b. Further, by inputting target sample data 1a and target sample data 2a into language model a for processing, target prediction data 1a corresponding to target sample data 1 and target prediction data 2a corresponding to target sample data 2 are obtained. At the same time, target sample data 1b and target sample data 2b are input into language model b for processing, target prediction data 1b corresponding to target sample data 1 and target prediction data 2b corresponding to target sample data 2 are obtained. The styles and qualities of each prediction data are different, which facilitates subsequent use.

[0090] In summary, by introducing prompts to update the sample data, the language model can output responses of different quality and style to the sample data, thereby enabling the subsequent construction of feedback data using diverse data.

[0091] Step S206: Use a preset text annotation model to perform text annotation processing on each target prediction data to obtain the text annotation information corresponding to each target prediction data.

[0092] Specifically, after obtaining multiple prediction data corresponding to the target sample data, in order to improve the efficiency of feedback data construction and reduce costs, a preset text annotation model can be used to perform text annotation processing on each target prediction data, thereby obtaining the text annotation information corresponding to each target prediction data. This realizes the automated text annotation method through the text annotation model, thereby improving the efficiency of feedback data construction.

[0093] Specifically, the text annotation model refers to a model that, after inputting the target prediction data corresponding to the target sample data, can generate feedback annotations for the target prediction data. This model can be implemented using GPT-4. Correspondingly, the text annotation information refers to the annotation information obtained after the text annotation model processes the target prediction data, which is used to simulate real user feedback data, thereby facilitating subsequent optimization of the language model.

[0094] Furthermore, when constructing text annotation information, the annotation text and predicted scores can be combined, which facilitates the subsequent filtering of high-quality feedback data. In this embodiment, the specific implementation is as follows:

[0095] The target sample data and the target prediction data are input into a preset text annotation model. The text annotation model is used to process the target prediction data in at least two annotation dimensions. The annotation text and the prediction scores corresponding to the at least two annotation dimensions are determined according to the processing results. Based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions, text annotation information corresponding to the target prediction data is generated.

[0096] Specifically, the prediction score refers to the score obtained by the text annotation model after scoring the target prediction data in at least two annotation dimensions. The at least two annotation dimensions may include instruction compliance, authenticity, honesty, and usefulness. The prediction scores corresponding to multiple dimensions reflect the usability of the target prediction data, so as to facilitate the subsequent division of positive and negative samples to complete model optimization.

[0097] Based on this, for any target prediction data, the target sample data and the target prediction data can be input into a preset text annotation model. The text annotation model is used to process the target prediction data in at least two annotation dimensions, thereby determining the annotation text and the prediction scores corresponding to the at least two annotation dimensions based on the processing results. Subsequently, based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions, text annotation information corresponding to the target prediction data can be generated to reflect the usability of the target prediction data for subsequent model optimization.

[0098] Following the previous example, after processing target sample data 1 and target sample data 2 using multiple language models, we obtain target prediction data 1a and target prediction data 1b corresponding to target sample data 1, and target prediction data 2a and target prediction data 2b corresponding to target sample data 2. Based on this, a text annotation model can be used to score the target prediction data across the four dimensions mentioned above and generate detailed text annotations. After the text annotation model scores and generates annotations for the prediction data corresponding to each sample data, we obtain the prediction score sa1 and annotation text 1ac for target prediction data 1a; the prediction score sb1 and annotation text 1bc for target prediction data 1b; the prediction score sa2 and annotation text 2ac for target prediction data 2a; and the prediction score sb2 and annotation text 2bc for target prediction data 2b. Subsequently, we can combine the above information to filter high-quality prediction data and annotation text to construct feedback data for training the language model.

[0099] In summary, by combining predicted scores and annotation text to construct text annotation information, a preliminary sample data set can be built, which facilitates subsequent model optimization.

[0100] Furthermore, to ensure the quality of the annotated text, quality can be improved through candidate text annotation model prediction. In this embodiment, the specific implementation is as follows:

[0101] A preset candidate text annotation model is determined, and the target prediction data is processed using the candidate text annotation model to obtain candidate annotation text corresponding to the target prediction data; the text similarity between the candidate annotation text and the annotation text is calculated; if the text similarity is greater than a preset similarity threshold, the step of generating text annotation information corresponding to the target prediction data is performed based on the prediction scores corresponding to the annotation text and the at least two annotation dimensions respectively.

[0102] Specifically, the candidate text annotation model refers to a text annotation model with the same capabilities as the text annotation model but a different model structure. It is used to generate candidate annotation texts for the predicted data. These candidate annotation texts are then compared with the original annotation texts to select high-quality annotation texts for subsequent sample data construction. Correspondingly, the candidate annotation texts are the annotation texts output by the candidate text annotation model. Correspondingly, text similarity specifically refers to calculating the similarity score between the candidate annotation texts and the original annotation texts. This can be accomplished by calculating text distance and cosine similarity, which is not limited in this embodiment.

[0103] Based on this, in order to obtain high-quality annotation text, a candidate text annotation model can be used to process any target prediction data. First, a preset candidate text annotation model can be determined, and then the target prediction data can be processed using the candidate text annotation model to obtain the candidate annotation text corresponding to the target prediction data. Based on this, the text similarity between the candidate annotation text and the annotation text can be calculated. If the text similarity is greater than a preset similarity threshold, it indicates that the annotation text output by the two different text annotation models is highly similar, thus indicating that the annotation text is high-quality text. Therefore, the step of generating text annotation information corresponding to the target prediction data based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions can continue.

[0104] Furthermore, if the text similarity is less than or equal to a preset similarity threshold, it indicates that the two different text annotation models output annotations with varying similarity, suggesting that one model's annotation is inaccurate. To obtain accurate annotations, both models can generate annotations again until a similarity greater than the threshold is obtained, at which point further processing can proceed. Moreover, if higher similarity annotations are not obtained for an extended period, a third text annotation model can be introduced to make decisions, thereby ensuring the accuracy of the annotations and improving the quality of the text annotation information.

[0105] In summary, by combining candidate text annotation models to construct annotation text, high-quality annotation text can be selected, thereby effectively improving the quality of sample data sets and enabling better model optimization.

[0106] Step S208: Construct a sample data group based on the target sample data, the target prediction data, and the text annotation information, wherein the sample data group is used to perform model training tasks and model validation tasks.

[0107] Specifically, after obtaining the target sample data, the target prediction data output by each language model, and the text annotation information corresponding to each target prediction data, a sample data set can be constructed by combining the target sample data, the target prediction data, and the text annotation information, so as to realize the subsequent model training and model validation tasks by combining the sample data set.

[0108] The sample data set refers to the feedback data used to optimize the language model. Correspondingly, the model training task specifically refers to the task of training the model using the sample data set, while the model validation task specifically refers to the task of validating the model using the sample data set.

[0109] Furthermore, after obtaining the sample data set, data cleaning and format conversion can be performed to improve quality, making it easier to reuse in the application stage. In this embodiment, the specific implementation is as follows:

[0110] The sample data set is added to the intermediate sample dataset; if the intermediate sample dataset meets the dataset construction conditions, data cleaning and format conversion are performed on the intermediate sample dataset; a target sample dataset is generated based on the data cleaning and format conversion results, wherein the target sample data set contained in the target sample dataset is used to perform the model training task and the model validation task.

[0111] Specifically, the intermediate sample dataset refers to the data set obtained after constructing sample data groups. Correspondingly, data cleaning refers to the process of removing duplicate and low-quality sample data groups from the intermediate sample dataset. Correspondingly, format conversion refers to the process of updating the sample data groups contained in the intermediate sample dataset to a unified data structure.

[0112] Therefore, in order to use high-quality data for model training and validation, after obtaining the sample data set, it can be added to the intermediate sample dataset. If the intermediate sample dataset meets the dataset construction conditions, it means that the sample data set contained in the intermediate sample dataset can be used for subsequent tasks. At this time, data cleaning and format conversion can be performed on the intermediate sample dataset first to remove redundant data and unify the data format. Then, the target sample dataset can be generated based on the data cleaning and format conversion results. Subsequently, model training and model validation tasks can be performed based on the target sample data set contained in the target sample dataset.

[0113] In summary, by cleaning and formatting the intermediate sample dataset, we can obtain a clean and high-quality sample data set with the same data format, which is more convenient for subsequent model training and evaluation.

[0114] Furthermore, after obtaining a high-quality target sample dataset, in order to simulate real-world model optimization scenarios, a reward model and a criticism model can be trained using the data first, and then the language model can be optimized by combining the trained models. In this embodiment, the specific implementation is as follows:

[0115] Determine the reward model and criticism model corresponding to the language model to be optimized; train the reward model and the criticism model based on the target sample data group contained in the target sample dataset, and determine the target reward ability and the target criticism model according to the training results; optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

[0116] Specifically, the reward model and the criticism model refer to auxiliary models used to optimize the language model to be optimized. The reward model is used to evaluate and improve the prediction quality of the language model to be optimized, while the criticism model is used to provide detailed improvement suggestions, thereby enabling the language model to achieve self-learning. Correspondingly, the optimization stopping condition refers to the condition for stopping the optimization of the language model to be optimized, such as the number of iterations, loss value comparison conditions, or validation set verification conditions, etc. This embodiment does not impose any limitations.

[0117] Based on this, in order to effectively optimize the language model to be optimized, the reward model and criticism model corresponding to the language model to be optimized can be determined first. At this time, the reward model and the criticism model can be trained based on the target sample data group contained in the target sample dataset. The target reward model and the target criticism model can be determined according to the training results. On this basis, the target reward model and the target criticism model can be used to optimize the language model to be optimized, so as to adjust the model to be optimized into a target language model that meets the optimization stopping condition, thereby facilitating its application in actual business scenarios.

[0118] Following the previous example, after obtaining the predicted data, predicted scores, and annotation text corresponding to target sample data 1 and target sample data 2, respectively, an intermediate sample dataset can be constructed by combining data from various dimensions. After data cleaning and format conversion, a high-quality target sample dataset will be obtained. Based on this, the reward and criticism models of the LLMs model can be trained using the target sample dataset. The reward model can improve the output quality of the LLMs model, and the criticism model can provide detailed suggestions for model improvement. Optimizing the LLMs model based on this yields a target LLMs model that meets the usage requirements. In the application phase, if a user inputs the question text "What is the capital of location A?", the model can provide the answer text "City B".

[0119] The data processing method provided in this embodiment aims to provide richer and more diverse feedback data for language models to optimize. It constructs feedback data by combining a text annotation model and multiple language models. Specifically, sampling can be performed on multiple datasets to determine target sample data based on the sampling results, thus increasing sample diversity. The target sample data can then be input into multiple language models for processing, obtaining target prediction data output by each model. Based on this, a pre-defined text annotation model can be used to annotate each target prediction data, obtaining corresponding text annotation information. To ensure quality and data scale when constructing text annotation information, multiple target prediction information outputs from multiple language models can be combined, effectively guaranteeing data richness. Then, a sample data set can be constructed based on the target sample data, each target prediction data, and the corresponding text annotation information. In the application phase, model training and validation tasks can be performed based on the sample data set, thereby enhancing the performance and alignment of the language model.

[0120] Referring to Figure 3, which shows a flowchart of a model optimization method according to an embodiment of this specification, the method includes the following steps.

[0121] Step S302: Determine the reward model and criticism model corresponding to the language model to be optimized;

[0122] Step S304: Based on the target sample data group contained in the target sample dataset, train the reward model and the criticism model, and determine the target reward ability and the target criticism model according to the training results, wherein the target sample data group contained in the target sample dataset is obtained by the above method;

[0123] Step S306: Optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

[0124] The description of the model optimization method provided in this embodiment can be found in the description of the data processing method in the above embodiment, and will not be elaborated further in this embodiment.

[0125] The following description, in conjunction with Figure 4, uses the application of the data processing method provided in this specification in a large language model scenario as an example to further illustrate the data processing method. Figure 4 shows a flowchart of the processing procedure of a data processing method provided in one embodiment of this specification, specifically including the following steps.

[0126] Step S402: Determine multiple datasets, wherein the data types of sample data contained in each dataset are different.

[0127] Step S404: Sample data is collected from each dataset to obtain multiple initial sample data, and an initial sample dataset is constructed based on the multiple initial sample data.

[0128] Step S406: Select target sample data from the initial sample dataset.

[0129] Step S408: Input the target sample data into multiple language models for processing to obtain target prediction data output by the multiple language models respectively.

[0130] Step S410: Use a preset text annotation model to perform text annotation processing on each target prediction data to obtain the text annotation information corresponding to each target prediction data.

[0131] The determination of text annotation information for any one of the target prediction data corresponding to the plurality of language models includes: inputting the target sample data and the target prediction data into a preset text annotation model; processing the target prediction data in at least two annotation dimensions using the text annotation model; determining the annotation text and the prediction scores corresponding to the at least two annotation dimensions based on the processing results; and generating text annotation information corresponding to the target prediction data based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions.

[0132] Step S412: Construct a sample data group based on the target sample data, target prediction data, and text annotation information, and add the sample data group to the intermediate sample dataset.

[0133] Step S414: If the intermediate sample dataset meets the dataset construction conditions, perform data cleaning and format conversion on the intermediate sample dataset.

[0134] Step S416: Generate the target sample dataset based on the data cleaning results and format conversion results.

[0135] Step S418: When the model optimization task is triggered, determine the reward model and criticism model corresponding to the language model to be optimized.

[0136] Step S420: Based on the target sample data group contained in the target sample dataset, train the reward model and the criticism model, and determine the target reward model and the target criticism model based on the training results.

[0137] Step S422: Optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

[0138] The data processing method provided in this embodiment aims to provide richer and more diverse feedback data for language models to optimize. It constructs feedback data by combining a text annotation model and multiple language models. Specifically, sampling can be performed on multiple datasets to determine target sample data based on the sampling results, thus increasing sample diversity. The target sample data can then be input into multiple language models for processing, obtaining target prediction data output by each model. Based on this, a pre-defined text annotation model can be used to annotate each target prediction data, obtaining corresponding text annotation information. To ensure quality and data scale when constructing text annotation information, multiple target prediction information outputs from multiple language models can be combined, effectively guaranteeing data richness. Then, a sample data set can be constructed based on the target sample data, each target prediction data, and the corresponding text annotation information. In the application phase, model training and validation tasks can be performed based on the sample data set, thereby enhancing the performance and alignment of the language model.

[0139] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5 shows a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification. As shown in Figure 5, the apparatus includes:

[0140] The determination module 502 is configured to sample from multiple datasets and determine the target sample data based on the sampling results.

[0141] Processing module 504 is configured to input the target sample data into multiple language models for processing, and obtain target prediction data output by the multiple language models respectively;

[0142] The annotation module 506 is configured to use a preset text annotation model to perform text annotation processing on each target prediction data to obtain the text annotation information corresponding to each target prediction data.

[0143] The construction module 508 is configured to construct a sample data group based on the target sample data, the target prediction data, and the text annotation information, wherein the sample data group is used to perform model training tasks and model validation tasks.

[0144] In an optional embodiment, the determining module 502 is further configured to:

[0145] Multiple datasets are identified, each containing sample data of different data types; sampling is performed in each dataset to obtain multiple initial sample data, and an initial sample dataset is constructed based on the multiple initial sample data; target sample data is selected from the initial sample dataset.

[0146] In an optional embodiment, the apparatus further includes:

[0147] The information determination module is configured to select multiple language models for the target sample data and determine the prompt information corresponding to each language model; wherein, the processing of the target sample data by any one of the multiple language models includes: updating the target sample data using the prompt information corresponding to the language model, generating model input context information based on the update result; and inputting the model input context information into the language model for processing to obtain target prediction data.

[0148] In an optional embodiment, the annotation module 506 is further configured to:

[0149] The target sample data and the target prediction data are input into a preset text annotation model. The text annotation model is used to process the target prediction data in at least two annotation dimensions. Based on the processing results, the annotation text and the prediction scores corresponding to the at least two annotation dimensions are determined. Based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions, text annotation information corresponding to the target prediction data is generated.

[0150] In an optional embodiment, the apparatus further includes:

[0151] The similarity calculation module is configured to determine a preset candidate text annotation model, process the target prediction data using the candidate text annotation model to obtain candidate annotation text corresponding to the target prediction data; calculate the text similarity between the candidate annotation text and the annotation text; and, if the text similarity is greater than a preset similarity threshold, execute the step of generating text annotation information corresponding to the target prediction data based on the prediction scores corresponding to the annotation text and the at least two annotation dimensions.

[0152] In an optional embodiment, the apparatus further includes:

[0153] The build set module is configured to add the sample data set to the intermediate sample dataset;

[0154] If the intermediate sample dataset meets the dataset construction conditions, data cleaning and format conversion are performed on the intermediate sample dataset;

[0155] The target sample dataset is generated based on the data cleaning and format conversion results. The target sample data group contained in the target sample dataset is used to perform the model training task and the model validation task.

[0156] In an optional embodiment, the apparatus further includes:

[0157] The model optimization module is configured to determine the reward model and the criticism model corresponding to the language model to be optimized; train the reward model and the criticism model based on the target sample data group contained in the target sample dataset; determine the target reward ability and the target criticism model based on the training results; and optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

[0158] The data processing device provided in this embodiment, in order to provide richer and more diverse feedback data for language models to optimize, can construct feedback data by combining a text annotation model and multiple language models. Specifically, sampling can be performed on multiple datasets to determine the target sample data based on the sampling results; multiple datasets increase sample diversity. Then, the target sample data can be input into multiple language models for processing, obtaining target prediction data output by each language model. Based on this, a preset text annotation model can be used to perform text annotation processing on each target prediction data, obtaining text annotation information corresponding to each target prediction data. To ensure quality and data scale when constructing text annotation information, multiple target prediction information can be output by multiple language models, effectively ensuring data richness. Then, a sample data set can be constructed based on the target sample data, each target prediction data, and the corresponding text annotation information. In the application phase, model training and model validation tasks can be performed based on the sample data set, thereby enhancing the performance and alignment effect of the language model.

[0159] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0160] Corresponding to the above method embodiments, this specification also provides embodiments of a model optimization device. Figure 6 shows a schematic diagram of the structure of a model optimization device provided in one embodiment of this specification. As shown in Figure 6, the device includes:

[0161] Model determination module 602 is configured to determine the reward model and criticism model corresponding to the language model to be optimized;

[0162] The training model module 604 is configured to train the reward model and the criticism model based on the target sample data set contained in the target sample dataset, and determine the target reward ability and the target criticism model based on the training results, wherein the target sample data set contained in the target sample dataset is obtained by the above method;

[0163] The model optimization module 606 is configured to optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

[0164] The above is a schematic scheme of a model optimization device according to this embodiment. It should be noted that the technical solution of this model optimization device and the technical solution of the above-described model optimization method belong to the same concept. For details not described in detail in the technical solution of the model optimization device, please refer to the description of the technical solution of the above-described model optimization method.

[0165] Figure 7 shows a structural block diagram of a computing device 700 according to one embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0166] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0167] In one embodiment of this specification, the aforementioned components of the computing device 700, as well as other components not shown in FIG. 7, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 7 is merely illustrative and not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0168] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0169] The processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned data processing method or model optimization method.

[0170] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the data processing method or model optimization method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method or model optimization method described above.

[0171] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method or model optimization method.

[0172] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the data processing method or model optimization method described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the data processing method or model optimization method described above.

[0173] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method or model optimization method.

[0174] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the data processing method or model optimization method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method or model optimization method described above.

[0175] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0176] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0177] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0178] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0179] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A data processing method, characterized in that, include: Sampling is performed on multiple datasets, and the target sample data is determined based on the sampling results. The target sample data is input into multiple language models for processing to obtain target prediction data output by the multiple language models respectively. The text annotation process is performed on each target prediction data using a pre-defined text annotation model to obtain the text annotation information corresponding to each target prediction data. A sample data set is constructed based on the target sample data, the target prediction data, and the text annotation information, wherein the sample data set is used to perform model training tasks and model validation tasks.

2. The data processing method according to claim 1, characterized in that, The step of sampling from multiple datasets and determining the target sample data based on the sampling results includes: Identify multiple datasets, where the data types of the sample data contained in each dataset are different; Samples are taken from each dataset to obtain multiple initial sample data, and an initial sample dataset is constructed based on the multiple initial sample data; Select the target sample data from the initial sample dataset.

3. The data processing method according to claim 1, characterized in that, Before the step of inputting the target sample data into multiple language models for processing to obtain the target prediction data output by the multiple language models, the method further includes: For the target sample data, select multiple language models and determine the corresponding prompt information for each language model; The processing of the target sample data by any one of the plurality of language models includes: The target sample data is updated using the prompt information corresponding to the language model, and model input context information is generated based on the update result; The model input context information is input into the language model for processing to obtain target prediction data.

4. The data processing method according to claim 1, characterized in that, The determination of text annotation information for any one of the target prediction data points corresponding to the multiple language models includes: The target sample data and the target prediction data are input into a preset text annotation model. The text annotation model is used to process the target prediction data in at least two annotation dimensions. Based on the processing results, the annotation text and the prediction scores corresponding to the at least two annotation dimensions are determined. Based on the annotation text and the prediction scores corresponding to the at least two annotation dimensions, text annotation information corresponding to the target prediction data is generated.

5. The data processing method according to claim 4, characterized in that, Also includes: A preset candidate text annotation model is determined, and the target prediction data is processed using the candidate text annotation model to obtain the candidate annotation text corresponding to the target prediction data; Calculate the text similarity between the candidate annotation text and the annotation text; If the text similarity is greater than a preset similarity threshold, the step of generating text annotation information corresponding to the target prediction data is performed based on the prediction scores corresponding to the annotation text and the at least two annotation dimensions.

6. The data processing method according to claim 1, characterized in that, After the step of constructing a sample data set based on the target sample data, the target prediction data, and the text annotation information is executed, the method further includes: Add the sample data group to the intermediate sample dataset; If the intermediate sample dataset meets the dataset construction conditions, data cleaning and format conversion are performed on the intermediate sample dataset; The target sample dataset is generated based on the data cleaning and format conversion results. The target sample data group contained in the target sample dataset is used to perform the model training task and the model validation task.

7. The data processing method according to claim 6, characterized in that, Also includes: Determine the reward model and criticism model corresponding to the language model to be optimized; Based on the target sample data group contained in the target sample dataset, the reward model and the criticism model are trained, and the target reward ability and the target criticism model are determined according to the training results. The target reward model and the target criticism model are used to optimize the language model to be optimized until a target language model that meets the optimization stopping condition is obtained.

8. A model optimization method, characterized in that, include: Determine the reward model and criticism model corresponding to the language model to be optimized; The reward model and the criticism model are trained based on the target sample data set contained in the target sample dataset, and the target reward ability and the target criticism model are determined according to the training results, wherein the target sample data set contained in the target sample dataset is obtained by the method of any one of claims 1 to 7; The target reward model and the target criticism model are used to optimize the language model to be optimized until a target language model that meets the optimization stopping condition is obtained.

9. A data processing apparatus, characterized in that, include: The determination module is configured to sample from multiple datasets and determine the target sample data based on the sampling results. The processing module is configured to input the target sample data into multiple language models for processing, and obtain the target prediction data output by the multiple language models respectively; The annotation module is configured to use a preset text annotation model to perform text annotation processing on each target prediction data to obtain the text annotation information corresponding to each target prediction data. The construction module is configured to construct a sample data set based on the target sample data, the target prediction data, and the text annotation information, wherein the sample data set is used to perform model training tasks and model validation tasks.

10. A model optimization device, characterized in that, include: The model determination module is configured to determine the reward model and criticism model corresponding to the language model to be optimized. The training model module is configured to train the reward model and the criticism model based on the target sample data set contained in the target sample dataset, and to determine the target reward capability and the target criticism model based on the training results, wherein the target sample data set contained in the target sample dataset is obtained by the method of any one of claims 1 to 7; The model optimization module is configured to optimize the language model to be optimized using the target reward model and the target criticism model until a target language model that meets the optimization stopping condition is obtained.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images