Data construction method, model training method, information processing method and equipment

Abstract

The invention provides a data construction method, a model training method, an information processing method and equipment, and the method comprises the steps: obtaining an initial data set, and enabling initial data in the initial data set to comprise basic information, at least one piece of associated information corresponding to the basic information, and behavior information corresponding to each piece of associated information, the behavior information is used for representing the behavior of the user for the associated information; pre-training preference data are constructed according to the initial data, and the pre-training preference data comprise basic information, associated information and corresponding preference information; wherein the preference information is used for representing the preference degree of the user for the associated information, and the preference information is determined by behavior information corresponding to the associated information; wherein the pre-training preference data is used for pre-training a preference determination model. According to the method, the value alignment optimization effect can be improved, the workload of manual labeling can be reduced, the cost is effectively reduced, and the efficiency is improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a data construction method, a model training method, an information processing method and a device. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, the application of various AI models is becoming increasingly widespread. Aligning these models with human values and preferences, and preventing their misuse, has become a pressing issue. Aligning these models with human values and preferences means that the values pursued by the model should be consistent with human values. The model should process tasks in accordance with human values and preferences to avoid harm to human society.

[0003] In order to align the model with human value preferences, humans can label the model's output information according to value preferences during the model training process, and adjust the model's value preferences through feedback on the model's output information.

[0004] However, manual labeling is inefficient, especially when the amount of data is large. It is difficult to complete the labeling task quickly and effectively, resulting in limited model training results and poor value alignment effects. Summary of the Invention

[0005] This application provides a data construction method, a model training method, an information processing method and a device to improve the value alignment effect of the model.

[0006] In a first aspect, an embodiment of the present application provides a data construction method, comprising:

[0007] Acquire an initial data set, where the initial data in the initial data set includes basic information, at least one associated information corresponding to the basic information, and behavior information corresponding to each associated information, where the behavior information is used to represent a user's behavior with respect to the associated information;

[0008] Constructing pre-trained preference data based on the initial data, the pre-trained preference data including basic information, associated information, and corresponding preference information; wherein the preference information is used to indicate the user's preference for the associated information, and the preference information is determined by the behavior information corresponding to the associated information;

[0009] The pre-trained preference data is used to pre-train a preference determination model.

[0010] In a second aspect, an embodiment of the present application provides a model training method, comprising:

[0011] Obtaining pre-trained preference data, and pre-training a preference determination model based on the pre-trained preference data;

[0012] Acquiring fine-tuning preference data, and performing fine-tuning training on the pre-trained preference determination model according to the fine-tuning preference data;

[0013] The pre-trained preference data includes the pre-trained preference data obtained by the method described in the first aspect.

[0014] Optionally, the method further includes:

[0015] Determine the model based on the preferences after fine-tuning training and perform reinforcement learning training on the target model;

[0016] The process of obtaining association information based on the target model, constructing fine-tuning preference data based on the association information, fine-tuning the preference determination model based on the fine-tuning preference data, and performing reinforcement learning training on the target model based on the preference determination model is repeated until the preset requirements are met.

[0017] In a third aspect, an embodiment of the present application provides an information processing method, including:

[0018] Get the basic information to be processed;

[0019] Inputting the basic information into the target model to obtain corresponding associated information, and outputting the associated information to the user;

[0020] Wherein, the target model is obtained by training through the method described in the second aspect.

[0021] In a fourth aspect, an embodiment of the present application provides an information processing method, including:

[0022] Questions to get user input;

[0023] Input the question into the target model, obtain the corresponding answer, and output the answer to the user;

[0024] Wherein, the target model is obtained by training through the method described in the second aspect.

[0025] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0026] at least one processor; and

[0027] a memory communicatively coupled to the at least one processor;

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0029] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above aspects is implemented.

[0030] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0031] The data construction method, model training method, information processing method and device provided in the embodiments of the present application can obtain basic information and its corresponding related information, and use existing user behavior information for related information to construct preference data. The behavior information reflects the value preference of the user group for the related information. Using the constructed preference data to train the preference determination model can enable the preference determination model to have the ability to distinguish whether the related information conforms to human value preferences, improve the effect of value alignment optimization, and reduce the workload of manual labeling, effectively reduce costs and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of another application scenario provided by an embodiment of the present application;

[0035] Figure 3 A flywheel iterative process based on reinforcement learning provided in an embodiment of the present application;

[0036] Figure 4 A flow chart of a value alignment optimization solution provided in an embodiment of the present application;

[0037] Figure 5 A flowchart of a data construction method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the input and output of a target model provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the principle of constructing an initial data set provided in an embodiment of the present application;

[0040] Figure 8 A flowchart of a model training method provided in an embodiment of the present application;

[0041] Figure 9 A flowchart of an information processing method applied to an intelligent customer service scenario provided in an embodiment of the present application;

[0042] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0043] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0044] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.

[0045] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0046] The embodiments of the present application can be implemented through a large model, for example, a large language model. Among them, a large language model refers to a deep learning language model with large-scale language model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion language model parameters. The large language model can also be called a cornerstone language model / foundation model (Foundation Model). The large language model is pre-trained by using large-scale unlabeled corpus to produce a pre-trained language model with more than 100 million parameters. This language model can adapt to a wide range of downstream tasks, and the language model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training language model, etc.

[0047] In practical applications, large language models only require a small number of samples to fine-tune the pre-trained language model and can be applied to different tasks. Large language models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large language models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0048] First, let’s explain the terms involved in this application:

[0049] SFT (Supervised Fine-Tuning): is a training method that allows a pre-trained model to be fine-tuned using labeled data on a specific task to improve its performance on the specific task.

[0050] RLHF (Reinforcement Learning from Human Feedback) is a machine learning method that combines reinforcement learning and human feedback mechanisms, learning and optimizing through feedback from human users.

[0051] The application scenario of this application is first described below.

[0052] In the field of artificial intelligence, value alignment is a key technology for achieving effective collaboration between models and humans. Optimizing model value alignment has become a significant research topic. For example, in the field of natural language processing, well-aligned large language models can better understand and respond to specific user needs, thus playing a greater role in various scenarios, such as text creation, knowledge question answering, and brainstorming.

[0053] Alignment optimization can be achieved through model training. Traditional SFT training methods face the challenge of balancing data diversity, quantity, and quality, and have little success in optimizing alignment. In contrast, using preference annotation can more effectively achieve value alignment for models.

[0054] In some technologies, model alignment optimization can be performed using RLHF. In RLHF, human annotators can annotate the output information generated by the model. These annotations can be used to indicate human value preferences for the model output information, thereby guiding the model output alignment process through annotations.

[0055] The RLHF solution leverages the model's generative capabilities and human-biased annotation to optimize model alignment. Compared to SFT, RLHF requires less skill from annotators and is easier to scale to large amounts of data. This is because annotators no longer need to perform time-consuming generative processes and can focus on simple, well-defined annotation tasks.

[0056] However, in the RLHF solution, a large number of labelers are still needed to label the model's output information to complete the value alignment process, which is costly, inefficient, and difficult to ensure data diversity.

[0057] In view of this, an embodiment of the present application provides a value alignment optimization solution. In order to achieve value alignment optimization of the target model, preference data can be constructed using various existing data, a preference determination model can be trained based on the preference data, and the target model can be trained using the preference determination model.

[0058] The target model can be any type of model, such as a large language model, and the preference determination model is used to judge whether the output of the target model conforms to human value preferences, so as to realize the training of the target model.

[0059] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, initial data can be obtained from a variety of channels, such as questions and answers in forums, encyclopedia entries, and text content. This public data is accompanied by a large amount of user behavior, such as user likes and edits. These behaviors can, to a certain extent, reflect the user group's value preferences for data. For example, answers with more likes are generally more consistent with the user group's values.

[0060] Therefore, in the embodiments of the present application, preference data can be automatically constructed using initial data, and the preference data can be used to train a preference determination model. The trained preference determination model can, to a certain extent, distinguish whether a particular answer aligns with human values. Furthermore, the ability of the preference determination model can be leveraged to perform reinforcement learning training on the target model, making the output of the trained target model more consistent with human values.

[0061] pass Figure 1The solution shown can use massive amounts of initial data to construct preference data, reduce the workload of manual labeling, lower costs, improve efficiency, and enhance the diversity of preference data.

[0062] Optionally, manually annotated preference data may be added based on the preference data constructed based on the initial data. Figure 2 Another application scenario diagram provided in the embodiment of the present application is as follows: Figure 2 As shown, in Figure 1 Based on the scheme shown, a question set containing diverse questions can be obtained, and the questions in the question set can be input into the target model. The target model outputs the corresponding answers, and then the annotators annotate the answers with value preferences. Based on the annotated results, preference data is obtained, and the preference determination model is trained. The trained preference determination model is then used to perform reinforcement learning training on the target model.

[0063] Optionally, after a certain amount of training, the target model can be released for use. For example, if the target model is a language model, it can implement functions such as chatting with users, answering questions, and assisting users in completing certain tasks. In actual use, users can enter questions, and the target model generates corresponding answers based on the user's input. After displaying the answers to the user, the user can provide feedback on the answers, such as giving likes. Based on user feedback, preference data can also be constructed, and this preference data can be used to further train the preference determination model and the target model to further improve the performance of the target model.

[0064] In an embodiment of the present application, the preference data automatically constructed using the initial data can be recorded as pre-trained preference data, and the preference data constructed using the feedback information of the annotator or the user can be recorded as fine-tuning preference data. It should be noted that the pre-trained preference data and fine-tuning preference data here are different from the pre-trained data and fine-tuning data in the traditional sense. The traditional pre-trained data and fine-tuning data are used to train the target model, while the pre-trained preference data and fine-tuning preference data constructed in this application are used to train the preference determination model to achieve value alignment with the target model.

[0065] pass Figure 2 The scheme shown can use the initial data to construct pre-trained preference data, and achieve preliminary training of the preference determination model on user preferences without human intervention, so that the preference determination model has a certain ability to distinguish value preferences. On this basis, through a diversified set of questions plus manual labeling, as well as the collection of user feedback, fine-tuning preference data is constructed to achieve fine-tuning of the preference determination model, enhance the preference determination model's accurate recognition of human value preferences, improve generalization ability, and effectively improve the value alignment effect of the target model.

[0066] Optionally, the target model can be trained using reinforcement learning plus flywheel iteration, where the preference determination model can serve as a reward model in the reinforcement learning process. Figure 3 The embodiment of the present application provides a flywheel iterative process based on reinforcement learning. Figure 3 As shown in the figure, after fine-tuning the preference determination model using the fine-tuning preference data, the preference determination model can be used to perform reinforcement learning training on the target model. The trained target model can generate new answers, and new fine-tuning preference data can be constructed based on the new answers. The preference determination model can then be further fine-tuned. After repeating this process multiple times, the training of the target model is completed.

[0067] Through the above methods, reinforcement learning methods can be used to align human preferences, and the preference determination model and target model can be continuously optimized through online iteration and data flywheel to ensure that the target model can produce answers that are highly consistent with human preferences, further improving the value alignment effect of the target model.

[0068] Corresponding to the above Figures 1 to 3 , the embodiment of the present application also provides a specific process of a value alignment optimization solution. Figure 4 A flow chart of a value alignment optimization solution provided in an embodiment of the present application. Figure 4 As shown in the figure, the value alignment optimization of the target model can be achieved through the following steps:

[0069] S1. Question set collection: A large number of diverse questions can be collected for the construction of fine-tuning preference data in S3.

[0070] S2. Pre-training preference data collection: Build pre-training preference data based on the initial data. Since neither S1 nor S2 requires manual labeling, data corresponding to these two steps can be collected in the early stages before executing subsequent steps. Of course, in practical applications, the order of these steps is not restricted; for example, S2 can be executed after S3.

[0071] S3. Fine-tuning preference data collection: Based on the questions collected in S1, fine-tuning preference data is constructed. Specifically, the target model can be used to process the questions and obtain corresponding answers. Annotators then annotate the answers to obtain fine-tuning preference data. Alternatively, fine-tuning preference data can be constructed based on user feedback during actual use.

[0072] S4, preference determination model training: first pre-train the preference determination model based on the pre-trained preference data constructed in S2, and then fine-tune the preference determination model based on the fine-tuning preference data constructed in S3 to further optimize the preference determination model.

[0073] S5. Value preference alignment based on reinforcement learning: Use the preference determination model to perform reinforcement learning training on the target model.

[0074] S6. Online iteration to implement data flywheel: Repeat S3 to build a fine-tuned preference dataset based on the output of the target model, fine-tune the preference determination model in S4, and optimize the target model using the preference determination model in S5 until the performance of the target model meets the requirements.

[0075] based on Figures 1 to 4 The concept shown in the embodiment of the present application is that the value alignment optimization framework based on human preference data can improve the target model by analyzing and utilizing feedback data from real user interactions or annotations by human annotators. While ensuring model performance, it can better adapt to user needs and preferences and improve user experience.

[0076] In addition, this framework has advantages that traditional model value alignment frameworks do not have, such as easy scalability, iterativeness, and high upper limits. It provides an executable implementation plan for the continuous optimization and iteration of the target model, greatly reduces the workload and training difficulty of annotators, and can align the values of humans and models on a larger scale.

[0077] Some embodiments of the present application are described in detail below with reference to the accompanying drawings. In the absence of conflicts between the embodiments, the following embodiments and the features in the embodiments may be combined with each other. The timing of the steps in the following method embodiments is only an example and not a strict limitation. In addition, the deployment mode of the method and apparatus is not limited. For example, it can be locally deployed, implemented in the cloud, implemented on the client, implemented on IOT (Internet of Things) devices, etc.

[0078] For ease of description, the following first describes in detail the process of constructing pre-training preference data and fine-tuning preference data, and then introduces the specific process of reinforcement learning.

[0079] Figure 5 This is a flow chart of a data construction method provided in an embodiment of the present application. Figure 5 As shown, the method may include:

[0080] Step 501: Acquire an initial data set, where the initial data in the initial data set includes basic information, at least one associated information corresponding to the basic information, and behavior information corresponding to each associated information, where the behavior information is used to represent the user's behavior with respect to the associated information.

[0081] Optionally, the initial data set may include at least one initial data, and the initial data may include data obtained from the network (such as a public network or an internal network) or data obtained from a non-network (such as offline copied data), which is not limited in this embodiment. In an optional implementation, the initial data may include data obtained from a web page, such as data displayed on a web page such as social media, online forums, blogs, and encyclopedias, or may also include open source data downloaded directly from the network. The initial data obtained are all compliant and authorized data.

[0082] The initial data may include basic information, associated information, and behavior information corresponding to the associated information, which is used to construct preference data.

[0083] The basic information can be any type of information. If the constructed preference data is ultimately used to align the value of a target model, the basic information can be matched with the input information of the target model. The associated information is any information related to the basic information and can be matched with the output information of the target model.

[0084] Optionally, the target model may be a language model used to process text-related tasks. The basic information includes at least one of the following: requirement information and title information; and the associated information corresponding to the basic information includes at least one of the following: at least one reply message for replying to the requirement information and at least one version description information for describing the title information.

[0085] The requirement information may be any information used to express a user requirement, and the corresponding response information may be information used to resolve the requirement. For example, the requirement information and its corresponding at least one response information may include: a question and at least one answer to the question, a task requirement and at least one implementation solution for achieving the task requirement, etc.

[0086] The title information may be in the form of words or sentences, and the corresponding description information may be content used to expand on the title information. For example, the title information and its corresponding at least one description information may include: an encyclopedia entry and at least one corresponding version of the encyclopedia text, an article title and at least one corresponding version of the article text, etc.

[0087] Both the basic information and the associated information may be information of at least one modality, and the at least one modality may include but is not limited to: text, image, audio, video, etc.

[0088] Figure 6 This is a schematic diagram of the input and output of a target model provided in an embodiment of the present application. Figure 6As shown, the target model can be used to implement at least one of the following functions:

[0089] In the first function, the input information of the target model may include questions, and the output information includes answers corresponding to the questions, thereby realizing the question-answering function.

[0090] In the second function, the input information of the target model may include nouns, and the output information may include explanation information corresponding to the nouns, thereby realizing the noun explanation function.

[0091] In the third function, the input information of the target model may include text, and the output information may include an image corresponding to the text, thereby realizing the function of generating an image based on the text.

[0092] In the fourth function, the input information of the target model may include a video, and the output information may include a description text corresponding to the video, thereby realizing the function of describing the video.

[0093] The above functions are all exemplary functions. In actual applications, the functions that the target model can realize are not limited to these. The input information and output information of the target model can be designed according to the actual task requirements, and the target model can be trained to have the corresponding functions.

[0094] When constructing preference data, matching initial data can be obtained based on the input and output information of the target model. For example, when training the question-answering function of the target model, the basic information and associated information in the obtained initial data can be questions and answers, respectively. When training the target model to generate images from text, the basic information and associated information in the obtained initial data can be text and corresponding images, respectively.

[0095] The target model may be any type of model, and may be a single-modal model or a multi-modal model, which is not limited in this embodiment.

[0096] In addition to basic information and associated information that match the input and output of the target model, the initial data obtained may also include behavioral information corresponding to the associated information. The behavioral information is used to reflect the user's behavior on the associated information. The user here can be used to represent a single user in the network or a user group in the network.

[0097] Optionally, the behavior information can be used to represent at least one of the following user behaviors: evaluation and editing. Evaluation behavior can include any one or more of the following: likes and comments. These behaviors can, to a certain extent, reflect human value preferences, thus providing a basis for constructing preference data.

[0098] Step 502: Construct pre-trained preference data based on the initial data, where the pre-trained preference data includes basic information, associated information, and corresponding preference information.

[0099] The preference information is used to indicate the user's preference for the associated information, and the preference information is determined by the behavior information corresponding to the associated information.

[0100] Optionally, the preference level may refer to a value preference level, that is, the extent to which the associated information is recognized. The preference level may be automatically generated based on the user's behavior with respect to the associated information.

[0101] In an optional implementation, step 502 may include at least one of the following:

[0102] When the basic information includes demand information, in at least one reply message corresponding to the demand information, determining preference information corresponding to each reply message according to evaluation information corresponding to each reply message;

[0103] When the basic information includes title information, preference information corresponding to each version of the description information is determined based on the editing time of each version of the title information. The evaluation information may include at least one of the following: number of likes, number of comments, etc. The number of likes, number of comments, and degree of preference may be positively correlated. That is, the higher the number of likes for the associated information, the higher the user's preference for the associated information, indicating that the associated information is more consistent with the user's value preferences.

[0104] When there are multiple versions of related information, the editing time of each version can also reflect value preferences. Usually, when a version needs to be optimized, it will be updated. Therefore, the new version is generally better than the old version. It can be considered that the later the editing time, the higher the corresponding preference, and vice versa.

[0105] Furthermore, the stability of each version can be determined by the editing time of each version. If the editing time of a version is less than a preset time threshold compared to the editing time of the next version, it can be assumed that the editing of the previous version was not long before the editing of the next version. This may be because there are some problems in the current version that require re-editing after the editing is completed. Therefore, the stability of the current version can be considered poor. If the stability of a version is poor, the preference level of the corresponding version can be reduced.

[0106] The degree of preference is expressed in the form of preference information, which can be qualitative or quantitative. For example, preference information can be good or bad, indicating whether the associated information aligns with or disagrees with the values of the human community. Alternatively, preference information can be quantitative, for example, a score from 0 to 100, where a higher score indicates a higher degree of preference and greater alignment with human values.

[0107] After obtaining the preference information, preference data can be constructed for pre-training the preference determination model. A preference data may include basic information, associated information, and preference information.

[0108] In this embodiment, according to the amount of associated information included in the initial data, one or more preference data may be constructed based on one piece of network information.

[0109] Optionally, one initial data may include one basic information, n associated information corresponding to the basic information, and behavior information corresponding to each of the n associated information, so that n preference data may be constructed, where n is a positive integer greater than or equal to 1.

[0110] For example, initial data is collected from a forum. One initial data may include a question, n answers to the question, and the number of likes corresponding to each answer. Then, n preference data can be constructed, and each preference data corresponds to one answer.

[0111] Through the above method, the Internet can be used to collect questions, answers, encyclopedia data, etc. These data are often accompanied by comments, editing history and other information that can reflect the user's value preferences, so that data that is highly concerned (such as high number of likes and comments) can be distinguished. These data reflect the preferences of a wide range of user groups. Preference data is constructed based on this information, which can achieve rapid construction of preference data. In addition, the evaluation information is used to determine the preference information corresponding to the question and answer data, and the editing time of each version is used to determine the preference information of the encyclopedia data. Preference data can be constructed in a targeted manner according to the characteristics of the initial data, thereby improving the accuracy of the constructed preference data.

[0112] Each of these types of behavioral information can be used individually or in combination. For example, likes and comments can be combined to determine preference information. For encyclopedia data, ratings can be used to determine corresponding preference information. For question-and-answer data, the edit time of the answer version can be used to determine corresponding preference information. Furthermore, in addition to ratings and edit time, other information can also be considered, such as popularity, views, and favorites.

[0113] After constructing the preference data, the preference determination model can be pre-trained using the preference data. The preference determination model can be a model for determining preference information based on basic information and associated information. Specifically, given a set of basic information and associated information, the basic information and associated information are input into the preference determination model, and the output of the preference determination model is the corresponding preference information.

[0114] Optionally, the association information may be generated by the target model or obtained through other means. In the case where the association information is generated by the target model, the preference determination model may be used to evaluate whether the output of the target model conforms to human value preferences.

[0115] Optionally, the preference determination model can be used to perform reinforcement learning training on the target model. During reinforcement learning training, the preference determination model can serve as a reward model, and the obtained preference information can be used as a reward during the reinforcement learning process, thereby achieving training of the target model. In addition to reinforcement learning, the preference determination model can also be used to train the target model through other methods.

[0116] In summary, the data construction method provided in this embodiment can obtain basic information and its corresponding related information, and use the user's behavioral information on related information that already exists in the network to construct preference data. The behavioral information reflects the user group's value preference for related information. Using the constructed preference data to train the preference determination model can enable the preference determination model to have the ability to distinguish whether the related information conforms to human value preferences, improve the effect of value alignment optimization, and reduce the workload of manual labeling, effectively reduce costs, and improve efficiency.

[0117] In one or more embodiments of the present application, before constructing the pre-training preference data, the initial data may be evaluated and screened.

[0118] Optionally, multiple initial data can be obtained and the labels corresponding to each initial data can be determined, wherein the labels corresponding to the initial data include labels corresponding to the basic information in the initial data and / or labels corresponding to the associated information; any initial data corresponds to at least one label; based on complexity and / or diversity, part of the initial data is selected from the multiple initial data to construct the initial data set; wherein the complexity is used to represent the number of labels, and the diversity is used to represent the coverage of the labels.

[0119] Specifically, multiple initial data can be obtained first, and labels corresponding to each initial data can be determined. The labels can be obtained through classifiers or keyword matching. The labels of the initial data can be determined by the basic data, the associated information, or both. Each initial data can correspond to one or more labels.

[0120] After obtaining the labels corresponding to the multiple initial data, some initial data can be selected from the multiple initial data according to the labels to construct an initial data set. The principle of constructing the initial data set is: the higher the complexity and diversity, the better.

[0121] The complexity can be specifically expressed by the number of labels in the initial data. The more labels there are, the higher the complexity of the initial data. For example, if initial data i has 10 labels and initial data j has 3 labels, then the complexity of initial data i is higher than that of initial data j.

[0122] Diversity refers to the coverage of labels. Specifically, it can be used to represent the coverage of labels by the initial dataset. When constructing the initial dataset, it is hoped that the initial dataset can cover as many labels as possible, so that the initial data in the initial dataset has high diversity.

[0123] Optionally, a complexity-first diversity sampling algorithm may be used to construct an initial data set. Specifically, based on complexity and / or diversity, some initial data are selected from a plurality of initial data to construct the initial data set, including:

[0124] Sort multiple initial data according to complexity to obtain an initial data queue;

[0125] Process the initial data in the initial data queue in sequence until the stop condition is met;

[0126] The processing of the initial data includes: if at least one label corresponding to the initial data has at least some labels that are different from labels corresponding to existing initial data in the initial data set, then adding the initial data to the initial data set.

[0127] Figure 7 This is a schematic diagram of the principle of constructing an initial data set provided in an embodiment of the present application. Figure 7 As shown, multiple initial data can be sorted by complexity, where the complexity of each initial data is the number of tags corresponding to the initial data. The more tags there are, the higher the ranking.

[0128] The initial data queue obtained after sorting is: initial data 1, ..., initial data i, ..., initial data j, ..., initial data N. In the queue, the number of labels corresponding to each initial data decreases in sequence.

[0129] Sequentially traverse the initial data in the initial data queue. For the currently traversed initial data, perform the following processing: determine whether at least one tag corresponding to the initial data has been covered by the tags corresponding to the current initial data set. In other words, determine whether there is a tag among the at least one tag corresponding to the current initial data that is not in the current initial data set. If not, it means that the tags corresponding to the current initial data have been covered by the current initial data set. In this case, the current initial data can be discarded and the next initial data can be processed. If so, it means that at least one tag of the current initial data is not fully covered by the current initial data set. In this case, the current initial data can be added to the initial data set and the next initial data can be processed.

[0130] The labels corresponding to the current initial data set may be all labels of the initial data currently existing in the initial data set.

[0131] See also Figure 7 First, add initial data 1 to the initial dataset and continue processing subsequent initial data. When processing initial data i, it is found that the labels corresponding to initial data i already exist in the initial dataset. In this case, initial data i can be discarded. When processing initial data j, it is found that initial data j still has labels that are not covered by the current initial dataset. In this case, initial data j can be added to the initial dataset. And so on, until the stopping condition is met.

[0132] Optionally, the stop condition may be: all initial data in the initial data queue are processed, or the amount of initial data in the initial data set reaches a first preset amount, or the number of tags covered by the initial data set reaches a second preset amount.

[0133] Through the above method, a complex and diverse initial data set can be constructed according to complexity and diversity, and preference data can be constructed based on the initial data set, so that the preference data is not only rich and diverse, but also covers as many user preferences and behavior patterns as possible that the preference determination model needs to identify, thereby improving the data processing complexity of the preference determination model and training the processing capabilities of the preference determination model in various situations as evenly as possible.

[0134] In addition to the complexity-first diversity sampling algorithm, it is also possible to construct the initial dataset using only diversity or only complexity. For example, if only diversity is considered, one can first randomly select an initial data point and add it to the initial dataset. Then, from the remaining initial data, select the initial data point with the most different labels from the labels in the current initial dataset and add it to the initial dataset. This process continues until the label range covered by the initial dataset meets certain requirements. Alternatively, if only complexity is considered, the initial dataset can be constructed using the portion of initial data with the highest complexity.

[0135] Optionally, before constructing the pre-training preference data based on the initial data, the initial data may be filtered according to a filtering condition, where the filtering condition includes at least one of the following:

[0136] the editing time and / or language style of the associated information;

[0137] Whether the basic information and / or related information belongs to the preset topic.

[0138] Optionally, the filtering conditions can be set based on actual needs. For example, the editing time of the associated information can be set to the most recent period, thereby filtering out answers from very old times. The corresponding language style and preset topics can be determined based on the usage scenario of the target model to be trained. For example, language styles may include: concise, detailed, polite, serious, etc., and topics may include news and technology.

[0139] Optionally, you can also set other filtering conditions, such as whether it contains sensitive information, whether it is low-quality information, the number of words needs to be within a certain range, etc.

[0140] Optionally, the basic information and / or associated information may be input into a large model or classifier to determine whether there is sensitive information, low-quality information, or whether it belongs to a preset language style or a preset topic.

[0141] Through the above method, different filtering conditions can be set according to different scenarios, so as to filter the related information and basic information, so that the initial data obtained meets the pre-set standards, such as language style, topic relevance and other related standards, to meet the actual needs in different scenarios.

[0142] Optionally, an automated filtering and classification system can be constructed. The system can first filter the initial data and then construct an initial data set based on principles such as complexity and diversity, so as to efficiently implement the evaluation and screening operations of the initial data, and use a massive number of networks to obtain pre-trained preference data at the level of millions or even higher. The entire process does not require manual labeling, and the constructed preference data has a wide range of and diverse performance, laying a good foundation for the subsequent training of the preference determination model.

[0143] The following is a detailed introduction to the construction process of fine-tuning preference data.

[0144] The process of constructing fine-tuning preference data may specifically include: obtaining a basic information set, wherein the basic information set includes at least one basic information; processing the basic information in the basic information set based on the target model to obtain corresponding associated information; and obtaining fine-tuning preference data by labeling the associated information output by the target model.

[0145] The fine-tuning preference data includes basic information, corresponding associated information, and preference information for the associated information, and the preference information is determined by the annotation result.

[0146] Optionally, the basic information in the basic information set may be the same as or different from the basic information used to construct the pre-trained preference data. In some implementations, the basic information may be reused. In other implementations, other basic information may be obtained to construct the fine-tuning preference data.

[0147] For example, various types of questions can be collected as basic information from channels such as user usage log records, annotator annotation records, open source data sets, etc. that meet the requirements, and a basic data set can be constructed.

[0148] Optionally, the collected basic information may be screened to obtain a basic data set. Specifically, multiple basic information may be obtained, tags corresponding to each basic information may be determined, and some basic information may be selected from the multiple basic information according to the tags to construct a basic information set.

[0149] The labels corresponding to any basic information include labels corresponding to at least one of the following dimensions:

[0150] Intent, used to represent the task corresponding to the basic information, such as rewriting task, summary generation task, etc.

[0151] Knowledge, which is used to represent the industry to which the basic information relates, such as sports, economics, computer science, etc.;

[0152] Scenario, used to represent the corresponding tool usage behavior when processing the basic information;

[0153] Among them, the tool usage behavior includes at least one of the following: calling an API (Application Programming Interface), querying documents, using plug-ins, and using a multimodal module; the multimodal module is used to process modalities other than the target modality, and the target modality is a modality that can be processed by the target model.

[0154] Optionally, when processing more complex basic information, it is often necessary to use certain tools, which may include APIs, document systems, plug-ins, etc.

[0155] For example, if the basic information is used to answer a question based on a certain version of a legal document, the corresponding legal document can be searched in the document system first, and then the target model can use the legal document to provide the corresponding answer to the question. In this case, the corresponding tool usage behavior can include querying the document.

[0156] Alternatively, if the basic information is used to answer a user's question based on their historical payment records, the payment query API can be called first to determine the corresponding query record, and then the target model can answer the question based on the query record. In this case, the corresponding tool usage behavior may include calling the API.

[0157] Alternatively, the basic information is used to represent the answer generated based on the question, and the answer is displayed in the form of a flowchart for the user to view. In this case, the target model can generate a textual answer, and then call a plug-in for drawing a flowchart to obtain the corresponding flowchart. The corresponding tool usage behavior may include the use of the plug-in.

[0158] Furthermore, tool usage can also include the use of multimodal modules. Specifically, the target model is generally a model that can process one or more modalities. If there are modalities that the target model itself cannot handle, the capabilities of the multimodal model can be used to process the underlying information.

[0159] Exemplarily, the target model is a language model that can process text and cannot directly process images or other modal information. When the basic information is used to represent the processing of an image, the image can be first input into a multimodal module that can process the image to obtain the text representation corresponding to the image, and then input into the target model for processing.

[0160] Based on the three dimensions of intent, knowledge, and scenario, a fine-grained classification system can be designed to obtain labels corresponding to basic information. Each basic information can correspond to one or more labels.

[0161] A simple example is that the basic information is "Help me rewrite the following sports news release with text and pictures". After passing through the classification system, the basic information will be labeled as follows: rewriting (intention), sports, news (industry knowledge), image processing (scene).

[0162] Optionally, the classification system can be implemented based on a large model and a separately trained scene classification model. Specifically, determining the label corresponding to each basic information may include at least one of the following:

[0163] For any basic information, the basic information and the intention analysis prompt information are input into the large model to obtain a label corresponding to the intention, wherein the intention analysis prompt information is used to prompt the large model to analyze the intention of the basic information;

[0164] Inputting the basic information and knowledge analysis prompt information into the big model to obtain labels corresponding to the knowledge, wherein the knowledge analysis prompt information is used to prompt the big model to analyze the industry involved in the basic information;

[0165] The basic information is input into a scene classification model to obtain a label corresponding to the scene; wherein the scene classification model is a trained model.

[0166] For example, the intent analysis prompt could be: "Determine the intent corresponding to the following question," and the knowledge analysis prompt could be: "Determine which industry knowledge is used in the following question." The large model can output intent labels and knowledge labels corresponding to the basic information based on the relevant prompts.

[0167] Scene-related labels require information about available tools. Large models may not be able to process scene labels. Therefore, a separate scene classification model can be trained to extract scene labels corresponding to the basic information.

[0168] After obtaining labels corresponding to multiple pieces of basic information, the basic information can be filtered based on complexity and diversity to construct a basic information set. The definitions of complexity and diversity, as well as the specific implementation process for constructing a basic information set based on complexity and diversity, can be found in the process of constructing the initial data set in the previous embodiment. Simply replace the initial data with the basic information.

[0169] For example, the basic information is questions. According to the three dimensions of intent, knowledge, and scenario, a 6000 fine-grained classification system can be constructed. By inputting the questions into the classification system, the corresponding labels can be obtained. The more labels a question has, the higher the complexity. The more complex the problem, the more helpful it is to improve the performance of the model. Sort multiple questions by complexity, and sample the sorted questions according to the principle of label diversification. During the sampling process, try to collect labels that have not appeared in the current problem set. If a question has 8 labels, and all 8 labels have appeared in the current problem set, skip the question and continue to process the next question. If one or more of the 8 labels still appear in the current problem set, put the question into the problem set. Continue collecting according to this method until all 6000 labels are covered.

[0170] Through the above method, labels for basic information can be constructed based on the three dimensions of intent, knowledge, and scenario. This allows the labels to reflect not only the task type and industry knowledge used in the basic information, but also the way the basic information is processed, thereby comprehensively reflecting the nature of the basic information and obtaining fine-grained classification labels corresponding to the basic information. The labels can then be used to filter the basic information more finely. The filtered basic information can comprehensively improve the processing capabilities of the target model and the preference determination model in all aspects, thereby improving the overall training effect of the model. Furthermore, in the process of filtering basic information, a complexity-first diversity sampling algorithm can be used to analyze and filter multiple existing alternative basic information, filtering out diverse and complex basic information from the alternative basic information to improve the upper limit of the model and cover balanced capability tasks.

[0171] After the basic information set is constructed, the basic information in the basic information set can be processed based on the target model to obtain corresponding associated information. For example, the basic information can be input into the target model, and the target model generates the corresponding associated information.

[0172] After obtaining the associated information, the associated information can be annotated to obtain fine-tuned preference data, wherein the annotation work can be completed by annotators.

[0173] Optionally, fine-tuning preference data is obtained by annotating the associated information output by the target model, including:

[0174] The annotation specification, basic information and associated information output by the target model are displayed through an interactive interface, so that the annotator can annotate the associated information according to the annotation specification and basic information. The obtained annotation result is used to indicate the degree to which the associated information meets the annotation specification;

[0175] According to the labeling results, fine-tuning preference data is obtained.

[0176] The annotation standards include at least one of the following dimensions: compliance with basic information requirements, conformity to facts, language expression that meets requirements, and no risk information. Specifically, it can be expressed in the following forms:

[0177] Compliance with instructions, used to measure whether the associated information complies with the requirements of the basic information;

[0178] Factual accuracy, which measures whether the associated information is true to the facts;

[0179] Language fluency, used to measure whether the language expression of related information meets the requirements;

[0180] Safe and unbiased, used to measure whether the associated information contains risk information.

[0181] The annotation specifications can be in the form of text, which is displayed to the annotators. The annotators score the related information according to the annotation specifications as the annotation results.

[0182] In the first dimension, instruction compliance measures whether the target model complies with the input instructions, that is, whether it provides corresponding associated information according to the requirements of the basic information. Optionally, instruction compliance can include both intent and constraints.

[0183] Among them, intent refers to the user's intention reflected by the basic information, and constraints refer to the requirements other than the intent reflected in the basic information.

[0184] For example, if the basic information is "Please write a poem about the sun and flowers in 50 words or less," the corresponding intent is "Write a poem," and the constraints are: include the sun, include flowers, and be within 50 words. Command compliance depends not only on whether the relevant information output by the target model satisfies the "Write a poem" intent, but also on whether the relevant constraints are met.

[0185] Optionally, the annotation specification can provide specific scoring rules. For example, if the maximum score for instruction compliance is 10, then if the intent and all constraints are met, the score is 10. If the intent is met, but only some constraints are met, and some constraints are not met, the score is 7. If only the intent is met, but all constraints are not met, the score is 5. If neither the intent nor the constraints are met, the score is 0.

[0186] The above scoring rules are only examples. In actual applications, the scores and scoring rules can be adjusted as needed.

[0187] In the second dimension, factual accuracy can also be further included in multiple standards: 1. Whether there is any content that contradicts the facts of the real world, for example, whether the area of a certain region appearing in the answer is consistent with the facts; 2. Whether there is any contradiction in logical reasoning, for example, the process of mathematical reasoning should be correct; 3. Whether there is any situation that cannot be executed, for example, the answer given includes code, the code needs to be accurate and executable, and it must be able to achieve the corresponding function; 4. Whether the answer conflicts with the context.

[0188] In the third dimension, language fluency can include: 1. Accurate and easy-to-read language expression; 2. Neat paragraphs with correct introduction and development relationships; 3. No formal errors such as punctuation; 4. In addition to the language itself, some style and layout requirements can be set to determine whether the related information meets these requirements.

[0189] In the fourth dimension, the requirements for safety and bias-freeness can be specifically: not containing sensitive content, biased discrimination, or violating the law. If such risk information exists in the related information, the corresponding score can be lowered.

[0190] The annotator can annotate each of the above four dimensions and obtain corresponding scores. Based on the scores of the four dimensions, the final score of the associated information can be obtained as preference information.

[0191] By providing a multi-dimensional annotation specification framework, requiring annotators to score based on four key dimensions: instruction compliance, factual accuracy, language fluency, and safety and bias-free, and constructing preference data based on the scores, the final trained target model can ensure that its answers to various tasks meet the requirements of instruction compliance, accuracy with facts, fluent language style, and safety and bias-free, thereby more comprehensively achieving alignment of human preference values.

[0192] Optionally, the preference determination model is used to perform reinforcement learning training on the target model; the reinforcement learning training of the target model includes multiple rounds. When the annotation specification includes multiple dimensions, the method further includes:

[0193] Determine the importance of each dimension in the multiple dimensions based on the current round and / or the performance of the target model and the preference determination model, so as to determine the corresponding preference information based on the importance of each dimension and the labeling results of the associated information in each dimension after obtaining the associated information based on the current target model;

[0194] The importance level includes weight or priority.

[0195] For example, importance can be represented by weights, with different weights corresponding to different dimensions. The scores of the four dimensions are weighted and summed to obtain a final score as the preference information. Alternatively, importance can be represented by priority. For multiple related information corresponding to a basic information, the highest priority dimension is first found. Then, the scores of the multiple related information on the highest dimension are determined to be consistent. If they are inconsistent, the score of that dimension is used as the final score. If they are consistent, the scores of the multiple related information on the next priority dimension are determined to be consistent, and so on, until the final score is obtained.

[0196] After each round of training, the trained target model can be used to process the basic information to obtain new associated information. This new associated information is then annotated to obtain scores corresponding to the four dimensions. The scores of the four dimensions are then integrated according to their importance to obtain the final score, thereby constructing preference data for the next round of training of the preference determination model and the target model. The importance of the four dimensions can be continuously adjusted during the iterative training process of the target model.

[0197] Optionally, in the early stages of training, for example, in the first few rounds, more attention is paid to the effective helpfulness of the target model to users, so the weights of instruction compliance and factual accuracy can be higher. After a certain number of rounds of training, more attention is paid to safety, and the weight of safety and unbiasedness can be increased.

[0198] Alternatively, the importance of each dimension can be adjusted according to the performance of the target model and the preference determination model during the training process. The performance of the target model and the preference determination model can be tested using test data. If the target model or the preference determination model currently has a better processing effect on some dimensions and a worse processing effect on other dimensions, the weights of the other dimensions can be increased.

[0199] In this way, the importance of the four dimensions is adjusted according to the progress of training, so that the target model and the preference determination model can learn human value preferences according to the expected learning process, which facilitates more targeted training on weaker dimensions or more important dimensions, and further improves the training effect.

[0200] Optionally, the method further includes: inputting basic information into a large model to obtain an example of associated information; and displaying the example of associated information through an interactive interface so that annotators can annotate the associated information according to the example of associated information.

[0201] The large model can be a model with stronger generative capabilities than the current target model. Based on the large model's capabilities, it can generate examples of related information. For example, a question can be input into the large model to generate corresponding example answers. These example answers can then be displayed to annotators, who can then use the example answers to annotate the answers output by the target model.

[0202] Optionally, when the instruction compliance is specifically used to measure whether the associated information complies with the intention and constraints of the basic information, the basic information is input into the large model to obtain an example of the associated information, which may include:

[0203] The basic information and output prompt information are input into the big model, and the output prompt information is used to prompt the big model to determine the intention and at least one constraint corresponding to the basic information, generate an associated information example based on the intention and at least one constraint, and output the intention, at least one constraint and the associated information example.

[0204] Correspondingly, displaying the associated information example through the interactive interface includes: displaying the intention corresponding to the basic information, at least one constraint condition and the associated information example through the interactive interface, so that the annotator can annotate the associated information according to the intention corresponding to the basic information, at least one constraint condition and the associated information example.

[0205] Specifically, when the target model processes basic information, it is relatively easy to understand the intent contained in the basic information. However, the understanding and execution of the constraints are not necessarily in place. Therefore, in the embodiment of the present application, instruction compliance not only takes into account the intent, but also the constraints. When obtaining examples of related information through the large model, the intent and constraints can also be used to optimize the output of the large model and the annotation work of the annotator.

[0206] Outputting prompt information allows the large model to first analyze the intent and at least one constraint involved in the basic information, and then generate examples of related information based on the intent and at least one constraint, improving the effectiveness of the large model in generating related information examples. Furthermore, presenting the intent, at least one constraint, and the related information examples together to the annotator facilitates their annotation of the related information generated by the target model based on the displayed content, improving annotation efficiency and consistency.

[0207] The above provides a solution for obtaining fine-tuning preference data through the annotation operations of annotators. In addition, the fine-tuning preference data used for fine-tuning the preference determination model can also include fine-tuning preference data constructed based on user feedback information. The fine-tuning preference data constructed based on user feedback information can be constructed in the following ways:

[0208] Get the basic information to be processed input by the user;

[0209] Generate associated information corresponding to the basic information to be processed through the target model, and output the generated associated information to the user;

[0210] Obtain user feedback on the associated information, and construct fine-tuning preference data based on the feedback information, wherein the fine-tuning preference data constructed based on the feedback information includes: the basic information to be processed, the generated associated information and the corresponding preference information, and the preference information is determined by the feedback information.

[0211] Specifically, after training, the target model can be released for use. During use, users can input basic information, such as questions, and the target model can generate corresponding related information, such as answers. After the answers are displayed to users, users can provide feedback on the answers, such as likes or dislikes. Based on the user's feedback information, fine-tuned preference data can be constructed. For example, answers with likes will have higher scores, and answers with dislikes will have lower scores.

[0212] In this way, before the target model is released, the target model can be trained using the fine-tuning preference data obtained by annotators. After the target model is released, the target model can be further optimized using the fine-tuning preference data constructed based on user feedback. Alternatively, the fine-tuning preference data constructed based on user feedback and the fine-tuning preference data obtained by annotators can be combined to optimize the target model.

[0213] The data annotated by human annotators is consistent with annotation specifications, but it may not be completely consistent with the needs of real users. Through the above feedback mechanism, not only can the distribution of preference data be continuously adjusted and optimized, but the target model can also more accurately identify and adapt to the changing preferences of different user groups.

[0214] In summary, during the construction phase of fine-tuning preference data, we focus on constructing fine-tuning preference data based on the collected basic information set and the associated information generated by the target model. Fine-tuning preference data can fine-tune the pre-trained preference determination model and improve the effect of the preference determination model. In addition, by establishing a user feedback reflow data pipeline, we can collect and analyze user feedback on the output of the target model in real time, so that the target model can be aligned with human preferences on a wider scale, achieving significant improvements in the model's instruction compliance, factuality, language style, and security.

[0215] In one or more embodiments of the present application, optionally, a construction strategy for pre-trained preference data and / or fine-tuned preference data input by a customer may also be obtained, wherein the construction strategy includes at least one of the following:

[0216] Screening conditions, used to filter initial data or basic information;

[0217] The sampling strategy is used to obtain the initial data set or basic information set by sampling according to the labels of the initial data or basic information.

[0218] Here, the customer refers to the role used to build pre-trained preference data / fine-tune preference data. For example, when the trained target model is used to interact with users as an intelligent customer service, the customer can be an enterprise that provides intelligent customer service functions to users, and the relevant capabilities of building preference data and training models can be opened to the enterprise, such as screening and sampling strategies.

[0219] For example, the screening conditions may include: editing time, language style, whether it belongs to a preset topic, etc. The sampling strategy may include: a complexity-first diversity algorithm, an algorithm that only considers complexity, or an algorithm that only considers diversity, etc.

[0220] For example, if a corporate customer hopes that the intelligent customer service provided to users has characteristics such as a concise style and expertise in sports topics, relevant conditions can be added to the filtering conditions, so that data that meets the corporate customer's requirements can be filtered out from the massive data on the Internet to construct preference data, thereby improving the performance of the preference determination model under relevant conditions, and then improving the processing ability of the target model under relevant conditions, improving the effect of the target model's interaction with the user, and improving the user experience.

[0221] Figure 8 A flow chart of a model training method provided in an embodiment of the present application. Figure 7 As shown, the method includes:

[0222] Step 801: Obtain pre-trained preference data, and pre-train a preference determination model based on the pre-trained preference data.

[0223] The pre-trained preference data includes the pre-trained preference data obtained by the method described in any of the aforementioned embodiments.

[0224] Optionally, the preference determination model may be any trainable model, such as a large language model or other models.

[0225] During the pre-training process, the basic information and associated information in the pre-trained preference data can be input into the preference determination model, and the preference determination model outputs the corresponding predicted preference information. The predicted preference information is compared with the preference information in the pre-trained preference data, and the parameters of the preference determination model are adjusted according to the comparison results, so that the output of the trained preference determination model is close to the preference information in the pre-trained preference data.

[0226] By initially training the preference determination model on pre-trained preference data, the preference determination model can learn the common sense discrimination to identify and process various types of data, thereby improving its generalization ability on heterogeneous data distributions.

[0227] Step 802: Obtain fine-tuning preference data, and perform fine-tuning training on the pre-trained preference determination model according to the fine-tuning preference data.

[0228] The fine-tuning preference data includes the fine-tuning preference data obtained by the method described in any of the aforementioned embodiments.

[0229] The fine-tuning training process is similar to the pre-training process, in that both require the preference determination model to output the desired preference information based on the input basic information and associated information. The difference is that the associated information in the pre-trained preference data is mined from existing data (user-generated content or reviews), while the associated information in the fine-tuning preference data is generated by the target model. Furthermore, the preference information in the pre-trained preference data is automatically derived through data analysis, while the preference information in the fine-tuning preference data is manually annotated by human annotators.

[0230] In this embodiment, the preference determination model is trained using preference data automatically constructed from massive data, so that the preference determination model has the ability to preliminarily evaluate the output of the target model. The preference determination model is then further trained using the refined labeled preference data. The preference determination model is further adjusted and optimized, which can improve the performance of the preference determination model, make the output of the preference determination model more in line with the needs of actual users, ensure that the performance of the preference determination model in specific situations is more accurate and efficient, and effectively improve the training effect of the preference determination model.

[0231] Optionally, a model may be determined based on the preferences after fine-tuning training, and reinforcement learning training may be performed on the target model. Furthermore, the performance of the target model may be improved through a feedback loop.

[0232] Specifically, the process of obtaining associated information based on the target model, constructing fine-tuning preference data based on the associated information, fine-tuning the preference determination model based on the fine-tuning preference data, and performing reinforcement learning training on the target model based on the preference determination model can be repeated until the preset requirements are met.

[0233] The preset requirements may include: reaching a certain number of iterations, or the performance of the target model meeting the requirements, etc.

[0234] Optionally, during the iteration process, obtaining associated information based on the target model may refer to inputting basic information into the target model to obtain corresponding associated information. The basic information here may include basic information to be processed input by the user, or may include basic information in the basic information set.

[0235] Constructing the fine-tuning preference data based on the associated information may include constructing the fine-tuning preference data based on user feedback on the associated information, or may include obtaining the fine-tuning preference data based on annotations by annotators.

[0236] In practical applications, after the target model is released, user interaction data can be continuously collected through continuous online iteration. This data can be used to further improve the discrimination ability of the preference determination model between high-partitions, correct heterogeneous distribution distortion, and make up for the defects of the target model.

[0237] Specifically, in the initial stage, the output quality of the target model was uneven. The preference determination model had the ability to distinguish between good answers (high-scoring answers) and bad answers (low-scoring answers). However, its ability to distinguish between good answers (higher-scoring answers) and better answers (higher-scoring answers) was not strong enough. After iterative training, the target model's generation ability was greatly improved, and it could output more good answers. The preference determination model could further improve its ability to distinguish between high-scoring areas, and achieve better performance in the distribution of good and better answers.

[0238] Furthermore, as iterative training continues, the performance of the target model continues to improve, which can continuously improve the alignment effect of the target model and fix some discovered bad examples. Bad examples here refer to outputs that do not meet user requirements.

[0239] A simple example is that the target model initially tends to output longer and more detailed answers, which has the defect of redundant answers and does not meet user needs. After multiple rounds of iterative training, the preference determination model will tend to give low scores to longer answers. After further training, the target model will be more inclined to output concise answers to better meet user needs.

[0240] Optionally, during the iteration process, the learning progress and performance of the preference determination model and target model can be closely monitored to ensure that the preference determination model is identifying human preferences as expected. Optionally, corresponding test data can be designed for the aforementioned dimensions such as instruction compliance, factual accuracy, language fluency, and safety and bias-freeness. Based on the test data, the performance of the preference determination model and target model in each dimension can be determined, and relevant strategies can be adjusted based on the performance, for example, adjusting the importance of the four dimensions and the distribution of preference data.

[0241] Through the above-mentioned iterative process, not only can the target model and preference determination model continue to improve in actual application scenarios, but the target model can also respond to user needs and market changes more quickly and accurately, helping the target model to always maintain a better state and have wide applicability.

[0242] Optionally, when performing reinforcement learning training on the target model, a PPO (Proximal Policy Optimization) algorithm can be used to make the target model's decisions more closely aligned with human preferences. Optionally, the target model in this embodiment can serve as the policy model in PPO, used to determine the corresponding action based on the current state. The preference determination model can score the output of the target model, and the score can serve as a reward. After a certain calculation, the reward can be used to update the parameters of the policy model, thereby optimizing the policy model.

[0243] Optionally, the corresponding loss in the reinforcement learning process can refer to the following formula:

[0244]

[0245] Among them, L CLIP (θ) represents the loss function; θ represents the policy parameters, specifically the parameters of the policy model (i.e., the target model) in PPO; It is the probability ratio of the new strategy to the old strategy, that is, the change ratio of the strategy probability.

[0246] s t is the state, corresponding to the input of the target model, a t is the action, corresponding to the output of the target model; A t is an advantage function, which is used to evaluate the advantage of an action relative to the average situation; clip is a cutting function, which is used to limit the ratio to a certain range. ∈ is a hyperparameter that can be set according to actual needs.

[0247] Optionally, the model involved in reinforcement learning may also include a critic model. Specifically, the model is determined based on the preferences after fine-tuning training, and reinforcement learning training is performed on the target model, which may include:

[0248] Freezing the parameters of the target model and training the evaluation model for a preset number of steps; wherein the evaluation model is used to determine the expected reward during the reinforcement learning process, and during the training process for the preset number of steps, updating the parameters of the evaluation model based on the expected reward and the actual reward, wherein the actual reward is determined by the reward obtained by the preference determination model, and the reward is the preference information generated by the preference determination model based on the output of the target model;

[0249] After a preset number of steps of training, the parameters of the target model are unfrozen so that in the subsequent number of steps of training, the parameters of the target model are updated based on the evaluation model and the preference determination model.

[0250] Specifically, during reinforcement learning, at each time step, the target model determines the corresponding action based on the current state. The preference determination model is used to generate the corresponding reward for that time step based on the action obtained by the target model at that time step. This reward is an immediate reward that reflects the advantage of taking the corresponding action at that time step. The evaluation model can be used to determine the expected return based on the current state. The return is a long-term value expression used to represent the cumulative reward over multiple time steps.

[0251] When performing reinforcement learning, you can first freeze the parameters of the target model and train the evaluation model for a preset number of steps. In this embodiment, freezing refers to fixing the parameters of the target model so that they remain unchanged during training and are not updated. Unfreezing refers to restarting parameter updates on the previously frozen model.

[0252] During the training process of a preset number of steps, the parameters of the target model remain unchanged, and only the parameters of the evaluation model are updated. The parameters of the evaluation model can be updated by the critic loss (critic loss), which can be determined by the expected return and the actual return. The expected return is the output of the evaluation model, and the actual return can be determined by the reward obtained by the preference determination model.

[0253] After training for a preset number of steps, the evaluation model's capabilities have been optimized to a certain extent, and the parameters of the target model can be unfrozen, thereby continuously updating the parameters of the target model through training to achieve optimization of the target model. The preset number of steps can be set according to actual needs, for example, 200 or 300 steps.

[0254] By fixing the target model in reinforcement learning and training the evaluation model for a preset number of steps, the evaluation model can output more accurate rewards in subsequent steps, thereby improving the overall effect of reinforcement learning and effectively improving the performance of the target model.

[0255] In one or more embodiments of the present application, optionally, in order to further improve the training effect, the parameters of the target model may be updated according to the performance of the target model on various tasks.

[0256] Specifically, determining the model based on the preferences after fine-tuning training and performing reinforcement learning training on the target model may include:

[0257] Determine the model based on the preferences after fine-tuning training and determine the corresponding loss for reinforcement learning;

[0258] Obtain unsupervised training data corresponding to the pre-training task and SFT data corresponding to the supervised fine-tuning SFT task;

[0259] Determining a pre-training loss of a target model on a pre-training task and an SFT loss on an SFT task based on the unsupervised training data and the SFT data;

[0260] A mixed loss is determined based on the loss corresponding to the reinforcement learning, the pre-training loss, and the SFT loss, and the parameters of the target model are updated based on the mixed loss.

[0261] Among them, unsupervised training data can include text paragraphs, etc., allowing the target model to predict other characters based on certain characters in the text paragraphs; SFT data can include dialogue data, such as questions and answers, etc., allowing the target model to predict answers based on questions.

[0262] In practical applications, since reinforcement learning may be unstable, when using PPO for training, pre-training tasks and SFT tasks can be added to perform multi-task learning. According to the losses corresponding to multiple tasks, a total loss is obtained as a mixed loss, and the parameters of the target model are updated according to the mixed loss.

[0263] By mixing pre-training loss and SFT loss into reinforcement learning, the respective advantages of reinforcement learning, pre-training, and SFT can be brought into play, improving the stability of training and the final performance of the target model.

[0264] Optionally, during reinforcement learning, you can also use techniques such as the KL (Kullback-Leibler) constraint to improve training results. The KL constraint uses the KL divergence as a penalty term during reinforcement learning to prevent the target model from deviating too far during training.

[0265] In summary, the method for constructing preference data and model training provided in the embodiments of the present application has at least the following effects:

[0266] (1) Collection of diversified preference data: including pre-trained preference datasets, fine-tuned preference data based on multi-dimensional scoring, and user return feedback data. Among them, the multi-dimensional scoring annotation method can annotate more and more accurate preference data within the capabilities of the annotator, thereby improving the resolution of the preference determination model.

[0267] Furthermore, a two-stage training approach was employed: first, training on pre-trained preference data to improve the generalization of the preference determination model to externally distributed data, and then fine-tuning the preference data to enhance the accuracy of the preference determination model. By combining user feedback data with value alignment technology, the target model's optimization capabilities in areas such as instruction compliance, factuality, language style, and security were enhanced, efficiently producing model outputs that were highly consistent with human preferences, significantly improving the practicality and accuracy of the target model.

[0268] (2) Human Preference Alignment Based on Reinforcement Learning: The PPO algorithm is adopted and improvements are made in the evaluation model and loss to improve the stability and effectiveness of training. In addition, through multiple rounds of online iteration, the discrimination ability of the preference determination model is continuously improved, especially the discrimination ability between high-level partitions. Bad examples are promptly repaired to ensure the continuous progress and improvement of the preference determination model, thereby improving the performance of the target model.

[0269] Corresponding to the above-mentioned model training method, an embodiment of the present application also provides an information processing method, including: obtaining basic information to be processed; inputting the basic information into the target model to obtain corresponding associated information, and outputting the associated information to the user; wherein, the target model is trained by the method described in any of the above-mentioned embodiments.

[0270] Optionally, the basic information and associated information may be any type of information in any field and scenario.

[0271] The specific implementation principles, processes and beneficial effects of the information processing method provided in this embodiment can be found in the aforementioned embodiments and will not be repeated here.

[0272] An embodiment of the present application also provides an information processing method, which is applied to intelligent customer service scenarios. Figure 9 This is a flow chart of an information processing method applied to intelligent customer service scenarios provided by an embodiment of the present application. Figure 9 As shown, the method includes:

[0273] Step 901: Obtain the question input by the user.

[0274] Step 902: Input the question into the target model, obtain the corresponding answer, and output the answer to the user.

[0275] The target model is obtained by training using the method described in any of the aforementioned embodiments.

[0276] Specifically, after training the target model using the aforementioned method, the target model can be put into use. In actual use, users can enter questions online, and the target model can generate corresponding answers based on the questions. The answers can be displayed to the user, thereby utilizing the target model to implement intelligent customer service functions.

[0277] Optionally, the information processing method may further include the following steps:

[0278] Step 903: Obtain user feedback information on the answer.

[0279] Optionally, the feedback information may be used to indicate the user's evaluation of the answer, which may include likes, dislikes, ratings, etc.

[0280] Step 904: If the feedback information indicates that the answer does not meet the user's needs, regenerate the answer based on the feedback information and the question, and output the regenerated answer to the user.

[0281] Optionally, if the user's feedback information for the answer is a thumbs down or a low score, it is considered that the answer does not meet the user's needs. At this time, the answer can be regenerated and the new answer can be displayed to the user after the answer is regenerated.

[0282] After the new answer is displayed to the user, the user can further input feedback information on the new answer. For example, when the new answer meets the user's needs, the user may like the new answer or give a higher score.

[0283] Step 905: Generate corresponding preference information based on the feedback information to obtain preference data including questions, questions and answers, and preference information.

[0284] The obtained preference data is used to optimize the preference determination model, and the preference determination model is used to perform reinforcement learning training on the target model.

[0285] Optionally, the feedback information collected in this step may include feedback information corresponding to the initial answer, and may also include feedback information corresponding to the new answer.

[0286] Based on the feedback information, preference data can be constructed. The specific construction process can be referred to the aforementioned embodiment and will not be repeated here. After the preference data is constructed, the preference determination model can be optimized based on the preference data, and then the target model can be trained based on the optimized preference determination model to improve the performance of the target model.

[0287] Through the above solution, basic information and its corresponding associated information can be obtained. Preferred data can then be constructed using existing user behavior information regarding associated information on the network. This behavior information reflects the user group's value preferences for associated information. Using this constructed preference data to train a preference determination model enables the preference determination model to discern whether associated information aligns with human value preferences, improving the effectiveness of value alignment optimization and reducing the workload of manual labeling, effectively lowering costs and improving efficiency. Furthermore, by establishing a user feedback reflow data pipeline, user feedback on the target model's output can be collected and analyzed in real time, enabling the target model to align with human preferences on a wider scale, further improving the target model's performance.

[0288] Corresponding to the above method, an embodiment of the present application further provides a data construction device, including:

[0289] an acquisition module, configured to acquire an initial data set, wherein the initial data in the initial data set includes basic information, at least one associated information corresponding to the basic information, and behavior information corresponding to each associated information, wherein the behavior information is used to represent the user's behavior with respect to the associated information;

[0290] A construction module is used to construct pre-trained preference data based on the initial data, wherein the pre-trained preference data includes basic information, associated information, and corresponding preference information; wherein the preference information is used to indicate the user's preference for the associated information, and the preference information is determined by the behavior information corresponding to the associated information;

[0291] The pre-trained preference data is used to pre-train a preference determination model.

[0292] The present application also provides a model training device, including:

[0293] A pre-training module, used to obtain pre-training preference data and pre-train the preference determination model based on the pre-training preference data;

[0294] A fine-tuning module, configured to obtain fine-tuning preference data and perform fine-tuning training on the pre-trained preference determination model according to the fine-tuning preference data;

[0295] The pre-trained preference data includes the pre-trained preference data obtained by the method described in any of the aforementioned embodiments, and the fine-tuning preference data includes the fine-tuning preference data obtained by the method described in any of the aforementioned embodiments.

[0296] The present application also provides an information processing device, including:

[0297] The acquisition module is used to obtain the basic information to be processed;

[0298] An output module, configured to input the basic information into a target model, obtain corresponding associated information, and output the associated information to a user;

[0299] Wherein, the target model is obtained by training using the model training method described in any of the aforementioned embodiments.

[0300] The present application also provides an information processing device, including:

[0301] The acquisition module is used to obtain questions input by users;

[0302] An output module, configured to input the question into the target model, obtain the corresponding answer, and output the answer to the user;

[0303] Wherein, the target model is obtained by training using the model training method described in any of the aforementioned embodiments.

[0304] The specific implementation principles and processes of each device provided in the embodiments of the present application can be referred to the aforementioned method embodiments and will not be repeated here.

[0305] The device provided in the embodiment of the present application can obtain an initial data set, wherein the initial data in the initial data set includes basic information, at least one associated information corresponding to the basic information, and behavioral information corresponding to each associated information, wherein the behavioral information is used to represent the user's behavior with respect to the associated information, and pre-trained preference data is constructed based on the initial data, wherein the pre-trained preference data includes basic information, associated information and corresponding preference information, wherein the preference information is used to represent the user's preference degree for the associated information, and the preference information is determined by the behavioral information corresponding to the associated information, and the pre-trained preference data is used to pre-train a preference determination model, so that the user's behavioral information with respect to the associated information that already exists in the network can be used to construct preference data, wherein the behavioral information reflects the user group's value preference for the associated information, and the preference determination model is trained using the constructed preference data, so that the preference determination model can have the ability to distinguish whether the associated information conforms to human value preferences, thereby improving the effect of value alignment optimization, and can reduce the workload of manual labeling, effectively reduce costs, and improve efficiency.

[0306] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 10 As shown, the electronic device of this embodiment may include:

[0307] At least one processor 1001; and a memory 1002 communicatively connected to the at least one processor; wherein the memory 1002 stores instructions executable by the at least one processor 1001, and the instructions are executed by the at least one processor 1001 to cause the electronic device to perform the method as described in any of the above embodiments. Optionally, the memory 1002 can be independent or integrated with the processor 1001.

[0308] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0309] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above embodiments is implemented.

[0310] An embodiment of the present application further provides a computer program product, including a computer program, which implements the method described in any of the aforementioned embodiments when executed by a processor.

[0311] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not implemented.

[0312] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0313] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, referred to as CPU), or it can be other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0314] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0315] An exemplary storage medium is coupled to a processor, such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit. Of course, the processor and storage medium can also exist as discrete components in an electronic device or a host control device.

[0316] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0317] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0318] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0319] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data construction method, characterized in that: The method comprises: Acquire an initial data set, where the initial data in the initial data set includes basic information, at least one associated information corresponding to the basic information, and behavior information corresponding to each associated information, where the behavior information is used to represent a user's behavior with respect to the associated information; Constructing pre-trained preference data based on the initial data, the pre-trained preference data including basic information, associated information, and corresponding preference information; wherein the preference information is used to indicate the user's preference for the associated information, and the preference information is determined by the behavior information corresponding to the associated information; The pre-trained preference data is used to pre-train a preference determination model.

2. The method according to claim 1, characterized in that The basic information includes at least one of the following: demand information, title information; The associated information corresponding to the basic information includes at least one of the following: at least one reply information for replying to the demand information, and description information for describing at least one version of the title information; The behavior information is used to represent at least one of the following behaviors of the user: evaluation and editing.

3. The method according to claim 2, characterized in that Constructing pre-training preference data based on the initial data includes at least one of the following: When the basic information includes demand information, in at least one reply message corresponding to the demand information, determining preference information corresponding to each reply message according to evaluation information corresponding to each reply message; When the basic information includes title information, preference information corresponding to each version of the description information is determined according to the editing time corresponding to each version of the title information.

4. The method according to claim 1, wherein The method further comprises: Acquire multiple initial data and determine labels corresponding to each initial data, wherein the labels corresponding to the initial data include labels corresponding to basic information and / or labels corresponding to associated information in the initial data; any initial data corresponds to at least one label; Selecting part of the initial data from the plurality of initial data based on complexity and / or diversity to construct the initial data set; The complexity is used to represent the number of tags, and the diversity is used to represent the coverage of tags.

5. The method according to claim 4, characterized in that Selecting part of the initial data from the plurality of initial data based on complexity and / or diversity to construct the initial data set includes: Sort multiple initial data according to complexity to obtain an initial data queue; Process the initial data in the initial data queue in sequence until the stop condition is met; The processing of the initial data includes: if at least one label corresponding to the initial data has at least some labels that are different from labels corresponding to existing initial data in the initial data set, then adding the initial data to the initial data set.

6. The method according to claim 1, characterized in that Before constructing pre-training preference data based on the initial data, the method further includes: The initial data is filtered according to a filtering condition, where the filtering condition includes at least one of the following: the editing time and / or language style of the associated information; Whether the basic information and / or related information belongs to the preset topic.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquire a basic information set, where the basic information set includes at least one basic information; Process the basic information in the basic information set based on the target model to obtain corresponding related information; By labeling the associated information obtained based on the target model, fine-tuning preference data is obtained; The fine-tuning preference data is used to fine-tune the preference determination model, including basic information, corresponding associated information, and preference information for the associated information. The preference information is determined by the labeling results.

8. The method according to claim 7, characterized in that By annotating the associated information output by the target model, fine-tuning preference data is obtained, including: The annotation specification, basic information and associated information output by the target model are displayed through an interactive interface, so that the annotator can annotate the associated information according to the annotation specification and basic information. The obtained annotation result is used to indicate the degree to which the associated information meets the annotation specification; According to the labeling results, fine-tuning preference data is obtained; Among them, the labeling specifications include at least one of the following dimensions: compliance with basic information requirements, conformity with facts, language expression meeting requirements, and no risk information.

9. The method according to claim 8, characterized in that The preference determination model is used to perform reinforcement learning training on the target model; wherein the reinforcement learning training on the target model includes multiple rounds; When the annotation specification includes multiple dimensions, the method further includes: According to the current round, and / or, the target model and the performance corresponding to the preference determination model, the importance corresponding to each dimension of the multiple dimensions is determined, so that after obtaining the associated information according to the current target model, the corresponding preference information is determined according to the importance of each dimension and the labeling results of the associated information in each dimension.

10. The method according to claim 8, characterized in that Also includes: Input basic information into the big model to get examples of related information; The associated information examples are displayed through an interactive interface, so that the annotator can annotate the associated information according to the associated information examples.

11. The method according to claim 10, characterized in that The requirement to comply with basic information specifically includes: complying with the intent and constraints of basic information; Input the basic information into the large model to obtain examples of related information, including: Inputting basic information and output prompt information into the large model, wherein the output prompt information is used to prompt the large model to determine the intent and at least one constraint corresponding to the basic information, generate an associated information example based on the intent and the at least one constraint, and output the intent, the at least one constraint, and the associated information example; Accordingly, the example of the associated information is displayed through the interactive interface, including: The intention corresponding to the basic information, at least one constraint condition and an example of related information are displayed through an interactive interface, so that the annotator can annotate the related information according to the intention corresponding to the basic information, at least one constraint condition and the example of related information.

12. The method according to claim 7, characterized in that Also includes: Acquire multiple basic information, determine labels corresponding to each basic information, and select part of the basic information from the multiple basic information according to the labels to construct a basic information set; The labels corresponding to any basic information include labels corresponding to at least one of the following dimensions: Intent, used to indicate the task corresponding to the basic information; Knowledge, used to represent the industry involved in the basic information; Scenario, used to represent the corresponding tool usage behavior when processing the basic information; Among them, the tool usage behavior includes at least one of the following: calling API, querying documents, using plug-ins, and using multimodal modules; the multimodal module is used to process modalities other than the target modality, and the target modality is the modality that the target model can process.

13. The method according to any one of claims 1 to 6, characterized in that The fine-tuning preference data used to train the preference determination model further includes: fine-tuning preference data constructed based on feedback information; the fine-tuning preference data constructed based on feedback information is specifically obtained in the following manner: Get the basic information to be processed input by the user; Generate associated information corresponding to the basic information to be processed through the target model, and output the generated associated information to the user; Obtain user feedback on the associated information, and construct fine-tuning preference data based on the feedback information, wherein the fine-tuning preference data constructed based on the feedback information includes: the basic information to be processed, the generated associated information and the corresponding preference information, and the preference information is determined by the feedback information.

14. The method according to claim 7, wherein: The method further comprises: Obtaining a construction strategy for pre-trained preference data and / or fine-tuned preference data input by a customer, wherein the construction strategy includes at least one of the following: Screening conditions, used to filter initial data or basic information; The sampling strategy is used to obtain the initial data set or basic information set by sampling according to the labels of the initial data or basic information.

15. A model training method, characterized in that: include: Obtaining pre-trained preference data, and pre-training a preference determination model based on the pre-trained preference data; Acquiring fine-tuning preference data, and performing fine-tuning training on the pre-trained preference determination model according to the fine-tuning preference data; The pre-trained preference data includes the pre-trained preference data obtained by the method according to any one of claims 1 to 6, and the fine-tuning preference data includes the fine-tuning preference data obtained by the method according to any one of claims 7 to 14.

16. The method according to claim 15, characterized in that Also includes: Determine the model based on the preferences after fine-tuning training and perform reinforcement learning training on the target model; The process of obtaining association information based on the target model, constructing fine-tuning preference data based on the association information, fine-tuning the preference determination model based on the fine-tuning preference data, and performing reinforcement learning training on the target model based on the preference determination model is repeated until the preset requirements are met.

17. The method according to claim 16, characterized in that Determine the model based on the preferences after fine-tuning training and perform reinforcement learning training on the target model, including: Freezing the parameters of the target model and training the evaluation model for a preset number of steps; wherein the evaluation model is used to determine the expected reward during the reinforcement learning process, and during the training process for the preset number of steps, updating the parameters of the evaluation model based on the expected reward and the actual reward, wherein the actual reward is determined by the reward obtained by the preference determination model, and the reward is the preference information generated by the preference determination model based on the output of the target model; After a preset number of steps of training, the parameters of the target model are unfrozen so that in the subsequent number of steps of training, the parameters of the target model are updated based on the evaluation model and the preference determination model.

18. The method according to claim 16, characterized in that Determine the model based on the preferences after fine-tuning training and perform reinforcement learning training on the target model, including: Determine the model based on the preferences after fine-tuning training and determine the corresponding loss for reinforcement learning; Obtain unsupervised training data corresponding to the pre-training task and SFT data corresponding to the supervised fine-tuning SFT task; Determining a pre-training loss of a target model on a pre-training task and an SFT loss on an SFT task based on the unsupervised training data and the SFT data; A mixed loss is determined based on the loss corresponding to the reinforcement learning, the pre-training loss, and the SFT loss, and the parameters of the target model are updated based on the mixed loss.

19. An information processing method, characterized in that: include: Get the basic information to be processed; Inputting the basic information into the target model to obtain corresponding associated information, and outputting the associated information to the user; Wherein, the target model is obtained by training through the method described in any one of claims 16-18.

20. An information processing method, characterized in that: include: Questions to get user input; Input the question into the target model, obtain the corresponding answer, and output the answer to the user; Wherein, the target model is obtained by training through the method described in any one of claims 16-18.

21. The method according to claim 20, characterized in that The method further comprises: Obtaining user feedback on the answer; If the feedback information indicates that the answer does not meet the user's needs, regenerating the answer based on the feedback information and the question, and outputting the regenerated answer to the user; Based on the feedback information, corresponding preference information is generated to obtain preference data including questions, questions and answers, and preference information. The obtained preference data is used to optimize the preference determination model, and the preference determination model is used to perform reinforcement learning training on the target model.

22. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method according to any one of claims 1 to 21.

23. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 21 is implemented.

24. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 21 is implemented.

Images