A reinforcement learning alignment model training method and system based on AI feedback

Abstract

The present invention belongs to the field of artificial intelligence technology, and provides a reinforcement learning alignment model training method and system based on AI feedback. The method includes: collecting text data related to the direction of the medical field and performing preliminary processing; determining a preference data set, extracting features related to user behavior and text context, further forming aggregate features, generating aggregate feature vectors, forming a model generation data set, screening to obtain a target data set, and further dividing it into a training set, a validation set, and a test set; using a large language model, combining COT and RAG methods, using context learning to generate preference labels, establishing a text classification model, obtaining a target pre-training model, constructing a strategy model based on AI feedback, performing reinforcement learning on the strategy model, and using the RM model as a response to allocate rewards to obtain a target domain model; receiving a pending user query, and obtaining a matching recommendation list. The present invention can accurately match a recommendation list for a user.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a reinforcement learning alignment model training method and system based on AI feedback. Background Art

[0002] Existing multi-round dialogue methods are mainly based on large language model technology (LLM). LLM is an important breakthrough in the current field of artificial intelligence. It is based on deep learning technology and can achieve efficient understanding and generation of natural language by training massive text data. Its structure usually adopts Transformer or its variants, and uses the self-attention mechanism to capture contextual information to understand the meaning of language. When building a multi-round dialogue system, first collect text data related to the dialogue task through data collection and preprocessing, and perform necessary cleaning and annotation. Then, use the pre-trained large language model for fine-tuning training to adapt it to the dialogue task in a specific field. In the dialogue generation stage, the system can receive user input, use the trained model to generate appropriate responses, and interact with the user in multiple rounds. However, since the large language model generates segments based on probability statistics, it is prone to hallucinations, which may lead to inaccurate or irrelevant answers. At the same time, due to the high cost of training and the requirement for privacy protection, large language models are limited in the selection and scale of training data. When facing specific fields or highly specialized problems, large language models may lack the necessary expertise.

[0003] In addition, Retrieval-augmented Generation (RAG) is an advanced artificial intelligence technology that combines the advantages of information retrieval and deep learning models to address the challenges of large language models in knowledge limitations and "hallucination" problems. The main structure of RAG includes three key parts: retrieval, utilization, and generation. However, retrieval-augmented generation (RAG) also faces many challenges in practical applications. The quality of user queries varies, making it difficult to accurately parse user intent and retrieve appropriate information; while reducing interactions with LLM (Large Language Model), it is difficult to effectively align user intent with high-quality structured knowledge; the retrieved knowledge tends to be overly correlated and lacks diversity, so it is difficult to query all information related to the question.

[0004] In addition, the existing technology has the following problems: collecting high-quality manually labeled data sets takes a long time and is costly; data processing efficiency and scalability need to be improved, etc.

[0005] Therefore, it is necessary to provide a reinforcement learning alignment model training method and system based on AI feedback to solve the above problems. Summary of the invention

[0006] The present invention aims to provide a reinforcement learning alignment model training method and system based on AI feedback to solve the technical problems in the prior art that it takes a long time and is costly to collect high-quality manually labeled data sets, and LLM may have inaccurate or incomplete feedback due to the knowledge limitations of the AI ​​model. The technical problems to be solved by the present invention are achieved through the following technical solutions.

[0007] The first aspect of the present invention proposes a reinforcement learning alignment model training method based on AI feedback, the method comprising: collecting text data related to the medical field and performing preliminary processing, the text data comprising collecting patient satisfaction questionnaires from hospitals, online reviews, patient experience sharing on social media, and notes of doctors and nurses; the preliminary processing comprises cleaning, word segmentation, and text classification; determining a preference data set, extracting features related to user behavior and text context, and further forming aggregate features to generate an aggregate feature vector; based on the generated aggregate feature vector, forming a model generation data set, screening the model generation data to obtain a target data set, and further dividing it into a training set, a validation set, and a test set; using a large language model, combining COT and RA G method, and generates preference labels by context learning, specifically including: using two inference processes to obtain preference distribution; establishing a text classification model, using a training set containing preference labels to train the text classification model to obtain a target pre-trained model; on the basis of the target pre-trained model, building a strategy model based on AI feedback, performing reinforcement learning on the strategy model, and using the RM model as a response to distribute rewards, including training the reward model by minimizing the loss; fine-tuning the strategy model using labeled data to obtain a target domain model; receiving a user query to be processed, inputting the target domain model, and obtaining a recommendation list matching the user query to be processed, wherein the recommendation list contains the current user's interest, preference information, and need information for a specified topic.

[0008] The second aspect of the present invention proposes a reinforcement learning alignment model training system based on AI feedback, which adopts the reinforcement learning alignment model training method based on AI feedback, including: a data collection and processing module, which is used to collect text data related to the medical field and perform preliminary processing, wherein the text data includes patient satisfaction questionnaires collected from hospitals, online reviews, patient experience sharing on social media, and notes of doctors and nurses; the preliminary processing includes cleaning, word segmentation, and text classification; a determination processing module, which is used to determine a preference data set, extract features related to user behavior and text context, and further form aggregate features to generate an aggregate feature vector; a division processing module, which forms a model generation data set based on the generated aggregate feature vector, screens the model generation data to obtain a target data set, and further divides it into a training set, a validation set, and a test set; a label generation module, A large language model is adopted, and RAG and CoT are combined to generate preference labels by contextual learning, which specifically includes: two inference processes are used to obtain preference distribution; a model building module is used to build a text classification model, and a training set containing preference labels is used to train the text classification model to obtain a target pre-trained model; a reinforcement learning module is used to build a policy model based on AI feedback on the basis of the target pre-trained model, and reinforcement learning is performed on the policy model, and the RM model is used as a response to distribute rewards, including training the reward model by minimizing the loss; a fine-tuning processing module is used to fine-tune the policy model using labeled data to obtain a target domain model; a prediction module is used to receive a user query to be processed, and the target domain model is input to obtain a recommendation list that matches the user query to be processed, and the recommendation list contains the current user's interest, preference information, and need information for the specified topic.

[0009] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the reinforcement learning alignment model training method described in the first aspect of the present invention.

[0010] The fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the reinforcement learning alignment model training method described in the first aspect of the present invention is implemented.

[0011] The embodiments of the present invention include the following advantages:

[0012] Compared with the prior art, the present invention adopts a large language model, and combines RAG (Retrieval Augmented Generation) with CoT (Chain-of-Thought), and generates preference labels by contextual learning, specifically adopting two inference processes to obtain the preference distribution; establishes a text classification model, adopts a training set containing preference labels, trains the text classification model, obtains a target pre-training model, and further constructs a strategy model based on AI feedback on the basis of the target pre-training model, performs reinforcement learning on the strategy model, and uses the RM model as a response to allocate rewards, and uses labeled data to fine-tune the strategy model to obtain an optimized target domain model, which is used to provide a matching recommendation list for the user query to be processed, and can achieve a more accurate prediction model, and can also solve the problems in the prior art that it takes a long time and is costly to collect high-quality manually labeled data sets, and the feedback of LLM is inaccurate or incomplete due to the knowledge limitations of the AI ​​model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flowchart of an example of a reinforcement learning alignment model training method based on AI feedback of the present invention;

[0014] Figure 2 It is a partial principle schematic diagram of the reinforcement learning alignment model training method based on AI feedback of the present invention;

[0015] Figure 3 It is a structural block diagram of the reinforcement learning alignment model training system based on AI feedback of the present invention;

[0016] Figure 4 is a schematic structural diagram of an electronic device embodiment according to the present invention;

[0017] Figure 5 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0020] In view of the above problems, the present invention proposes a reinforcement learning alignment model training method and system based on AI feedback. The method uses LLM (Large Language Model) combined with RAG (Retrieval Augmented Generation) and CoT (Chain-of-Thought), generates preference labels in a contextual learning manner, retrieves relevant information when generating feedback, and can ensure that the feedback content is based on a wider knowledge base and context, thereby improving the quality and pertinence of the feedback. Based on COT, the transparency of the AI ​​feedback generation process is increased by constructing and displaying a decision-making reasoning prompt template (including a reasoning chain). By combining RAG and COT, the reinforcement learning agent can be better guided to follow long-term goals and strategies. The AI ​​feedback specifically includes high-quality (i.e., first quality) and low-quality (i.e., second quality) labels for candidate answer texts, thereby providing valuable reward signals for subsequent reinforcement learning models. These reward signals will help the reinforcement learning model learn to generate high-quality content that is more in line with user preferences. Ultimately, users will obtain more accurate and useful information when querying academic materials.

[0021] Example 1

[0022] Refer to the following Figures 1 to 3 , the contents of the present invention will be described in detail.

[0023] Figure 1 This is a flowchart of an example of the AI ​​feedback-based reinforcement learning alignment model training method of the present invention. Figure 2 It is a schematic diagram of an application example of the reinforcement learning alignment model training method based on AI feedback of the present invention.

[0024] like Figure 1 As shown, in step S101, text data related to the medical field is collected and preliminarily processed, and the text data includes collecting patient satisfaction questionnaires from hospitals, online reviews, and patient experience sharing on social media; the preliminarily processing includes cleaning, word segmentation, and text classification.

[0025] In a specific implementation, information that best reflects user satisfaction and preference patterns is selected from a large amount of text data such as patient feedback, medical records, and service evaluations to optimize medical service processes and service quality. Medical-related text data is collected from the Internet, such as public, latest scientific research papers, journal articles, blogs, and news reports, as well as patient satisfaction questionnaires from hospitals, online reviews, patient experience sharing on social media, and public notes from doctors and nurses. The collected texts are preliminarily processed, including cleaning, word segmentation, and text classification. After text classification, it can be ensured that each text topic or text type has at least two related texts.

[0026] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0027] Next, in step S102, a preference data set is determined, features related to user behavior and text context are extracted, and aggregate features are further formed to generate an aggregate feature vector.

[0028] After preliminary processing of text data, a contextual learning method is used to generate a high-quality preference dataset. Specifically, natural language processing techniques are used to extract text features, such as bag-of-words model, TF-IDF, word embedding (Word2Vec, GloVe, etc.), and features related to user behavior and text context, such as user historical behavior sequence, timestamp, location information, etc. The extracted text features are combined with context features to further form aggregate features, which are then converted into corresponding vectors, thus forming a richer feature vector.

[0029] Specifically, for user historical behavior sequences, user interaction records can be merged by weighted averaging. Weights can be based on recency (recent behaviors have higher weights), importance (some behaviors are more important than others), or frequency (events that are higher than a specified frequency and occur frequently may be more representative). For example, using word embeddings, each historical behavior can be represented as a vector, and then averaged by a weighted time decay factor; considering the importance of time, newer behaviors may be more relevant than older behaviors, so a time decay function, such as exponential decay, is applied to give recent behaviors a higher weight (for example, a weight that is one or more times higher than the previous behavior); for recurring items or behaviors, their frequency can be used as an additional feature. This helps the model understand which items or behaviors are particularly liked by users; if location information is available, it can be converted into a vector representation, obtained by pre-trained geographic coordinate embedding or using a neural network. The above-mentioned vectors can be concatenated with the user behavior vector or fused through an attention mechanism to obtain a fused user historical behavior sequence. The specific calculation process of the user historical behavior sequence will be described below.

[0030] Assume that there is a user historical behavior sequence, where each behavior is represented as a vector of length d. For a user u, there are n historical behavior vectors v1, v2, …, v n The weighted average vector v can be calculated using the time decay function ω(t) u :

[0031] ;

[0032] where t i is the timestamp of the ith action, ω(t i ) is the decay function, which can be defined as:

[0033] ;

[0034] Among them, ω(t) represents the decay function of user behavior in the time dimension; t now is the current time, α is the parameter that controls the decay rate, 0<α<1, as time t gets closer to t now , t now The smaller the value of -t, α>0, resulting in a smaller negative number in the exponential part (or closer to 0), so the value of ω(t) is larger. This means that the user's most recent behavior will be given a higher weight. The parameter α that controls the decay rate is specifically based on the user behavior sequence formed in a specified historical time period (for example, a historical time period of 3 months, 4 months, half a year, one year, etc. calculated from the current time), for example, a multidimensional vector formed by the number of times keywords are entered in the search records at multiple time points, and a user type score (user classification based on information such as age and income, such as using an existing machine model to calculate a category score) to obtain an evaluation value by weighted calculation, and based on the calculated evaluation value, determine that α is within the following data ranges: 0-0.3, 0.3-0.8, 0.8-1.

[0035] Next, in step S103, a model generation data set is formed based on the generated aggregated feature vector, the model generation data set is screened to obtain a target data set, and the target data set is further divided into a training set, a validation set, and a test set.

[0036] In a specific implementation, based on the generated aggregate feature vector, for example, the aggregate feature vector and the aggregate features are input into a large language model, and the large language model outputs a response text.

[0037] Preferably, corresponding prompt words are used. For example, before performing preference marking, several groups of examples are given to mark the text to be marked as first quality answers and second quality answers, where the quality of the first quality answer is higher than the quality of the second quality answer; wherein the first quality answer and the second quality answer are used to distinguish the quality of the text and the degree of match with the user's preference.

[0038] Specifically, the first quality answers have higher text quality, clear and complete content, and a higher degree of match in terms of the relevance of the content topic compared to the second quality answers.

[0039] It should be noted that the quality of text and the degree of match between user preferences are two key dimensions for evaluating whether text content can meet specific user needs. The relationship between the two is that even if the text itself is of high quality, if it does not meet the user's preferences or needs, then its value from the user experience perspective will be greatly reduced, but if the text is very consistent with the user's preferences, but the quality of the text itself is not high, even if it meets the user's preferences, the user will still be dissatisfied because the information is inaccurate, incomplete or difficult to understand.

[0040] Furthermore, the model generated data is screened to obtain a target data set, and further divided into a training set, a validation set, and a test set.

[0041] Data filtering is performed to ensure high quality and the target dataset is divided into training, validation, and test sets by ratio.

[0042] Additionally, to enable rapid experimental iterations when evaluating AI tagging techniques, the training splits of each preference dataset were randomly downsampled. Specifically, for summaries of text segments, an additional filter was applied to include only summaries of preferences obtained by human annotators with a confidence level above a specified value. After downsampling and filtering, AI labeler alignment metrics were calculated on these downsampled datasets. By calculating these alignment metrics on downsampled datasets, the performance of the AI ​​labeler can be quickly evaluated, and necessary adjustments can be made based on the results to improve the accuracy and efficiency of the model. This iterative process helps to continuously optimize the model to more accurately understand and reflect user preferences.

[0043] It should be noted that in this example, AI labels refer to the first quality and second quality text labels obtained using a large language model (LLM); the AI ​​labeler alignment metric refers to the alignment metric between the first quality text labels and the second quality text labels generated by the large language model (LLM model), which is an indicator used to evaluate the consistency or similarity between AI labels and manual annotation results. The above is only explained as an optional example and cannot be understood as a limitation of the present invention.

[0044] Next, in step S104, a large language model is used, and preference labels are generated by using context learning, specifically including: using two inference processes to obtain preference distribution.

[0045] Specifically, we use LLM (Large Language Model), combined with RAG and CoT, and adopt contextual learning to generate preference labels.

[0046] It should be noted that LLM is a pre-trained or instruction-tuned language model designed to understand and generate human language. It is trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc., but it is not used for specific downstream tasks.

[0047] In a preferred embodiment, before performing preference marking, several groups of examples are given to mark the text to be marked as first-quality answers and second-quality answers to guide LLM generation. The quality of the first-quality answer is higher than that of the second-quality answer. Specifically, the answer text data with higher fluency, accuracy, and relevance to the question is the first-quality answer. In the early stage, each sample and text answer data pair can be mapped to a point in the vector space for subsequent retrieval.

[0048] Next, a large language model is used to perform preference marking on the text to be marked. Specifically, the examples mentioned above and the text to be marked are input into the large language model to obtain a preference marking result, which includes a candidate answer pair, specifically a first candidate answer and a second candidate answer.

[0049] Optionally, before preference tagging is performed, additional information related to user behavior and user queries is retrieved from external knowledge sources using RAG techniques.

[0050] According to the scenario information, a reasoning prompt template is customized, and the reasoning prompt template is a CoT template. The user question and the reasoning prompt template embedded in the user question are used as model inputs, input into the large language model, and the output result is obtained.

[0051] Two inferences are performed for each candidate answer pair. For the second inference, the order in which the first candidate answer and the second candidate answer are input into the large language model is reversed, and the results of the two inference processes are averaged to obtain the final preference distribution.

[0052] It should be noted that in this example, the candidate answer is the medical text collected above. The "first candidate answer" mentioned here is a text with higher quality and more in line with user needs and preferences, corresponding to the first quality answer; preference distribution is a probability distribution. Through two inferences with different input orders, LLM will output the probability value of which answer the model thinks is better or more popular. The results of the two inferences are averaged to obtain a comprehensive preference score. This comprehensive preference score reflects the model's estimate of the preference for the two answers without considering the input order.

[0053] After the two-step reasoning procedure, a hybrid strategy is implemented, that is, while the model generates a chain of thoughts, it can also retrieve related knowledge or previous cases as auxiliary information to enhance the accuracy and coverage of reasoning. A prompt template is formulated to guide the model's step-by-step reasoning, and the CoT prompt is embedded in the input provided to the LLM. This means that when requesting the LLM to generate an answer, not only the original question is provided, but also a guiding CoT framework or example is attached to guide the model to organize its output according to this framework or example. After giving the prompt to the LLM, the softmax is calculated to obtain the preference distribution.

[0054] In one specific implementation, a large language model is used to understand the meaning in user document titles, abstracts, and search queries. Next, RAG technology is used to retrieve additional information related to user behavior and queries from external knowledge sources (such as academic databases, online libraries, etc.). In this scenario, for example, RAG is used to retrieve relevant information about two candidate articles, and then a large language model is used for CoT reasoning, combining user behavior data (such as user search records, article download records, etc.) with the information of candidate articles to build a structured reasoning process, in which each step is based on the output of the previous step. For example, the following reasoning prompt template is constructed:

[0055] prompt = "Based on user behavior data {user_behavior_data} and candidate article information {article1_info}, {article2_info}, infer the candidate articles that the user is most interested in.".

[0056] According to the above process, texts that are more in line with the user's interests can be found to generate a specific preference tag, such as articles on preferred physics, articles on preferred heart index detection, and so on.

[0057] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0058] Next, in step S105, a text classification model is established, and a training set containing preference labels is used to train the text classification model to obtain a target pre-trained model.

[0059] A text classification model is established by using the Robbins-Monro algorithm (an iterative algorithm for solving nonlinear equations), and a training set containing preference labels is used to train the text classification model to obtain a target pre-trained model.

[0060] Specifically, after the preference labels are annotated using a large language model, the process of training the model using a training set containing the preference labels will generate soft labels (e.g. [0.6, 0.4]). To avoid the generation of soft labels, the cross entropy loss is applied to the classification function softmax of the reward scores generated by the RM model (Reward Model). The classification function softmax converts the RM score into a probability distribution.

[0061] Training RM on AI-labeled datasets can be viewed as a form of model distillation.

[0062] RM has fewer parameters and lower computational complexity than the LLM that generates labels. By training on labeled datasets, RM can learn to mimic the behavior of complex models but operate in a more efficient way.

[0063] In an alternative embodiment, LLM feedback is used directly as a reward signal in RL, which enables bypassing the intermediate stage of training a RM that approximates the preferences of the LLM. The LLM is prompted to rate the quality of the text and whether it matches the user's preferences with a score between 1 and 10. The prompt contains high-level details about the input structure (i.e., query text structure) and dimensions related to user preferences (e.g., authenticity, consistency). Then, the likelihood of each score token between 1 and 10 is calculated, the likelihood is normalized to a probability distribution, and the weighted score is calculated as ,Then, the scores are normalized again to the range [-1, 1].

[0064] The pairwise accuracy of RM measures the accuracy of the trained reward model with respect to a set of user class preferences. Given an input context and a pair of candidate responses, according to the user label, if RM scores the preferred candidate higher than the non-preferred candidate, the value is 1. Otherwise, the value is 0. The calculated values ​​are averaged over multiple instances to obtain the pairwise accuracy of RM.

[0065] The score is directly calculated using the AI ​​annotator as a reward for the RL. The RM is fine-tuned on the entire training split of the corresponding preference dataset, where the labels are the AI ​​preferences for the AI ​​feedback RM and the original human preference labels in the dataset for the human feedback RM. The AI ​​feedback RM is initialized from the SFT model and the human feedback RM is initialized from the open source LLM. Here, the use of LLM is an attempt to solve the problem of using the SFT model to initialize the human feedback RM, which leads to low accuracy of the user preference set.

[0066] The reward model training method is as follows: given an input x, a pair of responses (y1, y2) ~π is sampled from one or more models, where π is usually an SFT model and the two responses y1 and y2 represent two possible output options. The input and response are generated by the LLM model to score which response is better according to some criteria. These annotations form the question and candidate answer text, that is, the triple D={(x,y w ,y l )}, where x is the input, y is w and l The responses selected as better are

[0067] and inferior responses. The reward model r is trained by minimizing the following loss ϕ :

[0068] ;

[0069] in, represents the negative log probability of the reward difference between the preferred response and the non-preferred response corresponding to the user query; is the sigmoid function, used to classify responses of different qualities;

[0070] x represents the user's input question; y w Indicates the first preferred response corresponding to the user's question; y l Indicates a second preferred response corresponding to the user's question; It means to extract samples (x, y) from triples D. w , y l ), the function The expected value of D is the user question, i.e., the question and the candidate answer text, i.e., the triple D={(x,y w ,y l )}, where x is the input, y is the w and l are selected as the first preferred response and the second preferred response respectively.

[0071] The goal of this formula is to minimize the negative log probability of the difference in reward between those preferred responses and those non-preferred responses. If the reward model correctly identifies y w is a better response, then will be a positive number, resulting in The value of is close to 1. This way, the negative logarithmic part of the loss function tends to 0, minimizing the loss. If the model fails to correctly identify the preferred response, then May be a negative number, resulting in Values ​​of are close to 0.5 or smaller, which in turn lead to large negative logarithmic values, thus increasing the loss.

[0072] Taking the example of providing customized academic and research materials to users, a supervised fine-tuning (SFT) model is used to generate different article summaries for user queries. Based on the LLM scoring results, a triple is created and the above loss function is used to train the reward model r ϕ , through such a training process, the reward model r ϕ It has learned how to distinguish high-quality summaries from low-quality summaries, which can provide valuable reward signals for subsequent reinforcement learning models. These reward signals will help the reinforcement learning model learn to generate high-quality summaries that are more in line with user preferences. Ultimately, users will get more accurate and useful information when searching for academic materials.

[0073] Next, in step S106, based on the target pre-trained model, a policy model based on AI feedback is constructed, reinforcement learning is performed on the policy model, and rewards are distributed using the RM model as a response, including training the reward model by minimizing the loss.

[0074] Specifically, based on the target pre-training model, a strategy model based on AI feedback is constructed, namely, a reinforcement learning alignment strategy model.

[0075] The policy model is trained through reinforcement learning, and RM is used to assign rewards to model responses, namely the reward model.

[0076] For reinforcement learning, for example, the SFT model for each task (e.g., text classification task in the medical field, etc.) is used as the initial policy, and a modified version of REINFORCE suitable for the language modeling field is used to train the policy. Both the policy and value models are initialized from the SFT model.

[0077] The reward model is trained by minimizing the following loss:

[0078] ;

[0079] in, represents the negative log probability of the reward difference between the preferred response and the non-preferred response corresponding to the user query; It is a sigmoid function, which is used to classify responses of different qualities. Specifically, it can convert the reward difference between two responses into a probability value between (0,1);

[0080] in, represents the negative log probability of the reward difference between the preferred response and the non-preferred response corresponding to the user query; It is a sigmoid function, which is used to classify responses of different qualities. Specifically, it can convert the reward difference between two responses into a probability value between (0,1);

[0081] x represents the user's input question; y w Indicates the first preferred response corresponding to the user's question; y l Indicates a second preferred response corresponding to the user's question; It means to extract samples (x, y) from triples D. w , y l ), the function The expected value of D is the user question, i.e., the question and the candidate answer text, i.e., the triple D={(x,y w ,y l )}, where x is the input, y is the w and l are selected as the first preferred response and the second preferred response respectively;

[0082] when is a positive number, The value of is close to 1, indicating that the reward model correctly identifies y w is the better response; when the reward model fails to correctly identify the preferred response, then is a negative number, The value is 0.5 or less.

[0083] It should be noted that the reason why the sigmoid function is used here is that it can convert the reward difference between two responses of different quality into a probability value between (0,1). > 0, the output of the sigmoid function is close to 1, indicating that the model correctly judges y w Better than y l On the contrary, QUOTE <0, the output of the sigmoid function is close to 0, indicating that the model incorrectly judged y l Better than y wAfter using the sigmoid function, the logarithm is taken to facilitate the calculation of the gradient and make the loss function smoother, which is conducive to the optimization during the training process. During the training process, the loss function is minimized, even if When it is positive, the output of the sigmoid function is close to 1, so the output of the logarithmic function will be close to 0 (small loss). On the contrary, if the output of the sigmoid function is close to 0, the output of the logarithmic function will be a large negative number (large loss). In general, the role of the sigmoid function here is to convert the reward difference into a probability form, and further convert it into a part of the loss function by taking the logarithm to guide the learning of the model.

[0084] The process of reinforcement training using a modified version of the REINFORCE algorithm is as follows: define an MDP M=(X,A,R,P,γ) with deterministic state transitions and a finite time step. At each step t, given the current state X t ∈X, the next action is A t ∈A, the model receives a reward R t =R(X t ,A t ), and transition to the next state X t+1 =R(X t ,A t ).

[0085] It should be noted that in the designed medical patient experience optimization system, at each time step t , X t is all the text information received so far, including patient feedback, parts of medical records, and text generated by the system itself. This can be understood as the current state of the model's understanding of the patient's experience. Action A t is the next token selected to be added to the current text sequence at time t. This is determined by the specified strategy π θ Determined by, where θ is the parameter of the model. Reward R t Depends on X t and A t , evaluated by a reward model (RM). In this scenario, RM is a pre-trained model that is specifically designed to determine whether the generated text accurately reflects patient satisfaction and preference patterns. The score given by RM is the reward value. The state transition function P(X t ,A t ) represents a transition from a state X t Take action A t Another state X reached later t+1In NLP, this means adding a new token to the current text sequence. RM is evaluated only when the language model response has been fully generated; all rewards before the final token are set to 0, while the reward corresponding to the final token is set to R T .

[0086] Starting at time step t, follow the specified policy π θ The cumulative sum of rewards obtained when is called the return. Usually, it is defined as . Where Rs represents the immediate reward obtained by the agent based on the current state Xs and the action As taken at a specific time step s, and Rs here is the value of the reward function at a specific time step s. However, since only the terminal reward is non-zero and γ=1 is set, the return can be simplified to Z T =R T .

[0087] Given a specified strategy π θ The trajectory generated by , then the policy gradient loss from REINFORCE is defined as follows:

[0088] ;in, represents the policy gradient loss value of REINFORCE, specifically the policy gradient loss for the parameter θ, which is used to guide the policy network, i.e. the parameter update of the Actor, to maximize the long-term return; t represents the time step; X t Indicates the state at time step t; A t represents the action performed at time step t; π θ Indicates a specified strategy; Z T Indicates that the specified strategy π is followed from time step t until a certain termination time T θ The cumulative sum of rewards received, i.e., the return; In state X t , following the expected return value of the policy function, where π refers to the strategy that the agent will take next. The agent here uses the policy function π to select the action of the token to be generated next, so as to form a comprehensive description of patient feedback, medical records or service evaluation; the π contains model parameters, which determine the probability of selecting different tokens in different states. For example, when describing patient satisfaction, if the token "satisfied" often appears in positive feedback in historical data, then π will learn to increase the probability of selecting "satisfied" when describing positive experiences.

[0089] also, and θ correspond to the parameters of different model components, where ψ usually represents the value function , θ represents the parameters of the policy function, that is, the policy network parameters; the value function It is approximated by a network called Critic, whose task is to evaluate the expected return of a given strategy under different states; the parameter ψ is optimized by training the Critic network to predict the state value; the given strategy π θ According to the current state X t Generate action A t The probability distribution of Update the gradient of the policy network parameter θ, specifically by minimizing it through gradient descent , so that the specified strategy π θ Perform better in the environment so that the agent learns to take actions that maximize long-term rewards.

[0090] It should be noted that the “specified strategy π θ " and "estimate action selection corresponding strategy π" below do not have the same meaning. They refer to different objects used in different algorithms.

[0091] Baseline value function Estimates the following policy π θ The profit Z t and through Parameterize. The loss during training is as follows:

[0092] ;

[0093] in, is the value function The loss is , the goal is to minimize this loss so that the value function can more accurately predict the state X when following the specified policy π t The expected return at the start; Z t It represents the sum of the cumulative rewards obtained from the current time step t to a certain termination time point T at the end of the sequence or the completion of the task, that is, starting from the current state, following the specified strategy π θ The discounted sum of all immediate rewards obtained until the end of the task, T is the termination time point of the sequence, or the time step when a certain termination condition is reached.

[0094] In addition, KL regularization is incorporated into the above policy gradient loss to optimize the regularization target, maintain the smoothness of policy updates, and prevent drastic changes in the policy.

[0095] Optionally, the training process using the A2C (Advantage Actor-Critic) algorithm is as follows: The actor part of the A2C algorithm is a network used to estimate the corresponding strategy for action selection, and the input value is the current state s t , outputs the probability of each action a, and there are as many output values ​​as there are action networks. The critic is the estimated state value function V π , denoted as v(s; ω). After training for a specified step, the actor will modify its own network parameters according to the critic's score, thereby affecting the probability of action selection behavior. In the context of the language model, the agent selects actions through the policy function π (here the agent uses the policy function π to select the action of the token to be generated next in order to form a comprehensive description of patient feedback, medical records or service evaluations), where the policy value function Q π (s t ,a t ) According to the current state s t and strategy a t The value of the strategy value function Q π (s t ,a t ) is calculated as:

[0096] ;

[0097] ;

[0098] Among them, R t is the cumulative reward value; r i (i=1,2,…,n) is the reward value generated after the strategy is executed; a t Indicates that the agent performs action a at time t, specifically selecting the next token to be added to the current text sequence at time t; s t is the state value at time t (for example, in the designed medical patient experience optimization system, s t represents all text information received so far); γ is the reduction coefficient of the reward value when calculating the cumulative reward value Rt.

[0099] The A2C algorithm introduces the idea of ​​advantage function based on the original AC algorithm and uses the following expression to represent the advantage function A π :

[0100] ;

[0101] Among them, Q π (s t,a t ) is the strategy value function; V π (s t ) is the state value function. Use t Good or bad state value function V π (s t ) Correction Q π (s t ,a t ), which is used to determine the value of the benefit generated by the current action a compared to the average benefit of all actions, which is the predicted Q π (s t ,a t ) value to add a baseline.

[0102] ;

[0103] Among them, V π (s t ) represents the strategy value function corresponding to all possible strategies in this state; Q π (s t ,a t ) and the algebraic sum of the probability of adopting this strategy. π is greater than 0, the current action has an overall advantage; if A π If it is less than or equal to 0, it is not as good as the average value produced by the overall action.

[0104] At time t, the TD target is calculated by the output value v(s;ω) of the value network for:

[0105] ;

[0106] because In order to be closer to the true value based on the calculation results of some actual reward values, the loss function L(ω) of the value network in the A2C algorithm is defined as:

[0107] ;

[0108] Update the parameter ω by gradient descent to make it closer , the parameter update process is:

[0109] ;

[0110] In the policy network of the A2C algorithm, the unbiased estimate of the policy gradient is ;

[0111] Among them, g(s t ,a t;θ) represents the unbiased estimate of the policy gradient, Q π (s t ,a t ) is the policy value function, s t represents the current state at time t, a t represents the action performed by the agent at time t; V π (s t ) represents the state value function, which represents the current state s t The quality of the policy function Q π (s t ,a t ), which is used to determine whether the benefit generated by the current action is higher than the average benefit of all actions, that is, the value of the predicted strategy value function, and add a baseline. is the policy function In the current state t Take action a t The probability distribution of the gradient of the policy network parameter θ; if the advantage function is positive, that is, taking action a t is favorable, then the policy network parameter θ will move toward increasing the action a t The probability of the direction update; if the advantage function is negative, that is, take action a t is unfavorable, then the policy network parameters θ will be directed toward reducing the action a t The probability direction update;

[0112] According to the state s at time t+1 t+1 With reward value r t For g(s t ,a t ; The expectation in θ) is approximated by Monte Carlo to obtain ,use To update the policy network parameters θ, the policy gradient method is used. To ensure stability, the advantage function is used as the weight to adjust the update direction of the policy network:

[0113] ;

[0114] Among them, θ' represents the updated policy network parameters; θ represents the policy network parameters before updating; β is the learning rate, which is used to control the amplitude of parameter update.

[0115] The final checkpoint is selected for each RL policy. First, a fixed number of candidate checkpoints are selected from the RL training that have achieved high rewards on the validation prompt. Then, an off-the-shelf LLM is prompted to judge the win rate of the summary of the RL checkpoints against the summary of the SFT policy, and finally the checkpoint with the best combination of win rate and quality judged by manual inspection is selected as the final RL policy.

[0116] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0117] Next, in step S107, the strategy model is fine-tuned using the labeled data to obtain a target domain model.

[0118] Specifically, we use the transfer learning method on the target dataset, and use the pre-trained strategy from the source task as the initial strategy for the target task. We use the pre-trained model strategy that has been successfully learned in a similar environment to guide the agent, thereby speeding up the learning process and obtaining the target domain model.

[0119] In a specific implementation, in a medical patient experience optimization system, the source task is, for example, patient interaction data from another medical scenario or related field, such as in a large hospital appointment system, where the model learns how to optimize the patient appointment process, reduce waiting time, and improve satisfaction. For example, in the source task, the model has learned a set of effective strategies through a large amount of appointment data, patient feedback, and medical records. This set of strategies can effectively predict patient preferences, optimize appointment processes, and improve patient satisfaction.

[0120] The target task is a new domain where you want to apply or optimize its behavior. For example, a system for monitoring a specific disease. Learning the model from scratch will be slow and may require more time and resources due to the small amount of data and different environments. In the target task, the challenges faced include different patient groups, more complex disease management, or specific medical service processes, which all need to be relearned and adapted.

[0121] The initial strategy refers to the strategy used when starting to learn in the target task. In the framework of transfer learning, the initial strategy usually comes from the pre-trained model in the source task. In the medical patient experience optimization system, the strategy learned from the source task can be used as the initial strategy for the target task.

[0122] Specifically, the transfer learning method is used on the target dataset, and the strategy pre-trained by the model in the source task is first used as the initial strategy for the target task. This means that when the model starts learning the target task, it does not start from scratch, but already has a starting point based on successful experience in a similar environment. In the target task, new experience is collected through interaction with the environment, and these experiences will be used to fine-tune or update its initial strategy π0. Since the initial strategy already has a certain adaptability and optimization basis, it can adapt to the new environment more quickly, reduce the exploration phase of learning, and reach or exceed the performance requirements of the target task more quickly. In this way, transfer learning can effectively utilize existing knowledge in the medical patient experience optimization system, accelerate the learning process of the model in the new environment, improve learning efficiency, reduce resource consumption, and ultimately achieve better patient experience optimization effects.

[0123] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0124] Next, in step S108, a user query to be processed is received, and a target domain model is input to obtain a recommendation list matching the user query to be processed, wherein the recommendation list includes the current user's interest, preference information, and need information on a specified topic.

[0125] In a specific implementation, for example, a user query to be processed is received, such as a user query for orthopedic examination items, the user query to be processed is input into the target domain model, and a recommendation list matching the user query to be processed is obtained, specifically including necessary orthopedic examination items, etc. In a specific implementation, for example, a user query to be processed is received, assuming that the user asks how to increase the speed of the home network, the user query to be processed is input into the target domain model, and a recommendation list matching the user query to be processed is obtained, specifically including but not limited to: methods for eliminating signal interference, router firmware update documents, etc.

[0126] In a specific implementation, for example, a user query to be processed is received, such as a query for junior high school physics learning materials, a target domain model is input, and a junior high school physics textbook, a junior high school physics knowledge point summary, a junior high school physics test paper question bank website, etc. are output.

[0127] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0128] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0129] Compared with the prior art, the present invention adopts a large language model, and combines RAG (Retrieval Augmented Generation) with CoT (Chain-of-Thought), and generates preference labels by contextual learning, specifically adopting two inference processes to obtain the preference distribution; establishes a text classification model, adopts a training set containing preference labels, trains the text classification model, obtains a target pre-training model, and further constructs a strategy model based on AI feedback on the basis of the target pre-training model, performs reinforcement learning on the strategy model, and uses the RM model as a response to allocate rewards, and uses labeled data to fine-tune the strategy model to obtain an optimized target domain model, which is used to provide a matching recommendation list for the user query to be processed, and can achieve a more accurate prediction model, and can also solve the problems in the prior art that it takes a long time and is costly to collect high-quality manually labeled data sets, and the feedback of LLM is inaccurate or incomplete due to the knowledge limitations of the AI ​​model.

[0130] Example 2

[0131] The following is a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. For details not disclosed in the system embodiment of the present invention, please refer to the method embodiment of the present invention.

[0132] Figure 3 is a structural diagram of an example of a reinforcement learning alignment model training system based on AI feedback according to the present invention.

[0133] Reference Figure 3 The second aspect of the present disclosure provides a reinforcement learning alignment model training system 300 based on AI feedback, which adopts the reinforcement learning alignment model training method based on AI feedback described in the first aspect of the present invention.

[0134] Specifically, the reinforcement learning alignment model training system 300 includes a data collection processing module 310, a determination processing module 320, a division processing module 330, a label generation module 340, a model building module 350, a reinforcement learning module 360, a fine-tuning processing module 370 and a prediction module 380.

[0135] In a specific embodiment, the data collection and processing module 310 is used to collect text data related to the medical field and perform preliminary processing. The text data includes collecting patient satisfaction questionnaires from hospitals, online reviews, and patient experience sharing on social media; the preliminary processing includes cleaning, word segmentation, and text classification. The determination processing module 320 is used to determine the preference data set, extract features related to user behavior and text context, and further form aggregate features to generate aggregate feature vectors. The division processing module 330 forms a model generation data set based on the generated aggregate feature vector, screens the model generation data to obtain a target data set, and further divides it into a training set, a validation set, and a test set. The label generation module 340 uses a large language model, combines RAG and CoT, and generates preference labels in a context learning manner, specifically including: using two inference processes to obtain preference distribution. The model building module 350 is used to establish a text classification model, using a training set containing preference labels to train the text classification model to obtain a target pre-training model. The reinforcement learning module 360 ​​builds a policy model based on AI feedback on the basis of the target pre-trained model, performs reinforcement learning on the policy model, and uses the RM model as a response to distribute rewards, including training the reward model by minimizing losses. The fine-tuning processing module 370 uses the labeled data to fine-tune the policy model to obtain the target domain model. The prediction module 380 is used to receive a pending user query, input the target domain model, and obtain a recommendation list that matches the pending user query, and the recommendation list contains the current user's interest, preference information, and need information for the specified topic.

[0136] According to an optional implementation, a large language model is used, and preference labels are generated by context learning, specifically including: performing two inference processes to obtain preference distribution.

[0137] Before preference marking, the text to be marked is marked as an example of a first quality answer and a second quality answer, where the quality of the first quality answer is higher than the quality of the second quality answer; wherein the first quality answer and the second quality answer are used to distinguish the quality of the text and the degree of matching with the user preference.

[0138] According to an optional implementation, a large language model is used to perform preference marking on the text to be marked, specifically, the aforementioned examples of the text to be marked and the text to be marked are input into the large language model to obtain a preference marking result, and the preference marking result includes a candidate answer pair, specifically a first candidate answer and a second candidate answer.

[0139] According to an optional implementation, it further includes: using RAG technology to retrieve additional information related to user behavior and user queries from an external knowledge source; customizing a reasoning prompt template based on scenario information, wherein the reasoning prompt template is a CoT template; using the user question and the reasoning prompt template embedded in the user question as model input, inputting the large language model, and obtaining an output result; performing two inferences on each pair of candidate answers, and for the second inference, reversing the order in which the first candidate answer and the second candidate answer are input into the large language model, and averaging the results of the two inference processes to obtain a final preference distribution.

[0140] According to an alternative embodiment, the RM model is trained by minimizing the following loss:

[0141] ;

[0142] in, represents the negative log probability of the reward difference between the preferred response and the non-preferred response corresponding to the user query; It is a sigmoid function, which is used to classify responses of different qualities. Specifically, it can convert the reward difference between two responses into a probability value between (0,1);

[0143] x represents the user's input question; y w Indicates the first preferred response corresponding to the user's question; y l Indicates a second preferred response corresponding to the user's question; It means to extract samples (x, y) from triples D. w , y l ), the function The expected value of D is the user question, i.e., the question and the candidate answer text, i.e., the triple D={(x,y w ,y l )}, where x is the input, y is the w and l are selected as the first preferred response and the second preferred response respectively;

[0144] when is a positive number, The value of is close to 1, indicating that the reward model correctly identifies y w is the better response; when the reward model fails to correctly identify the preferred response, then is a negative number, The value is 0.5 or less.

[0145] According to an optional implementation, the output result of the large language model is directly used as the reward signal of the strategy model, and the output result is a numerical value within [-1, 1], which represents the quality of the text and whether it meets the user's preference.

[0146] According to an optional implementation, based on the REINFORCE algorithm, reinforcement learning is performed on the policy model, specifically including:

[0147] Define an MDP M=(X,A,R,P,γ) with deterministic state transitions and finite time steps. At each step t, given the current state X t ∈X, the next action is A t ∈A, the model receives a reward R t =R(X t ,A t ), and transition to the next state X t+1 =R(X t ,A t );

[0148] Starting at time step t, follow the specified policy π θ The cumulative sum of rewards obtained when is called the return, defined as , set γ = 1, then the return is simplified to Z T =R T ; Rs represents the immediate reward obtained by the agent based on the current state Xs and the action As taken at a specific time step s. Here Rs is the value of the reward function at a specific time step s.

[0149] In the specified strategy πθ The trajectory generated by , then the policy gradient loss from REINFORCE is defined as:

[0150] ;

[0151] in, represents the policy gradient loss value of REINFORCE, specifically the policy gradient loss for the parameter θ, which is used to guide the policy network, i.e. the parameter update of the Actor, to maximize the long-term return; t represents the time step; X t A represents the state at time step t; t represents the action performed at time step t, Z t Represents the sum of the cumulative rewards obtained from time step t until a certain termination time T; In state X tHere, the agent uses the strategy function π to select the action of the token to be generated next, so as to form a comprehensive description of patient feedback, medical records or service evaluation.

[0152] and θ correspond to the parameters of different model components, where ψ usually represents the value function , θ represents the parameters of the policy function, that is, the policy network parameters; the value function It is approximated by a network called Critic, whose task is to evaluate the expected return of a given strategy under different states; the parameter ψ is optimized by training the Critic network to predict the state value; the given strategy π θ According to the current state X t Generate action A t The probability distribution of Update the gradient of the policy network parameter θ, specifically by minimizing it through gradient descent , so that the specified strategy π θ Perform better in the environment so that the agent learns to take actions that maximize long-term rewards.

[0153] Baseline value function Estimates the following policy π θ The profit Z t and through After parameterization, the loss during training is:

[0154] ;

[0155] in, is the value function The loss is , the goal is to minimize this loss so that the value function can more accurately predict the state X when following the specified policy π t Expected return at the start; Z t It represents the sum of the cumulative rewards obtained from the current time step t to a certain termination time point T at the end of the sequence or the completion of the task, that is, starting from the current state, following the specified strategy π θ The discounted sum of all immediate rewards obtained until the end of the task, T is the termination time point of the sequence, or the time step when a certain termination condition is reached.

[0156] According to an optional implementation, an A2C algorithm is used for reinforcement learning;

[0157] In the policy network of the A2C algorithm, the unbiased estimate of the policy gradient is g(s t ,a t ;θ),

[0158] ,

[0159] Among them, g(s t ,a t ;θ) represents the unbiased estimate of the policy gradient, Q π (s t ,a t ) is the policy value function, s t represents the current state at time t, a t represents the action performed by the Agent at time t; V π (s t ) represents the state value function, which represents the current state s t The quality of the policy function Q π (s t ,a t ), which is used to determine whether the benefit generated by the current action is higher than the average benefit of all actions, that is, the value of the predicted strategy value function, and add a baseline. is the policy function In the current state t Take action a t The probability distribution of the gradient of the policy network parameter θ; if the advantage function is positive, that is, taking action a t is favorable, then the policy network parameter θ will move toward increasing the action a t The probability of the direction update; if the advantage function is negative, that is, take action a t is unfavorable, then the policy network parameters θ will be directed toward reducing the action a t The probability direction update;

[0160] According to the state s at time t+1 t+1 With reward value r t For g(s t ,a t ; The expectation in θ) is approximated by Monte Carlo to obtain ,use To update the policy network parameters θ, the policy gradient method is used. To ensure stability, the advantage function is used as the weight to adjust the update direction of the policy network:

[0161] ;

[0162] Among them, θ' represents the updated policy network parameters; θ represents the policy network parameters before updating; β is the learning rate, which is used to control the amplitude of parameter update.

[0163] Compared with the prior art, the present invention adopts a large language model, and combines RAG (Retrieval Augmented Generation) with CoT (Chain-of-Thought), and generates preference labels in a contextual learning manner, specifically adopting two inference processes to obtain the preference distribution; establishes a text classification model, adopts a training set containing preference labels, trains the text classification model, obtains a target pre-training model, and further constructs a strategy model based on AI feedback on the basis of the target pre-training model, performs reinforcement learning on the strategy model, and uses the RM model as a response to allocate rewards, and uses labeled data to fine-tune the strategy model to obtain an optimized target domain model, which is used to provide a matching recommendation list for the user query to be processed, and can achieve a more accurate prediction model, and can also solve the problems in the prior art that it takes a long time and is costly to collect high-quality manually labeled data sets, and the LLM may have inaccurate or incomplete feedback due to the knowledge limitations of the AI ​​model.

[0164] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0165] like Figure 4 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0166] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0167] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0168] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0169] It should be understood that Figure 4 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0170] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 5 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0171] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0172] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0173] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0174] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.

[0175] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0176] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0177] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A reinforcement learning alignment model training method based on AI feedback, characterized in that: include: Collect text data related to the medical field and perform preliminary processing, including collecting patient satisfaction questionnaires from hospitals, online reviews, and patient experience sharing on social media; the preliminary processing includes cleaning, word segmentation, and text classification; Determine the preferred data set, extract features related to user behavior and text context, and further form aggregate features to generate an aggregate feature vector; Based on the generated aggregated feature vector, a model generation data set is formed, the model generation data set is screened to obtain a target data set, and the target data set is further divided into a training set, a validation set, and a test set; A large language model is used, combined with RAG and CoT, to generate preference labels using contextual learning, which includes: using two inference processes to obtain preference distribution; Establishing a text classification model, using a training set containing preference labels to train the text classification model to obtain a target pre-trained model; Based on the target pre-trained model, a policy model based on AI feedback is constructed, reinforcement learning is performed on the policy model, and rewards are assigned using the RM model as a response, including training the reward model by minimizing the loss; Fine-tune the policy model using labeled data to obtain the target domain model; Receive a user query to be processed, input a target domain model, and obtain a recommendation list matching the user query to be processed, wherein the recommendation list includes the current user's interest, preference information, and required information on a specified topic; wherein: The RM model is trained by minimizing the following loss: in, represents the negative logarithmic probability of the reward difference between the preferred response and the non-preferred response corresponding to the user query; σ is the sigmoid function, which is used to classify responses of different qualities. Specifically, it can convert the reward difference between two responses into a probability value between (0,1); χ represents the input user question; Indicates the first preferred response corresponding to the user's question; Indicates a second preferred response corresponding to the user's question; Represents a triple Extract samples from For all possible combinations of The expected value of Represents user questions, i.e., questions and candidate answer texts, i.e., triples A dataset where x is the input and y is w and l are selected as the first preferred response and the second preferred response respectively; when r φ (x,y w )-r φ (x,y l ) is a positive number, the value of σ is close to 1, indicating that the reward model correctly identifies y w is a better response; when the reward model fails to correctly identify the preferred response, then r φ (x,y w )-r φ (x,y l ) is a negative number, and the value of σ is 0.5 or less; Based on the REINFORCE algorithm, the strategy model is reinforced with learning, including: Define an MDP M = (X, A, R, P, γ) with deterministic state transitions and finite time steps. At each step t, given the current state X t ∈X, the next action is A t ∈A, the model receives a reward R t =R(X t ,A t ), and transition to the next state X t+1 =P(X t ,A t ); Starting at time step t, follow the specified policy π θ The cumulative sum of rewards obtained when is called the return, defined as Setting γ = 1, the return is simplified to Z T =R T ; In the specified strategy π θ The trajectory generated by Then the policy gradient loss from REINFORCE is defined as: in, represents the policy gradient loss value of REINFORCE, specifically the policy gradient loss for the parameter θ, which is used to guide the policy network, i.e. the parameter update of the Actor, to maximize the long-term return; t represents the time step; X t Indicates the state at time step t; A t represents the action performed at time step t, Z T Represents the sum of the cumulative rewards obtained from time step t until a certain termination time T; Indicates that in state X t Here, π refers to the strategy that the agent will take next. The agent here uses the strategy function π to select the action of the token to be generated next, so as to form a comprehensive description of patient feedback, medical records or service evaluation; ψ and θ correspond to the parameters of different model components, respectively, where ψ represents the value function , θ represents the parameters of the policy function, that is, the policy network parameters; the value function It is approximated by a network called Critic, whose task is to evaluate the expected return of a given strategy under different states; the parameter ψ is optimized by training the Critic network to predict the state value; the given strategy π θ According to the current state X t Generate action A t The probability distribution of Update the gradient of the policy network parameter θ, specifically by minimizing it through gradient descent So that the specified strategy π θ Perform better in the environment so that the agent learns to take actions that maximize long-term rewards; Baseline value function Estimates the following policy π θ The profit Z T , and is parameterized by ψ, and the loss during training is: in, is the value function The goal is to minimize this loss so that the value function can more accurately predict the state X when following the specified policy π. t Expected return at the start; Z T It represents the sum of the cumulative rewards obtained from the current time step t to a certain termination time point T at the end of the sequence or the completion of the task, that is, starting from the current state, following the specified strategy π θ The discounted sum of all immediate rewards obtained until the end of the task, T is the termination time point of the sequence, or the time step when a certain termination condition is reached; or, Use A2C algorithm for reinforcement learning; In the policy network of the A2C algorithm, the unbiased estimate of the policy gradient is g(s t ,a t ; θ), Among them, g(s t ,a t ; θ) represents the unbiased estimate of the policy gradient, Q π (s t ,a t ) is the policy value function, s t represents the current state at time t, a t represents the action performed by the Agent at time t; V π (s t ) represents the state value function, which represents the current state s t The quality of the policy function Q π (s t ,a t ), which is used to determine whether the benefit generated by the current action is higher than the average benefit of all actions, that is, the value of the predicted strategy value function, and add a baseline. is the policy function π(a t |s t ;θ) in the current state s t Take action a t The probability distribution of the gradient of the policy network parameter θ; if the advantage function is positive, that is, taking action a t is favorable, then the policy network parameter θ will move toward increasing the action a t The probability of the direction update; if the advantage function is negative, that is, take action a t is unfavorable, then the policy network parameters θ will be directed toward reducing the action a t The direction of the probability is updated; According to the state s at time t+1 t+1 With reward value r t For g(s t ,a t ; The expectation in θ) is approximated by Monte Carlo to obtain use To update the policy network parameters θ, the policy gradient method is used. To ensure stability, the advantage function is used as the weight to adjust the update direction of the policy network: Among them, θ' represents the updated policy network parameters; θ represents the policy network parameters before updating; β is the learning rate, which is used to control the amplitude of parameter update.

2. The reinforcement learning alignment model training method based on AI feedback according to claim 1 is characterized in that: The method of using a large language model and context learning to generate preference labels specifically includes: using two inference processes to obtain preference distribution, including: Before preference marking, the text to be marked is marked as an example of a first quality answer and a second quality answer, wherein the text answer data pair includes a first quality answer and a second quality answer, and the quality of the first quality answer is higher than the quality of the second quality answer; wherein the first quality answer and the second quality answer are used to distinguish the quality of the text and the degree of matching with the user preference.

3. The reinforcement learning alignment model training method based on AI feedback according to claim 2 is characterized in that: include: A large language model is used to perform preference marking on the text to be marked, specifically, the examples mentioned above and the text to be marked are input into the large language model to obtain a preference marking result, and the preference marking result includes a candidate answer pair, specifically a first candidate answer and a second candidate answer.

4. The reinforcement learning alignment model training method based on AI feedback according to claim 2 is characterized in that: Further including: Retrieve additional information related to user behavior and user queries from external knowledge sources using RAG techniques; Customize a reasoning prompt template according to the scene information, where the reasoning prompt template is a CoT template; The user question and the inference prompt template embedded in the user question are used as model input, input into the large language model, and the output result is obtained; Two inferences are performed for each candidate answer pair. For the second inference, the order in which the first candidate answer and the second candidate answer are input into the large language model is reversed, and the results of the two inference processes are averaged to obtain the final preference distribution.

5. The reinforcement learning alignment model training method based on AI feedback according to claim 1 is characterized in that: include: The output result of the large language model is directly used as the reward signal of the strategy model. The output result is a value in [-1, 1], which represents the quality of the text and whether it meets the user's preference.

6. A reinforcement learning alignment model training system based on AI feedback, characterized in that: The reinforcement learning alignment model training method based on AI feedback according to claim 1 comprises: A data collection and processing module is used to collect text data related to the medical field and perform preliminary processing. The text data includes collecting patient satisfaction questionnaires from hospitals, online reviews, patient experience sharing on social media, and notes from doctors and nurses. The preliminary processing includes cleaning, word segmentation, and text classification. A determination processing module, used to determine a preference data set, extract features related to user behavior and text context, and further form aggregate features to generate an aggregate feature vector; A partitioning processing module, which forms a model generation data set based on the generated aggregated feature vector, screens the model generation data to obtain a target data set, and further divides the data into a training set, a validation set, and a test set; The label generation module uses a large language model and combines RAG and CoT to generate preference labels using contextual learning, which specifically includes: using two inference processes to obtain preference distribution; A model building module is used to build a text classification model, and a training set containing preference labels is used to train the text classification model to obtain a target pre-trained model; The reinforcement learning module builds a policy model based on AI feedback based on the target pre-trained model, performs reinforcement learning on the policy model, and uses the RM model as a response to distribute rewards, including training the reward model by minimizing the loss; The fine-tuning processing module uses the labeled data to fine-tune the policy model to obtain the target domain model; the prediction module receives the user query to be processed, inputs the target domain model, and obtains a recommendation list that matches the user query to be processed, wherein the recommendation list contains the current user's interest, preference information, and required information for the specified topic; wherein, The RM model is trained by minimizing the following loss: in, represents the negative logarithmic probability of the reward difference between the preferred response and the non-preferred response corresponding to the user query; σ is the sigmoid function, which is used to classify responses of different qualities. Specifically, it can convert the reward difference between two responses into a probability value between (0,1); χ represents the input user question; Indicates the first preferred response corresponding to the user's question; Indicates a second preferred response corresponding to the user's question; Represents a triple Extract samples from For all possible combinations of The expected value of Represents user questions, i.e., questions and candidate answer texts, i.e., triples A dataset where x is the input and y is w and are selected as the first preferred response and the second preferred response respectively; when When σ is a positive number, the value of σ is close to 1, indicating that the reward model correctly identifies y w is the better response; when the reward model fails to correctly identify the preferred response, then is a negative number, and the value of σ is 0.5 or less; Based on the REINFORCE algorithm, the strategy model is reinforced with learning, including: Define an MDP M = (X, A, R, P, γ) with deterministic state transitions and finite time steps. At each step t, given the current state X t ∈X, the next action is A t ∈A, the model receives a reward R t =R(X t ,A t ), and transition to the next state X t+1 =P(X t ,A t ); Starting at time step t, follow the specified policy π θ The cumulative sum of rewards obtained when is called the return, defined as Setting γ = 1, the return is simplified to Z T =R T ; In the specified strategy π θ The trajectory generated by Then the policy gradient loss from REINFORCE is defined as: in, represents the policy gradient loss value of REINFORCE, specifically the policy gradient loss for the parameter θ, which is used to guide the policy network, i.e. the parameter update of the Actor, to maximize the long-term return; t represents the time step; X t Indicates the state at time step t; A t represents the action performed at time step t, Z T Represents the sum of the cumulative rewards obtained from time step t until a certain termination time T; Indicates that in state X t Here, π refers to the strategy that the agent will take next. The agent here uses the strategy function π to select the action of the token to be generated next, so as to form a comprehensive description of patient feedback, medical records or service evaluation; ψ and θ correspond to the parameters of different model components, respectively, where ψ represents the value function , θ represents the parameters of the policy function, that is, the policy network parameters; the value function It is approximated by a network called Critic, whose task is to evaluate the expected return of a given strategy under different states; the parameter ψ is optimized by training the Critic network to predict the state value; the given strategy π θ According to the current state X t Generate action A t The probability distribution of Update the gradient of the policy network parameter θ, specifically by minimizing it through gradient descent So that the specified strategy π θ Perform better in the environment so that the agent learns to take actions that maximize long-term rewards; Baseline value function Estimates the following policy π θ The profit Z T , and is parameterized by ψ, and the loss during training is: in, is the value function The goal is to minimize this loss so that the value function can more accurately predict the state X when following the specified policy π. t Expected return at the start; Z T It represents the sum of the cumulative rewards obtained from the current time step t to a certain termination time point T at the end of the sequence or the completion of the task, that is, starting from the current state, following the specified strategy π θ The discounted sum of all immediate rewards obtained until the end of the task, T is the termination time point of the sequence, or the time step when a certain termination condition is reached; or, Use A2C algorithm for reinforcement learning; In the policy network of the A2C algorithm, the unbiased estimate of the policy gradient is g(s t ,a t ; θ), Among them, g(s t ,a t ; θ) represents the unbiased estimate of the policy gradient, Q π (s t ,a t ) is the policy value function, s t represents the current state at time t, a t represents the action performed by the agent at time t; V π (s t ) represents the state value function, which represents the current state s t The quality of the policy function Q π (s t ,a t ), which is used to determine whether the benefit generated by the current action is higher than the average benefit of all actions, that is, the value of the predicted strategy value function, and add a baseline. is the policy function π(a t |s t ;θ) in the current state s t Take action a t The probability distribution of the gradient of the policy network parameter θ; if the advantage function is positive, that is, taking action a t is favorable, then the policy network parameter θ will move toward increasing the action a t The probability of the direction update; if the advantage function is negative, that is, take action a t is unfavorable, then the policy network parameters θ will be directed toward reducing the action a t The direction of the probability is updated; According to the state s at time t+1 t+1 With reward value r t For g(s t ,a t ; The expectation in θ) is approximated by Monte Carlo to obtain use To update the policy network parameters θ, the policy gradient method is used. To ensure stability, the advantage function is used as the weight to adjust the update direction of the policy network: Among them, θ' represents the updated policy network parameters; θ represents the policy network parameters before updating; β is the learning rate, which is used to control the amplitude of parameter update.

7. The reinforcement learning alignment model training system based on AI feedback according to claim 6 is characterized in that: Further including: Before performing preference marking, marking the text to be marked as an example of a first quality answer and a second quality answer, wherein the text answer data pair includes a first quality answer and a second quality answer, and the quality of the first quality answer is higher than the quality of the second quality answer; wherein the first quality answer and the second quality answer are used to distinguish the quality of the text and the degree of matching with the user preference; A large language model is used to perform preference marking on the text to be marked, specifically, the examples mentioned above and the text to be marked are input into the large language model to obtain a preference marking result, and the preference marking result includes a candidate answer pair, specifically a first candidate answer and a second candidate answer.

Images