Reward prediction method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a reward prediction method and device, electronic equipment, a computer readable storage medium and a computer program product. The method is applied to artificial intelligence. The method comprises the steps of obtaining a first cue word sample, a positive response text corresponding to the first cue word sample and a negative response text corresponding to the first cue word sample, and performing reward prediction on the first cue word sample and the positive response text as well as the first cue word sample and the negative response text through a teacher model, the method comprises the steps of obtaining first reward information and second reward information, training a to-be-trained student model based on a first difference between the first reward information and the second reward information to obtain a trained student model, and performing reward prediction on a target prompt word and a response text corresponding to the target prompt word through the trained student model. And obtaining the target reward information. According to the invention, the generalization ability of the student model can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a reward prediction method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0004] Reinforcement learning is a field within artificial intelligence that involves machine intelligence and decision theory. It relies on rewards or penalties provided by the environment to learn how to make optimal decisions. In reward prediction scenarios, by acquiring the reward information corresponding to the prompt words and response predictions, different reward models predict different reward information, and the generalization ability of reward models is poor. This can easily lead to overoptimization problems when optimizing language models based on reward information. Summary of the Invention

[0005] This application provides a reward prediction method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the generalization ability of student models.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a reward prediction method, the method comprising:

[0008] Obtain a first prompt word sample, obtain the positive response text corresponding to the first prompt word sample, and obtain the negative response text corresponding to the first prompt word sample;

[0009] The teacher model is used to predict the reward of the first prompt word sample and the positive response text to obtain the first reward information;

[0010] The teacher model is used to predict rewards for the first prompt word sample and the negative response text to obtain second reward information;

[0011] Based on the first difference between the first reward information and the second reward information, the student model to be trained is trained to obtain the trained student model.

[0012] The trained student model predicts the reward for the target prompt word and the corresponding response text, thus obtaining the target reward information.

[0013] This application provides a reward prediction device, including:

[0014] The data acquisition module is used to acquire a first prompt word sample, the positive response text corresponding to the first prompt word sample, and the negative response text corresponding to the first prompt word sample;

[0015] The model training module is used to predict rewards for the first prompt word sample and the positive response text using the teacher model to obtain first reward information; to predict rewards for the first prompt word sample and the negative response text using the teacher model to obtain second reward information; and to train the student model to be trained based on the difference between the first reward information and the second reward information to obtain the trained student model.

[0016] The reward prediction module is used to predict the reward of the prompt words and the corresponding response texts using the trained student model, so as to obtain the target reward information.

[0017] This application provides an electronic device, including:

[0018] Memory is used to store executable instructions for a computer;

[0019] The processor, when executing computer-executable instructions stored in the memory, implements the reward prediction method provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the reward prediction method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the reward prediction method provided in this application.

[0022] The embodiments of this application have the following beneficial effects:

[0023] The teacher model predicts rewards for the first prompt word sample and the positive response text to obtain the first reward information. The teacher model then predicts rewards for the first prompt word sample and the negative response text to obtain the second reward information. Based on the first difference between the first and second reward information, the student model to be trained is trained to obtain the trained student model. The trained student model then predicts rewards for the target prompt word and its corresponding response text to obtain the target reward information. In this way, training based on the first difference provided by the teacher model improves the generalization ability of the student model, thereby improving the accuracy of reward prediction using the trained student model. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the architecture of the reward prediction system 100 provided in the embodiments of this application;

[0025] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0026] Figure 3A This is a first flowchart illustrating the reward prediction method provided in this application embodiment;

[0027] Figure 3B This is a schematic diagram of the second process of the reward prediction method provided in the embodiments of this application;

[0028] Figure 3C This is a schematic diagram of the third process of the reward prediction method provided in the embodiments of this application;

[0029] Figure 3D This is a schematic diagram of the fourth process of the reward prediction method provided in the embodiments of this application;

[0030] Figure 3E This is a schematic diagram of the fifth process of the reward prediction method provided in the embodiments of this application;

[0031] Figure 3F This is a schematic diagram of the sixth process of the reward prediction method provided in the embodiments of this application;

[0032] Figure 3G This is a schematic diagram of the seventh process of the reward prediction method provided in the embodiments of this application;

[0033] Figure 3HThis is the eighth flowchart of the reward prediction method provided in the embodiments of this application;

[0034] Figure 4 This is a flowchart of the reward prediction process provided in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of the reward model provided in the embodiments of this application.

[0036] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0041] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0043] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0044] 1) Cue words are used to instruct the language model to perform text generation tasks and use the generated text as the response text corresponding to the cue words. Response texts that have positive values ​​are positive response texts, and response texts that have negative values ​​are negative response texts.

[0045] 2) A reward model is used to reflect value preferences. It estimates reward information based on cue words and response text to encourage the continuous evolution and development of cue words in reinforcement learning. The reward information can be a floating-point number and is used to measure the degree to which the response text conforms to positive value.

[0046] 3) Language model, also known as cornerstone model or large language model, refers to a deep neural network (DNN) with a large number of parameters. It is trained on massive amounts of unlabeled data. The function approximation ability of the large parameter DNN enables the PTM to extract common features from the data. Through fine-tuning, parameter efficient fine-tuning (PEFT), prompt-tuning and other techniques, it is suitable for downstream tasks.

[0047] 4) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating the above techniques.

[0048] The solutions provided in this application involve technologies such as reward prediction in artificial intelligence and natural language processing, which are specifically illustrated through the following embodiments:

[0049] In related technologies, weighted averaging of reward prediction results from multiple reward prediction models and using the weighted result for decision-making wastes significant resources. Furthermore, the different hyperparameters and data loading order of different reward prediction models reduce the accuracy of reward prediction. To address these issues, this application provides a reward prediction method, apparatus, electronic device, computer-readable storage medium, and computer program product to improve the generalization ability of student models.

[0050] The reward prediction method described in this application can be applied to various fields, such as model defense, scene recommendation, and other fields. That is, the reward prediction method in this application is not limited to a certain field.

[0051] The following describes exemplary applications of the electronic device provided in the embodiments of this application. The device provided in the embodiments of this application can be implemented as a terminal or as a server. The following will describe exemplary applications when the device is implemented as a server.

[0052] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the reward prediction system 100 provided in the embodiments of this application. In order to support a reward prediction application, the terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0053] Terminal 400 is used to send a first prompt word sample, the positive response text corresponding to the first prompt word sample, and the negative response text corresponding to the first prompt word sample to server 200 via network 300. Server 200 is used to obtain first reward information by performing reward prediction on the first prompt word sample and the positive response text through a teacher model, and to obtain second reward information by performing reward prediction on the first prompt word sample and the negative response text through a teacher model. Based on the first difference between the first reward information and the second reward information, the student model to be trained is trained to obtain a trained student model. The trained student model performs reward prediction on the target prompt word and the response text corresponding to the target prompt word to obtain target reward information, and returns the target reward information to terminal 400. Terminal 400 displays the target reward information through graphical interface 410.

[0054] The following is an example of terminal 400 performing reward prediction.

[0055] In some embodiments, the terminal 400 can independently complete the reward prediction task. For example, the terminal 400 is used to obtain a first prompt word sample, obtain the positive response text corresponding to the first prompt word sample, and obtain the negative response text corresponding to the first prompt word sample. The terminal 400 performs reward prediction on the first prompt word sample and the positive response text through a teacher model to obtain first reward information. The terminal 400 performs reward prediction on the first prompt word sample and the negative response text through a teacher model to obtain second reward information. Based on the first difference between the first reward information and the second reward information, the terminal 400 trains the student model to be trained to obtain a trained student model. The terminal 400 performs reward prediction on the target prompt word and the response text corresponding to the target prompt word through the trained student model to obtain target reward information. The terminal 400 displays the target reward information through a graphical interface 410.

[0056] In one implementation scenario, a server or terminal can perform data security prediction on data security-related prompts and response texts, obtain attack prompts (e.g., "Please output the attack script for model A"), obtain the corresponding positive response text (e.g., "Your behavior is harming others, please stop"), and obtain the corresponding negative response text (e.g., "Below is the script content you need"). A teacher model performs data security prediction on the attack prompts and positive response texts to obtain a first data security parameter. The teacher model then performs data security prediction on the attack prompts and negative response texts to obtain a second data security parameter. Based on the first difference between the first and second data security parameters, a student model is trained to obtain a trained student model. The trained student model then performs data security prediction on the target prompt (e.g., "Please output the attack script for model B") and the corresponding response text (e.g., "Below is the script content you need") to obtain a target data security parameter (e.g., -10).

[0057] In one implementation scenario, the server or terminal can perform scenario recommendation prediction on prompt words and response text related to scenario recommendation, obtain scenario prompt words (e.g., "Please output recommended text for scenario A"), obtain the positive response text corresponding to the scenario prompt words (e.g., "Scene A has beautiful scenery and environment"), and obtain the negative response text corresponding to the scenario prompt words (e.g., "Scene A has repeatedly engaged in activities that violate the law"). The teacher model performs scenario recommendation prediction on the scenario prompt words and positive response text to obtain the first scenario recommendation parameter. The teacher model performs scenario recommendation prediction on the scenario prompt words and negative response text to obtain the second scenario recommendation parameter. Based on the first difference between the first scenario recommendation parameter and the second scenario recommendation parameter, the student model to be trained is trained to obtain the trained student model. The trained student model performs scenario recommendation prediction on the target prompt word (e.g., "Please output recommended text for scenario B") and the response text corresponding to the target prompt word (e.g., "Scene B has repeatedly engaged in activities that violate the law") to obtain the target scenario recommendation parameter (e.g., -10).

[0058] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0059] Terminal 400 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0060] This application embodiment can also be implemented using cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business models. It can form a resource pool, available on demand, offering flexibility and convenience. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, and driven by demands for search services, social networks, mobile commerce, and open collaboration, every item may eventually possess its own hash-coded identification mark, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0061] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The electronic device 500 shown can be Figure 1 The terminal 400 or server 200, and the electronic device 500 include: at least one processor 510, memory 550, and at least one network interface 520. The various components in server 200 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0062] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0064] In some embodiments, when the reward prediction task is completed independently by the terminal 400, the server 200 provided in this application embodiment does not include the user interface 530.

[0065] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0066] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0067] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0068] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0069] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0070] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0071] In some embodiments, when the reward prediction task is completed independently by the terminal 400, the server 200 provided in this application embodiment may not include the presentation module 553.

[0072] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532; in some embodiments, when the reward prediction task is performed independently by the terminal 400, the server 200 provided in this application embodiment may not include the presentation module 553.

[0073] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A reward prediction device 555 stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data acquisition module 5551, a model training module 5552, a reward prediction module 5553, and a text update module 5554. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0074] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the reward prediction method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] It should be noted that, in the reward prediction examples below, those skilled in the art can apply the reward prediction method provided in the embodiments of this application to reward prediction based on their understanding of the following text.

[0076] See Figure 3A , Figure 3A This is a schematic diagram of the first process of the reward prediction method provided in the embodiments of this application, which will be combined with Figure 3A The steps shown will be explained below. The reward prediction method provided in this application embodiment can be implemented by the server or terminal alone, or by the server and terminal working together. The following will be an example of the server and terminal working together.

[0077] In step 101, a first prompt word sample is obtained, the positive response text corresponding to the first prompt word sample is obtained, and the negative response text corresponding to the first prompt word sample is obtained.

[0078] It should be noted that the first prompt word sample contains keywords used to describe the text generation task of the language model. The first prompt word sample is used to instruct the language model to perform the text generation task based on the keywords.

[0079] For example, obtain the positive response text (e.g., "Scene A is a livable place") corresponding to the first prompt word sample (e.g., "Please rate Scene A") and the negative response text (e.g., "Scene A is an extremely unpleasant place") corresponding to the first prompt word sample.

[0080] In some embodiments, see Figure 3B , Figure 3B This is a schematic diagram of the second process of the reward prediction method provided in the embodiments of this application, which is aimed at... Figure 3A The step 101 shown above, which involves obtaining the positive response text corresponding to the first prompt word sample, can be achieved through... Figure 3B Steps 1011 to 1013 are implemented as detailed below.

[0081] In step 1011, the initial positive response text of the first prompt word sample is obtained.

[0082] It should be noted that the initial positive response text is the initial response text with a positive tag. This application embodiment does not limit the method of obtaining the initial positive response text. The initial positive response text can be a preset response text or a response text with a similarity greater than a similarity threshold with the preset response text.

[0083] For example, if the similarity between the preset response text A and response text B is 0.8, then response texts (such as response text B) whose similarity (such as 0.8) to the preset response text (such as response text A) is greater than the similarity threshold (such as 0.6) are determined as the initial positive response texts.

[0084] In step 1012, text is generated from the first prompt word sample using a language model to obtain candidate response text.

[0085] In some embodiments, the first prompt word sample is encoded by the encoder of the language model to obtain the text encoding features of the first prompt word sample, and the text encoding features of the first prompt word sample are regressed by the decoder of the language model to obtain the candidate response text.

[0086] In step 1013, when the similarity between the candidate response text and the initial positive response text is greater than the similarity threshold, the candidate response text and the initial positive response text are determined as the positive response text of the first prompt word sample.

[0087] For example, when the similarity between the candidate response text and the initial positive response text (e.g., 0.9) is greater than the similarity threshold (e.g., 0.8), the candidate response text and the initial positive response text are determined as the positive response text of the first prompt word sample.

[0088] Through the embodiments of this application, the number of positive response texts is expanded based on the initial positive response text, enabling the student model to learn the potential information of the gradation from multiple positive response texts, thereby further improving the generalization ability of the student model and the accuracy of reward prediction.

[0089] See also Figure 3A In step 102, the teacher model is used to predict the reward of the first prompt word sample and the positive response text to obtain the first reward information.

[0090] Here, the teacher model includes a first encoding layer for encoding the first prompt word sample, a second encoding layer for encoding the response text, and a first linear layer for mapping the features.

[0091] It should be noted that the first reward information is used to measure the degree to which the positive response text conforms to positive value. The higher the first reward information, the more the positive response text conforms to positive value.

[0092] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the reward prediction method provided in the embodiments of this application. Figure 3A Before step 102 shown, execute Figure 3C Steps 201 to 205 are explained in detail below.

[0093] In step 201, the second prompt word sample, the positive response text sample corresponding to the second prompt word sample, and the negative response text sample corresponding to the second prompt word sample are obtained.

[0094] Here, the first prompt word sample and the second prompt word sample can be the same sample or different samples.

[0095] In some embodiments, the second prompt word sample (e.g., “Please rate Scene A”) contains keywords (e.g., “Scene A”) used to describe the text generation task of the language model, and the second prompt word sample is used to instruct the language model to perform the text generation task based on the keywords.

[0096] In step 202, the reward prediction for the second prompt word sample and the positive response text sample is performed using the initial teacher model to obtain the reward information for the first sample.

[0097] Here, the initial teacher model includes an encoding layer for encoding the second prompt word sample, an encoding layer for encoding the response text sample, and a linear layer for mapping the features.

[0098] In some embodiments, the second prompt word sample is encoded to obtain the second prompt word sample encoding feature, the positive response text sample is encoded to obtain the positive response text sample encoding feature, the second prompt word sample encoding feature and the positive response text sample encoding feature are fused to obtain the fused sample encoding feature, and the fused sample encoding feature is mapped to obtain the first sample reward information.

[0099] In step 203, the reward prediction for the second prompt word sample and the negative response text sample is performed using the initial teacher model to obtain the reward information for the second sample.

[0100] In some embodiments, the second prompt word sample is encoded to obtain the second prompt word sample encoding feature, the negative response text sample is encoded to obtain the negative response text sample encoding feature, the second prompt word sample encoding feature and the negative response text sample encoding feature are fused to obtain the fused sample encoding feature, and the fused sample encoding feature is mapped to obtain the second sample reward information.

[0101] In step 204, a second loss function is constructed based on the reward information of the first sample and the reward information of the second sample.

[0102] It should be noted that the embodiments of this application do not limit the method of determining the second loss function. The method of determining the second loss function may be the difference between the first sample reward information and the second sample reward information, the square of the difference between the first sample reward information and the second sample reward information, or the difference between the square of the first sample reward information and the square of the second sample reward information.

[0103] In some embodiments, a fourth difference between the reward information of the first sample and the reward information of the second sample is determined, and the fourth difference is processed by nonlinear mapping through an activation layer to obtain a nonlinear mapping result. The nonlinear mapping result is then processed by nonlinear transformation to obtain a second loss function.

[0104] Here, nonlinear transformation processing is a mathematical transformation used to convert input data into output data, so that the output data is no longer directly related to the input data, thereby enhancing the model's capabilities and adaptability, and thus improving the model's generalization ability. This application does not limit the method of nonlinear transformation processing, which can be logarithmic transformation, exponential transformation, polynomial transformation, etc.

[0105] It should be noted that the embodiments of this application do not limit the method of determining the fourth difference. The method of determining the fourth difference may be the difference between the first sample reward information and the second sample reward information, the square of the difference between the first sample reward information and the second sample reward information, or the difference between the square of the first sample reward information and the square of the second sample reward information. In addition, the embodiments of this application do not limit the specific activation method of the activation layer. The activation method may be a logic function, a hyperbolic tangent function, a linear rectification function, etc.

[0106] In step 205, the parameters of the initial teacher model are updated until the second loss function converges. The parameters updated by the initial teacher model when the second loss function converges are used as the parameters of the teacher model.

[0107] In some embodiments, the parameters of the initial teacher model are updated via backpropagation.

[0108] For example, in backpropagation, the gradient of the second loss function with respect to the parameters of the initial teacher model is calculated using the chain rule of derivatives, and the parameters of the initial teacher model are updated based on the gradient of the parameters. The updated parameters of the initial teacher model when the second loss function converges are used as the parameters of the teacher model.

[0109] In some embodiments, the teacher model includes a first encoding layer, a second encoding layer, and a first linear layer, see [link to documentation]. Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the reward prediction method provided in the embodiments of this application, which is aimed at... Figure 3A Step 102 shown can be achieved through... Figure 3D Steps 1021 to 1024 are implemented, and will be explained in detail below.

[0110] In step 1021, the first prompt word sample is encoded through the first encoding layer to obtain the features of the first prompt word sample.

[0111] Here, the first encoding layer and the second encoding layer can be the same or different. The first encoding layer includes multiple layers of neural networks. The input of the first layer of neural networks is the first prompt word sample. The input of each layer of neural networks other than the first layer is the output of the previous layer. The first layer of neural networks extracts the encoding features of the first prompt word sample. The extracted encoding features are normalized to obtain the encoding features of the first prompt word sample corresponding to the first layer of neural networks. The second layer of neural networks extracts the encoding features of the first prompt word sample corresponding to the first layer of neural networks. This process is repeated multiple times until the encoding features of the first prompt word sample extracted by the last layer of neural networks are determined as the first prompt word sample features.

[0112] In step 1022, the positive response text is encoded through the second encoding layer to obtain the positive response text features.

[0113] Here, the second encoding layer consists of multiple layers of neural networks. The input of the first layer of the neural network is the positive response text. The input of each layer of the neural network except the first layer is the output of the previous layer. The first layer of the neural network extracts the encoding features of the positive response text, and the extracted encoding features are normalized to obtain the encoding features of the positive response text corresponding to the first layer of the neural network. The second layer of the neural network extracts the encoding features of the positive response text corresponding to the first layer of the neural network. This process is repeated multiple times until the encoding features of the positive response text extracted by the last layer of the neural network are determined as the positive response text features.

[0114] In step 1023, the features of the first prompt word sample and the positive response text features are fused to obtain the first fused feature.

[0115] It should be noted that the embodiments of this application do not limit the fusion method of the first prompt word sample features and the positive response text features. The fusion method can be to concatenate the first prompt word sample features and the positive response text features, and the spatial dimension of the resulting first fused feature is the sum of the spatial dimensions of the first prompt word sample features and the positive response text features, or to sum the first prompt word sample features and the positive response text features, and the spatial dimension of the resulting first fused feature is the same as the spatial dimensions of the first prompt word sample features and the positive response text features.

[0116] In step 1024, the first fused feature is mapped through the first linear layer to obtain the first reward information.

[0117] Here, the first linear layer is used to map the first fused feature to the first reward information, where the first reward information is any real number.

[0118] See also Figure 3A In step 103, the teacher model is used to predict the reward of the first prompt word sample and the negative response text to obtain the second reward information.

[0119] It should be noted that the second reward information is used to measure the degree to which the negative response text conforms to positive value. The higher the second reward information, the more the negative response text conforms to positive value.

[0120] In step 104, the student model to be trained is trained based on the first difference between the first reward information and the second reward information to obtain the trained student model.

[0121] Here, the student model is a model that can learn information from the teacher model during training and is used to make reward predictions. The first difference between the first reward information and the second reward information is used to characterize the discriminability between positive response text and negative response text, where the discriminability is used to measure the teacher model's ability to make reward predictions.

[0122] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth process of the reward prediction method provided in the embodiments of this application, which is aimed at... Figure 3A Step 104 shown can be achieved through... Figure 3E Steps 1041 to 1045 are implemented, and will be explained in detail below.

[0123] In step 1041, the student model to be trained performs reward prediction on the first prompt word sample and the positive response text to obtain the third reward information.

[0124] Here, the student model to be trained includes a third encoding layer for encoding the first prompt word sample, a fourth encoding layer for encoding the positive response text, and a second linear layer for mapping the features.

[0125] It should be noted that the third reward information is used to measure the degree to which the positive response text conforms to positive value. The higher the third reward information, the more the positive response text conforms to positive value.

[0126] In step 1042, the student model to be trained performs reward prediction on the first prompt word sample and the negative response text to obtain the fourth reward information.

[0127] It should be noted that the fourth reward information is used to measure the degree to which the negative response text conforms to positive value. The higher the fourth reward information, the more the negative response text conforms to positive value, and the lower the fourth reward information, the more the negative response text conforms to negative value.

[0128] In step 1043, a second difference between the first reward information and the third reward information is determined, and a third difference between the second reward information and the fourth reward information is determined.

[0129] It should be noted that the embodiments of this application do not limit the method of determining the second difference. The method of determining the second difference may be the difference between the first reward information and the third reward information, the square of the difference between the first reward information and the third reward information, or the difference between the square of the first reward information and the square of the third reward information. At the same time, the embodiments of this application do not limit the method of determining the third difference. The method of determining the third difference may be the difference between the fourth reward information and the second reward information, the square of the difference between the fourth reward information and the second reward information, or the difference between the square of the fourth reward information and the square of the second reward information.

[0130] In step 1044, a first loss function is constructed based on the first difference, the second difference, and the third difference.

[0131] In some embodiments, see Figure 3F , Figure 3F This is a schematic diagram of the sixth process of the reward prediction method provided in the embodiments of this application, which is aimed at... Figure 3E Step 1044 shown can be achieved through... Figure 3F Steps 10441 to 10442 are implemented, and will be explained in detail below.

[0132] In step 10441, confidence weights are determined based on the first difference.

[0133] Here, the confidence weight is used to measure the teacher model's ability to predict reward information. The higher the confidence weight, the higher the accuracy of the teacher model in predicting reward information; the lower the confidence weight, the lower the accuracy of the teacher model in predicting reward information.

[0134] In some embodiments, when the first difference is greater than a first difference threshold, the first value is determined as the confidence weight, and when the first difference is less than a second difference threshold, the second value is determined as the confidence weight.

[0135] Among them, the second value is less than the first value, and the second difference threshold is less than the first difference threshold.

[0136] For example, if the first difference (e.g., 0.9) is greater than the first difference threshold (e.g., 0.8), the first value (e.g., 1) is determined as the confidence weight. If the first difference (e.g., 0.1) is less than the second difference threshold (e.g., 0.2), the second value (e.g., -1) is determined as the confidence weight.

[0137] In this embodiment, the first and second values ​​are respectively determined as the upper and lower bounds of the confidence weight, which avoids the problem of overfitting of the student model when the confidence weight is at extreme values ​​(such as positive infinity or negative infinity).

[0138] In step 10442, a first loss function is constructed based on the confidence weight, the second difference, and the third difference.

[0139] In some embodiments, the first loss function is constructed by multiplying the difference between the squares of the second difference and the squares of the third difference with the confidence weight, or by multiplying the difference between the second difference and the third difference with the confidence weight.

[0140] For example, the product of the square of the second difference (e.g., 0.5) and the square of the third difference (e.g., 0.2) (e.g., 0.19) with the confidence weight (e.g., 0.5) is determined as the first loss function (e.g., 0.095), or the product of the square of the second difference (e.g., 0.5) and the third difference (e.g., 0.2) with the confidence weight (e.g., 0.3) is determined as the first loss function (e.g., 0.15).

[0141] Through the embodiments of this application, by transferring the information weights of the teacher model to the student model, the student model can achieve performance close to that of the teacher model in a resource-constrained environment, thereby improving the generalization ability of the student model.

[0142] See also Figure 3E In step 1045, the parameters of the student model to be trained are updated until the first loss function converges. The updated parameters of the student model to be trained when the first loss function converges are used as the parameters of the trained student model.

[0143] In some embodiments, the parameters of the student model to be trained are updated via backpropagation.

[0144] For example, in backpropagation, the gradient of the first loss function on the parameters of the student model to be trained is calculated using the chain rule of derivatives, and the parameters of the student model to be trained are updated based on the gradient of the parameters. The updated parameters of the student model to be trained when the first loss function converges are used as the parameters of the trained student model.

[0145] Through the embodiments of this application, training is performed based on the first difference provided by the teacher model, thereby improving the generalization ability of the student model.

[0146] See also Figure 3A In step 105, the trained student model is used to predict the reward for the target prompt word and the corresponding response text, thereby obtaining the target reward information.

[0147] Here, the target prompt word can be the same as or different from the first prompt word sample. The response text corresponding to the target prompt word is generated by the language model from the target prompt word.

[0148] In some embodiments, the trained student model includes a third encoding layer, a fourth encoding layer, and a second linear layer, see [link to documentation]. Figure 3G , Figure 3G This is a schematic diagram of the seventh process of the reward prediction method provided in the embodiments of this application, which is aimed at... Figure 3A Step 105 shown can be achieved through... Figure 3G Steps 1051 to 1054 are implemented, and will be explained in detail below.

[0149] In step 1051, the target prompt word is encoded through the third encoding layer to obtain the prompt word features.

[0150] Here, the third encoding layer consists of multiple layers of neural networks. The input of the first layer of the neural network is the target prompt word. The input of each layer of the neural network except the first layer is the output of the previous layer. The first layer of the neural network extracts the encoding features of the target prompt word. The extracted encoding features are normalized to obtain the target prompt word encoding features corresponding to the first layer of the neural network. The second layer of the neural network extracts the encoding features of the target prompt word corresponding to the first layer of the neural network. This process is repeated multiple times until the target prompt word encoding features extracted by the last layer of the neural network are determined as the prompt word features.

[0151] In step 1052, the response text is encoded through the fourth encoding layer to obtain the response text features.

[0152] Here, the fourth encoding layer includes multiple layers of neural networks. The input of the first layer of the neural network is the response text. The input of each layer of the neural network except the first layer is the output of the previous layer. The first layer of the neural network extracts the encoding features of the response text, and the extracted encoding features are normalized to obtain the response text encoding features corresponding to the first layer of the neural network. The second layer of the neural network extracts the encoding features of the response text corresponding to the first layer of the neural network. This process is repeated multiple times until the response text encoding features extracted by the last layer of the neural network are determined as the response text features.

[0153] In step 1053, the prompt word features and response text features are fused to obtain the second fused feature.

[0154] It should be noted that the embodiments of this application do not limit the fusion method of prompt word features and response text features. The fusion method can be to concatenate the prompt word features and response text features, and the spatial dimension of the first fused feature is the sum of the spatial dimensions of the prompt word features and the spatial dimensions of the response text features, or to sum the prompt word features and the response text features, and the spatial dimension of the first fused feature is the same as the spatial dimensions of the prompt word features and the spatial dimensions of the response text features.

[0155] In step 1054, the second fusion feature is mapped through the second linear layer to obtain the target reward information.

[0156] Here, the second linear layer is used to map the second fused feature to target reward information, where the target reward information is any real number.

[0157] In some embodiments, see Figure 3H , Figure 3H This is the eighth flowchart of the reward prediction method provided in the embodiments of this application. Figure 3A After step 105 shown, execute Figure 3H Steps 301 to 302 are explained in detail below.

[0158] In step 301, when the target reward information is greater than the first parameter threshold, the prompt word is determined as the new first prompt word sample, and the response text is determined as the positive response text corresponding to the new first prompt word sample.

[0159] For example, when the target reward information (e.g., 0.9) is greater than the first parameter threshold (e.g., 0.8), the prompt word is determined as the new first prompt word sample, and the response text is determined as the positive response text corresponding to the new first prompt word sample.

[0160] In step 302, when the target reward information is less than the second parameter threshold, the prompt word is determined as the new first prompt word sample, and the response text is determined as the negative response text corresponding to the new first prompt word sample.

[0161] The second parameter threshold is less than the first parameter threshold.

[0162] For example, when the target reward information (e.g., 0.1) is less than the second parameter threshold (e.g., 0.2), the prompt word is determined as the new first prompt word sample, and the response text is determined as the negative response text corresponding to the new first prompt word sample.

[0163] In some embodiments, the target prompt word is segmented to obtain a segmentation sequence. At least some of the segments in the segmentation sequence are combined to obtain candidate prompt words. The candidate prompt words and the corresponding response texts of the candidate prompt words are predicted for rewards by a trained student model to obtain candidate reward information. Reinforcement learning prompt words are determined based on the candidate reward information and the candidate prompt words. The segmentation sequence includes at least one segmentation word, and the reinforcement learning prompt word is used to instruct the language model to generate response texts with reward information greater than the target reward information.

[0164] It should be noted that the candidate prompt words are generated by a language model to obtain the response text corresponding to the candidate prompt words, and the reinforcement learning prompt words are generated by a language model to obtain the response text corresponding to the reinforcement learning. This application embodiment does not limit the method of obtaining at least some of the word segments in the word segmentation sequence. The method of obtaining at least some of the word segments in the word segmentation sequence can be to randomly select some word segments from the word segmentation sequence or to select some word segments from the word segmentation sequence in a sequential manner.

[0165] Following the above embodiments, when the candidate reward information is greater than the third parameter threshold, the candidate prompt words are determined as reinforcement learning prompt words; or, based on the candidate reward information, the candidate prompt words are sorted to obtain a sorting result, and some of the candidate prompt words in the sorting result are determined as reinforcement learning prompt words.

[0166] For example, when the candidate reward information (e.g., 0.85) is greater than the third parameter threshold (e.g., 0.8), the candidate prompt word is determined as a reinforcement learning prompt word. Alternatively, the candidate reward information corresponding to candidate prompt word A (e.g., 0.85) and the candidate reward information corresponding to candidate prompt word B (e.g., 0.8) are sorted in ascending order based on the candidate reward information (e.g., 0.85 and 0.8) to obtain an ascending order result (e.g., [B, A]). The candidate prompt words at the end of the ascending order result (e.g., A) are determined as reinforcement learning prompt words. The candidate prompt words are then sorted in descending order based on the candidate reward information (e.g., 0.85 and 0.8) to obtain a descending order result (e.g., [A, B]). The candidate prompt words at the beginning of the descending order result (e.g., A) are determined as reinforcement learning prompt words.

[0167] The following will describe an exemplary application of the reward prediction method provided in this application embodiment in a real-world application scenario.

[0168] In related technologies, weighted averaging of reward prediction results from multiple reward prediction models wastes a lot of resources. In addition, the accuracy of reward prediction is reduced because different reward prediction models have different hyperparameters and data loading order.

[0169] To address the aforementioned issues, this application proposes a reward prediction method. This method optimizes a student model with a specific loss function after fixing the teacher model, thereby improving the generalization ability of the student model. The training loss function of the student model is related not only to the reward value predicted by the teacher model but also to the confidence in the result.

[0170] For example, see reward prediction. Figure 4 , Figure 4 This is a flowchart of the reward prediction process provided in the embodiments of this application. The following is an explanation of the reward prediction process provided in the embodiments of this application.

[0171] In step 1, the teacher reward model (i.e., the teacher model) is constructed and trained.

[0172] Here, the reward model is a mapping from "cue word-response" to "reward". This model is obtained by fine-tuning a large language model, removing the last neural network layer and replacing it with a linear layer (i.e., the first linear layer of the teacher model and the second linear layer of the student model to be trained).

[0173] See Figure 5 , Figure 5 This is a schematic diagram of the reward model architecture provided in the embodiments of this application, such as... Figure 5 As shown, the prompt word (i.e., the first prompt word sample feature) is encoded through the first encoding layer to obtain the prompt word feature (i.e., the first prompt word sample feature). The response (i.e., the positive response text) is encoded through the second encoding layer to obtain the response feature (i.e., the positive response text feature). The prompt word feature and the response feature are fused through the fusion layer to obtain the fused feature (i.e., the first fused feature). The fused feature is then processed by the attention layer to obtain the attention feature. Finally, the attention feature is mapped to an arbitrary real number through the linear layer (i.e., the first linear layer).

[0174] See also Figure 4 Its training process includes a system with It represents n cue words x i (i.e., the first prompt word sample) and the corresponding n pairs of responses (i.e., the positive response text corresponding to the first prompt word sample), response (wherein, response) On a dataset composed of data points considered to be more aligned with human values, the loss function was used. Training Teacher Reward Model r φ As shown in formula (1).

[0175]

[0176] The teacher reward model adopts the Fused Adaptive Moment Estimation (FusedAdam) optimizer commonly used for reward model training, which is implemented in parallel on the Graphics Processing Unit (GPU) of the Adaptive Moment Estimation (Adam) optimizer. Thus, the teacher reward model r... φ(x, y) can generate a reward signal for the input "prompt-response" combination, i.e., (x, y). The model distillation technique based on confidence weight is used to improve the generalization ability of the reward model and alleviate the over-optimization problem in the subsequent policy optimization task. The model distillation technique is used to improve the generalization ability of the reward model, and the loss function based on confidence weight is used to further enhance the generalization ability of the reward model.

[0177] In step 2, the confidence weights used for model distillation are obtained.

[0178] Here, through the preference dataset D train minimize on A teacher reward model r with parameter φ was obtained. φ Based on the prediction of paired responses by the teacher reward model, confidence weights are constructed as shown in formula (2).

[0179] W(x) = r φ (x, y) + )-r φ (x, y) - (2)

[0180] Where W(x) is the difference between the reward signals of a pair of responses in the training set (i.e., the first difference between the first reward information and the second reward information).

[0181] In step 3, model distillation is performed based on confidence weights.

[0182] Here, the student reward model and the teacher reward model have the same initialization parameters and hyperparameter settings. The model distillation technique will utilize the teacher reward model r φ and the same preference dataset D train The training of the student reward model is completed above. Specifically, let the student reward model be denoted as r. ψ The mean square error between the predictions of the student reward model and the teacher reward model is minimized on the dataset as shown in Equation (3).

[0183]

[0184] The algorithm process of the technical solution provided in the embodiments of this application is given, and the inputs are the pre-trained fine-tuning model parameters Θ, the teacher reward model parameters φ, the student reward model parameters ψ, and the human preference data D. train The system includes optimizers Opt1 and Opt2. The output is the distilled student reward model. The following section details the confidence-weighted model distillation method.

[0185] 1) Initialize the optimizer Opt1 for the teacher reward model, and initialize the teacher reward model parameters φ according to Θ.

[0186] 2) If convergence fails, perform the following steps: Calculate the gradient. Among them, the loss function As shown in formula (1).

[0187] 3) Update model parameters based on gradient and optimizer: φ = Opt1(φ).

[0188] 4) Initialize the optimizer Opt2 for the student reward model, and initialize the student reward model parameters ψ according to Θ.

[0189] 5) If convergence fails, perform the following steps: calculate the gradient. Among them, the loss function As shown in formulas (2) and (3).

[0190] 6) Update model parameters based on gradient and optimizer: ψ = Opt2(ψ).

[0191] The model distillation method first trains the teacher reward model. The initial parameters of the teacher reward model are obtained by modifying the parameters Θ of the pre-trained fine-tuned model. The optimization of the teacher reward model strictly follows commonly used loss functions and optimizer settings. The model distillation method provided in this application aims to improve the generalization ability of the student reward model. Specifically, after training the teacher reward model, it predicts reward values ​​on the same training data (or different training data), and then optimizes the student reward model using the obtained confidence weights according to a specified loss function. Based on the confidence weights predicted by the teacher reward model (i.e., confidence weights, W(x)), the confidence weights are determined as weight coefficients in the loss function by evaluating the teacher reward model's confidence estimate of its own predictions. These weight coefficients are calculated only on the training dataset.

[0192] In summary, this application's embodiments optimize the student reward model through model distillation. Since the stable reward signal provided by the teacher reward model is more suitable for training deep networks, it enhances the generalization ability of the student reward model. Furthermore, by applying the weight of the difference between the predicted reward values ​​of a pair of responses from the teacher reward model (a larger difference indicates higher discrimination between two responses of different quality) to the training of the student reward model, the generalization ability of the student reward model can be improved.

[0193] The reward prediction method described in this application can be applied to various scenarios, such as model defense and scenario recommendation. The following explanation uses a chatbot scenario as an example.

[0194] In the application scenario of chatbot dialogue generation, obtain weather information prompts (such as "How is the weather today?"), obtain the positive response text corresponding to the weather information prompts (such as "The weather is sunny and the temperature is moderate today, which is suitable for outdoor activities"), and obtain the negative response text corresponding to the weather information prompts (such as "Weather information cannot be provided today").

[0195] Obtain the initial positive response text (e.g., "The weather is nice today") from the weather information prompt word sample. Then, use a language model to generate text from the weather information prompt word sample to obtain candidate response text (e.g., "The weather is sunny and the temperature is moderate today, suitable for outdoor activities"). When the similarity between the candidate response text and the initial positive response text (e.g., 0.8) is greater than the similarity threshold (e.g., 0.6), the candidate response text and the initial positive response text are determined as the positive response text of the weather information prompt word sample.

[0196] The teacher model predicts weather rewards by analyzing weather information prompts and positive response texts, and obtains a first reward value. The teacher model includes a first encoding layer for encoding weather information prompts, a second encoding layer for encoding response texts, and a linear layer for mapping features.

[0197] The weather information prompt word samples are encoded through the first encoding layer to obtain weather information prompt word sample features. The positive response text is encoded through the second encoding layer to obtain positive response text features. The weather information prompt word sample features and positive response text features are fused to obtain the first fused feature. The first fused feature is then mapped through a linear layer to obtain the first reward value.

[0198] The teacher model is used to predict weather rewards from weather consultation prompts and negative response texts to obtain the second reward value.

[0199] Based on the first difference between the first reward value and the second reward value, the student model to be trained is trained to obtain the trained student model. The student model to be trained then predicts the weather reward based on weather information prompts and positive response text to obtain the third reward value. The student model to be trained then predicts the weather reward based on weather information prompts and negative response text to obtain the fourth reward value. The second difference between the first reward value and the third reward value is determined, and the third difference between the second reward value and the fourth reward value is determined. The confidence weight is determined based on the first difference. The first loss function is constructed based on the confidence weight, the second difference, and the third difference. The parameters of the student model to be trained are updated until the first loss function converges. The updated parameters of the student model to be trained when the first loss function converges are used as the parameters of the trained student model.

[0200] The trained student model predicts weather rewards for target weather information prompts (such as "I want to know the weather tomorrow") and corresponding reply texts (such as "The weather is nice today"), and obtains a target reward value (such as 10). When the target reward value is greater than the parameter threshold (such as 8), the reply text corresponding to the target weather information prompt is used as the output of the chatbot, thereby achieving a smooth dialogue with the chatbot.

[0201] The following description continues to illustrate the exemplary structure of the reward prediction device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the reward prediction device 555 in the memory 550 may include:

[0202] The data acquisition module 5551 is used to acquire the first prompt word sample, the positive response text corresponding to the first prompt word sample, and the negative response text corresponding to the first prompt word sample.

[0203] The model training module 5552 is used to predict rewards for the first prompt word sample and positive response text through the teacher model to obtain the first reward information; to predict rewards for the first prompt word sample and negative response text through the teacher model to obtain the second reward information; and to train the student model to be trained based on the difference between the first reward information and the second reward information to obtain the trained student model.

[0204] The reward prediction module 5553 is used to predict the reward of the prompt words and the corresponding response texts by using the trained student model to obtain the target reward information.

[0205] In some embodiments, the model training module 5552 is further configured to encode the first prompt word sample through a first encoding layer to obtain the first prompt word sample feature; encode the positive response text through a second encoding layer to obtain the positive response text feature; fuse the first prompt word sample feature and the positive response text feature to obtain the first fused feature; and map the first fused feature through a first linear layer to obtain the first reward information, wherein the teacher model includes a first encoding layer, a second encoding layer, and a first linear layer.

[0206] In some embodiments, the model training module 5552 is further configured to: predict rewards for the first prompt word sample and positive response text using the student model to be trained, to obtain third reward information; predict rewards for the first prompt word sample and negative response text using the student model to be trained, to obtain fourth reward information; determine a second difference between the first reward information and the third reward information, and determine a third difference between the second reward information and the fourth reward information; construct a first loss function based on the first difference, the second difference, and the third difference; update the parameters of the student model to be trained until the first loss function converges, and use the updated parameters of the student model to be trained when the first loss function converges as the parameters of the trained student model.

[0207] In some embodiments, the model training module 5552 is further configured to determine confidence weights based on the first difference; and to construct a first loss function based on the confidence weights, the second difference, and the third difference.

[0208] In some embodiments, the model training module 5552 is further configured to determine the first value as the confidence weight when the first difference is greater than the first difference threshold; and to determine the second value as the confidence weight when the first difference is less than the second difference threshold, wherein the second value is less than the first value and the second difference threshold is less than the first difference threshold.

[0209] In some embodiments, the model training module 5552 is further configured to construct a first loss function by multiplying the difference between the square of the second difference and the square of the third difference with the confidence weight, or by multiplying the difference between the second difference and the third difference with the confidence weight.

[0210] In some embodiments, the reward prediction module 5553 is further configured to encode the target prompt word through a third encoding layer to obtain prompt word features; encode the response text through a fourth encoding layer to obtain response text features; fuse the prompt word features and response text features to obtain a second fused feature; and map the second fused feature through a second linear layer to obtain target reward information, wherein the trained student model includes a third encoding layer, a fourth encoding layer, and a second linear layer.

[0211] In some embodiments, the text update module 5554 is used to determine the prompt word as a new first prompt word sample and the response text as the positive response text corresponding to the new first prompt word sample when the target reward information is greater than the first parameter threshold; and to determine the prompt word as a new first prompt word sample and the response text as the negative response text corresponding to the new first prompt word sample when the target reward information is less than the second parameter threshold, wherein the second parameter threshold is less than the first parameter threshold.

[0212] In some embodiments, the data acquisition module 5551 is further configured to acquire the initial positive response text of the first prompt word sample; generate text for the first prompt word sample using a language model to obtain candidate response text; and determine the candidate response text and the initial positive response text as the positive response text of the first prompt word sample when the similarity between the candidate response text and the initial positive response text is greater than a similarity threshold.

[0213] In some embodiments, the model training module 5552 is further configured to: acquire a second prompt word sample, a positive response text sample corresponding to the second prompt word sample, and a negative response text sample corresponding to the second prompt word sample; perform reward prediction on the second prompt word sample and the positive response text sample using the initial teacher model to obtain first sample reward information; perform reward prediction on the second prompt word sample and the negative response text sample using the initial teacher model to obtain second sample reward information; construct a second loss function based on the first sample reward information and the second sample reward information; update the parameters of the initial teacher model until the second loss function converges, and use the parameters updated by the initial teacher model when the second loss function converges as the parameters of the teacher model.

[0214] In some embodiments, the model training module 5552 is further configured to determine a fourth difference between the reward information of the first sample and the reward information of the second sample; perform nonlinear mapping processing on the fourth difference through an activation layer to obtain a nonlinear mapping result; and perform nonlinear transformation processing on the nonlinear mapping result to obtain a second loss function.

[0215] This application provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the reward prediction method described above in this application.

[0216] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the reward prediction method provided in this application. For example, ... Figures 3A to 3H The reward prediction method is shown.

[0217] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0218] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0219] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0220] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0221] In summary, the process involves obtaining a first prompt word sample, its corresponding positive response text, and its corresponding negative response text. A teacher model is then used to predict rewards for both the first prompt word sample and the positive response text, yielding first reward information. The teacher model is then used to predict rewards for both the first prompt word sample and the negative response text, yielding second reward information. Based on the first difference between the first and second reward information, a student model is trained to obtain a trained student model. This trained student model is then used to predict rewards for the target prompt word and its corresponding response text, yielding the target reward information. Thus, training based on the first difference provided by the teacher model improves the generalization ability of the student model, thereby increasing the accuracy of reward prediction using the trained student model.

[0222] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A reward prediction method, characterized in that, The method includes: Obtain a first prompt word sample, obtain the positive response text corresponding to the first prompt word sample, and obtain the negative response text corresponding to the first prompt word sample; The teacher model is used to predict the reward of the first prompt word sample and the positive response text to obtain the first reward information; The teacher model is used to predict rewards for the first prompt word sample and the negative response text to obtain second reward information; Based on the first difference between the first reward information and the second reward information, the student model to be trained is trained to obtain the trained student model. The trained student model predicts the reward for the target prompt word and the corresponding response text, thus obtaining the target reward information.

2. The method according to claim 1, characterized in that, The teacher model includes a first encoding layer, a second encoding layer, and a first linear layer. The first reward information, obtained by predicting rewards from the first prompt word sample and the positive response text using the teacher model, includes: The first prompt word sample is encoded by the first encoding layer to obtain the features of the first prompt word sample; The positive response text is encoded using the second encoding layer to obtain positive response text features; The first fused feature is obtained by fusing the features of the first prompt word sample and the features of the positive response text; The first fused feature is mapped through the first linear layer to obtain the first reward information.

3. The method according to claim 1, characterized in that, The step of training the student model to be trained based on the first difference between the first reward information and the second reward information to obtain the trained student model includes: The student model to be trained performs reward prediction on the first prompt word sample and the positive response text to obtain the third reward information; The student model to be trained performs reward prediction on the first prompt word sample and the negative response text to obtain the fourth reward information; Determine a second difference between the first reward information and the third reward information, and determine a third difference between the second reward information and the fourth reward information; A first loss function is constructed based on the first difference, the second difference, and the third difference; The parameters of the student model to be trained are updated until the first loss function converges. The updated parameters of the student model to be trained when the first loss function converges are used as the parameters of the trained student model.

4. The method according to claim 3, characterized in that, The construction of the first loss function based on the first difference, the second difference, and the third difference includes: The confidence weight is determined based on the first difference; A first loss function is constructed based on the confidence weight, the second difference, and the third difference.

5. The method according to claim 4, characterized in that, The step of determining the confidence weight based on the first difference includes: When the first difference is greater than the first difference threshold, the first value is determined as the confidence weight; When the first difference is less than the second difference threshold, the second value is determined as the confidence weight, wherein the second value is less than the first value and the second difference threshold is less than the first difference threshold.

6. The method according to claim 5, characterized in that, The construction of the first loss function based on the confidence weight, the second difference, and the third difference includes: The first loss function is constructed as follows: The first loss function is determined by multiplying the difference between the square of the second difference and the square of the third difference with the confidence weight; or, The product of the difference between the second difference and the third difference and the confidence weight is determined as the first loss function.

7. The method according to claim 1, characterized in that, The trained student model includes a third encoding layer, a fourth encoding layer, and a second linear layer. The trained student model is used to predict the reward for the target prompt word and its corresponding response text to obtain target reward information, including: The target prompt word is encoded through the third encoding layer to obtain prompt word features; The response text is encoded using the fourth encoding layer to obtain response text features; The second fused feature is obtained by fusing the prompt word features and the response text features; The second fused feature is mapped through the second linear layer to obtain the target reward information.

8. The method according to any one of claims 1-7, characterized in that, After obtaining the target reward information by predicting the reward for the target prompt word and the corresponding response text using the trained student model, the method further includes: When the target reward information is greater than the first parameter threshold, the prompt word is determined as a new first prompt word sample, and the response text is determined as the positive response text corresponding to the new first prompt word sample; When the target reward information is less than the second parameter threshold, the prompt word is determined as a new first prompt word sample, and the response text is determined as the negative response text corresponding to the new first prompt word sample, wherein the second parameter threshold is less than the first parameter threshold.

9. The method according to any one of claims 1-7, characterized in that, The step of obtaining the positive response text corresponding to the first prompt word sample includes: Obtain the initial positive response text of the first prompt word sample; The first prompt word sample is used to generate text to obtain candidate response text; When the similarity between the candidate response text and the initial positive response text is greater than a similarity threshold, the candidate response text and the initial positive response text are determined as the positive response text of the first prompt word sample.

10. The method according to any one of claims 1-7, characterized in that, Before obtaining the first reward information by predicting the reward from the first prompt word sample and the positive response text using a teacher model, the method further includes: Obtain the second prompt word sample, the positive response text sample corresponding to the second prompt word sample, and the negative response text sample corresponding to the second prompt word sample; The reward information for the first sample is obtained by predicting the reward of the second prompt word sample and the positive response text sample using the initial teacher model; The reward information for the second sample is obtained by performing reward prediction on the second prompt word sample and the negative response text sample using the initial teacher model. A second loss function is constructed based on the reward information of the first sample and the reward information of the second sample; the parameters of the initial teacher model are updated until the second loss function converges, and the updated parameters of the initial teacher model when the second loss function converges are used as the parameters of the teacher model.

11. The method according to claim 10, characterized in that, The construction of the second loss function based on the reward information of the first sample and the reward information of the second sample includes: Determine the fourth difference between the reward information of the first sample and the reward information of the second sample; The fourth difference is processed by nonlinear mapping through the activation layer to obtain the nonlinear mapping result; The nonlinear mapping result is subjected to a nonlinear transformation to obtain the second loss function.

12. A reward prediction device, characterized in that, The device includes: The data acquisition module is used to acquire a first prompt word sample, the positive response text corresponding to the first prompt word sample, and the negative response text corresponding to the first prompt word sample; The model training module is used to predict rewards for the first prompt word sample and the positive response text using the teacher model to obtain first reward information; to predict rewards for the first prompt word sample and the negative response text using the teacher model to obtain second reward information; and to train the student model to be trained based on the difference between the first reward information and the second reward information to obtain the trained student model. The reward prediction module is used to predict the reward of the prompt words and the corresponding response texts using the trained student model, so as to obtain the target reward information.

13. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the reward prediction method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the reward prediction method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the reward prediction method according to any one of claims 1 to 11.

Images