Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a storage medium. The method comprises: based on user interaction associated with a target model, obtaining user feedback information for the target model, the user feedback information comprising corresponding user feedback for a plurality of model outputs, the plurality of model outputs being provided by the target model for the same user input; at least based on the user feedback information, training data is generated, and the training data comprises quality indexes used for evaluating output of the multiple models; and training a reward model for updating the target model by using the training data. Therefore, real-time updating of the reward model can be realized, and real-time updating of the reward model can be felt by the target model. Therefore, the target model can provide more accurate service for the user in interaction.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to data processing methods, apparatuses, devices, and computer-readable storage media. Background Art

[0002] With the development of information technology, various terminal devices can provide various services to people in work and life. For example, applications that provide services can be deployed in terminal devices. Terminal devices or their applications can use machine learning models to provide services to assist users in using terminal devices or applications to complete tasks. If such machine learning models should not be updated once deployed, it may not be possible to provide satisfactory services to users. Therefore, it is expected that machine learning models can be updated. Summary of the invention

[0003] In a first aspect of the present disclosure, a data processing method is provided. The method comprises: obtaining user feedback information for a target model based on user interactions associated with the target model, the user feedback information comprising corresponding user feedback for multiple model outputs, the multiple model outputs being provided by the target model for the same user input; generating training data based at least on the user feedback information, the training data comprising quality indicators for evaluating the multiple model outputs; and training a reward model for updating the target model using the training data.

[0004] In a second aspect of the present disclosure, a device for data processing is provided. The device includes: a user feedback module configured to obtain user feedback information for a target model based on user interactions associated with the target model, the user feedback information including corresponding user feedback for multiple model outputs, and the multiple model outputs are provided by the target model for the same user input; a data generation module configured to generate training data based at least on the user feedback information, the training data including quality indicators for evaluating the multiple model outputs; and a model training module configured to train a reward model for updating the target model using the training data.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A schematic diagram showing an example of updating a target model using a reward model according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram showing an example architecture of a training reward model according to some embodiments of the present disclosure;

[0012] Figure 4 A flowchart showing a process of data processing according to some embodiments of the present disclosure;

[0013] Figure 5 A block diagram showing an apparatus for data processing according to some embodiments of the present disclosure; and

[0014] Figure 6 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0016] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0017] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0019] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0022] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0023] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0024] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", which are used interchangeably in this article. A model may also include different types of processing units or networks.

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, a terminal device 130 can provide various services to a user 140. For example, an application 160 can be installed in the terminal device 130 to provide services to the user 140. The user 140 can interact with the application 160 via the terminal device 110 and / or an attached device of the terminal device 110. In some embodiments, the user 140 can interact by inputting a natural language.

[0026] In some embodiments, the terminal device 130 communicates with the server 150 to provide services. The terminal device 130 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 130 can also support any type of interface for the user (such as "wearable" circuits, etc.). The server 150 can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0027] In order to provide services, the server 150 may be deployed with a target model 110, which is a machine learning model. The target model 110 may give a model output according to user input. The model output is presented to the user 140 via the terminal device 130. For example, the user input may be a question, the model output may be an answer to the question, and so on. In some embodiments, the user input may include natural language, and accordingly, the target model 110 may be a model suitable for processing natural language. Of course, in such an embodiment, the target model 110 may also have the ability to process other types of data.

[0028] It should be understood that the structure and function of each element in the environment 100 are described for exemplary purposes only, and do not imply any limitation on the scope of the present disclosure. For example, the embodiments of the present disclosure may not be limited to applications, but may be applied to any scenario requiring a machine learning model.

[0029] In some embodiments, the training of the target model 110 may include multiple stages, such as a pre-training stage, a supervised fine-tuning (SFT) stage, a reward model training, and a reinforcement learning based on human feedback (RLHF) stage. In the RLHF stage, the reward model 120 may be used to update the target model 110. Figure 2 An example architecture of RLHF is described.

[0030] like Figure 2 As shown, in RLHF, a frozen reward model 221 and a Critic model 222 for generating evaluations (Critic) can be obtained based on the reward model 120. Accordingly, a frozen target model 212 and an Actor model 211 as an actor (Actor) can be obtained based on the target model 110. In addition, pre-trained data 230 of the target model 211 and data 240 generated using the Actor model 211 are also required. Further, according to the proximal policy optimization (PPO) algorithm, the target model 110 is updated based on the frozen reward model 221, the Critic model 222, the frozen target model 212, the Actor model 211, the pre-trained data 230 and the data 240.

[0031] It should be understood that reference Figure 2 The described RLHF process for the target model is exemplary only and is not intended to be limiting in any way.

[0032] As briefly mentioned above, it is desirable to be able to update the model after it is deployed online. However, such models that provide services to users are usually complex, such as complex network structures, numerous network parameters, etc. This makes it difficult to train and update such models online, but if such models are only trained offline, it will not be able to support real-time iteration of online application scenarios.

[0033] The inventors of the present application realize that, in actual training, the essence of RLHF is to align the target model with the reward model. In this case, the training signal received by the reward model is the direction to which the target model is ultimately trained.

[0034] In view of this, an embodiment of the present disclosure provides a data processing scheme, in particular, a scheme for processing user interaction data to construct training data. For example, based on user interactions associated with a target model, user feedback information for a target model is obtained. The user feedback information includes corresponding feedback from the user for multiple model outputs, which are provided by the target model for the same user input. Based at least on the user feedback information, training data is generated, and the training data includes quality indicators for evaluating these model outputs. Using the training data, a reward model is trained, and the trained reward model can be used to update the target model.

[0035] In the embodiments of the present disclosure, at least the user's online interaction data is used to construct training data for the reward model. This enables real-time updates of the reward model, and the real-time updates of the reward model can be felt by the target model. In this way, the target model is updated using online data, which is conducive to the target model providing more accurate services to users in the interaction.

[0036] The following describes example embodiments of the present disclosure with continued reference to the accompanying drawings.

[0037] Figure 3 Schematic diagram of an example architecture 300 for training a reward model according to some embodiments of the present disclosure is shown. Figure 3 As shown, during the online deployment of the target model 110, for the same user input, the target model 110 can provide multiple model outputs 320-1, 320-2, ..., 320-N, which are also individually or collectively referred to as model outputs 320, where N is a positive integer greater than 1. Exemplarily, one or more users 140 perform user interaction through a terminal device. During the interaction, the user 140 can provide user input. Accordingly, the target model 110 generates a model output 320. For example, the user 140 can input a question in natural language, and the target model 110 can generate a corresponding answer as a model output.

[0038] In the embodiments of the present disclosure, the same user input does not mean input with exactly the same expression form. For example, user input for natural language does not mean that the input is exactly the same in words, they can be inputs expressing the same semantics. In this case, multiple model outputs 320 can include various suitable sources. For example, multiple model outputs 320 can be generated for the same user input from different users. For another example, multiple model outputs 320 can be generated for the same user input at different times. For another example, multiple model outputs 320 can be a combination of the above examples.

[0039] User feedback information 330 for the target model can be obtained based on user interactions associated with the target model 110, which includes corresponding user feedback for multiple model outputs 320. In some embodiments, during the interaction through the terminal device, the user 140 can give explicit feedback on the model output 320. Such feedback can be positive (e.g., like) or negative (e.g., step down). In some embodiments, user feedback can be implicitly determined based on the subsequent interaction of the user 140 with the model output 320. For example, if the user 140 ignores the model output in subsequent interactions, it can be regarded as the user 140 giving negative feedback; if the user 140 adopts the model output in subsequent interactions, it can be regarded as the user 140 giving positive feedback. In some embodiments, the user 140 can be prompted to score the model output 140, etc.

[0040] Continuing with the architecture 300, training data 350 is generated based on at least the user feedback information 330. The training data 350 includes quality metrics for evaluating the model outputs 320. For example, the training data 350 may include user input, a plurality of model outputs 320, and corresponding quality metrics for the model outputs 320. The training data 350 may be structured in any suitable manner.

[0041] In some embodiments, the quality indicator may be a numerical indicator, such as a corresponding score of the model outputs 320. In such embodiments, the training data 350 may be referred to as pointwise data.

[0042] In some embodiments, the quality indicator may be an overall relative indicator of the model outputs 320, such as a ranking of the model outputs 320. In such embodiments, the training data 350 may be referred to as listwise data.

[0043] In some embodiments, the quality indicator can be a relative indicator of paired model outputs, for example, a relative ranking of a pair of model outputs. In such embodiments, the training data 350 can be referred to as pairwise data.

[0044] Real-time update of the reward model can be achieved by using online user feedback. It should be understood that the term "real-time update" used herein may have different time granularities depending on various factors (e.g., number of users, activity, etc.), and the embodiments of the present disclosure are not limited in this respect.

[0045] In some embodiments, feedback from other sources besides user feedback may be additionally considered to generate training data. As an example, reference feedback associated with the purpose of use of the target model 110 may be one source. Accordingly, reference feedback information for the target model 110 may be obtained. The reference feedback information may include corresponding reference feedback for a plurality of model outputs 320, each reference feedback being associated with the purpose of use of the target model 110. Similar to user feedback, reference feedback may also include ratings, overall ranking results, or pairwise comparison results, etc.

[0046] The purpose of use may depend on the application scenario of the target model 110, for example. For example, in the case where the target model 110 is applied to a popular science scenario, it is more appropriate for the model output to be popular science content or have a popular science style. The source of the reference feedback may be, for example, a developer or designer of an application or product using the target model 110. In this way, the reward model and thus the target model can be evolved toward the development or design goal. In addition, by using reference feedback, the training of the reward model and thus the target model can be appropriate.

[0047] Depending on the requirements for the training data 350 and / or the specific feedback included in the feedback information, any suitable method may be used to integrate the user feedback and the reference feedback to generate the training data 350 .

[0048] In some embodiments, the training data 350 may include point-by-point data. Exemplarily, corresponding user scores for multiple model outputs 320 may be determined based on the user feedback information 330. For example, the positive or negative feedback given by the user may be converted into a user score according to a preset rule. Accordingly, corresponding reference scores for multiple model outputs 320 may be determined based on the reference feedback information 340. For example, the reference feedback information 340 may include a score of the model output by a designer or developer, or the feedback on the model output may be converted into a reference score according to a preset rule. Furthermore, corresponding target scores for multiple model outputs 320 may be determined as at least part of the quality indicator based on the corresponding user scores and the corresponding reference scores. For example, the target score for a certain model output 320 may be an average or weighted average of the user score and the reference score for the model output.

[0049] In some embodiments, the training data 350 may include tabular data. Exemplarily, a user ranking result for a plurality of model outputs 320 may be determined based on the user feedback information 330. For example, the model outputs may be ranked according to the degree of positivity or negativity of the user feedback. Accordingly, a reference ranking result for a plurality of model outputs 320 may be determined based on the reference feedback information 340. For example, the reference feedback information 340 may include the ranking result of the model output 320 by the developer or designer. Further, a target ranking result for a plurality of model outputs 320 may be determined as at least a part of the quality indicator based on the user ranking result and the reference ranking result. For example, one ranking result may be adjusted according to another ranking result to obtain a target ranking result.

[0050] In some embodiments, the training data 350 may include paired data. Exemplarily, a user comparison result for the quality of a pair of model outputs in the plurality of model outputs 320 may be determined based on the user feedback information 330. For example, it is possible to determine which model output of the pair of model outputs has a higher quality, that is, to sort the pair of model outputs, based on whether the user feedback is positive or negative and the degree of positive or negative. Accordingly, a reference comparison result for the quality of the pair of model outputs may be determined based on the reference feedback information. For example, a developer or designer may sort the pair of model outputs. Further, a target comparison result for the pair of model outputs may be determined as at least a part of the quality indicator based on the user comparison result and the reference comparison result.

[0051] Continuing with the architecture 300, the reward model 120 is trained using the training data 350. Taking paired data as an example, an example training of the reward model 120 is described. In a pair of compared model outputs, the model output with a higher ranking in the target comparison result can be called a winning model output, and the other model output can be called a losing model output.

[0052] The reward model score for the winning model output can be expressed as follows:

[0053] sw=rθ (x, yw) (1)

[0054] where sw represents the reward model score output by the winning model, rθ represents the value network, x represents the input (e.g., question), and yw represents the winning model output.

[0055] The reward model score for the failed model output can be expressed as follows:

[0056] sl=rθ (x, yl) (2)

[0057] Where sl represents the reward model score of the failed model output and yl represents the failed model output.

[0058] Accordingly, the total loss function can be calculated as follows:

[0059] -log(σ(sw-sl))(3)where σ represents the sigmoid function.

[0060] The above is merely an example of reward model 120 training and is not intended to be limiting.

[0061] The trained reward model 120 can be used to update the target model 110. In some embodiments, the reward model 120 can be trained with the training data 350 generated by the above architecture 300 in different time periods. Thus, different versions of the reward model 120 corresponding to these time periods can be obtained and stored. The version used to update the target model 110 can be selected from the stored versions of the reward model 120.

[0062] If some events (e.g., being hacked, etc.) occur during the online use of the target model 110, resulting in abnormal user interactions, then inappropriate training data will be generated. This also results in the reward model being inappropriate. By storing different versions of the reward model, if an event that causes abnormal user interactions occurs, the reward model stored before the event occurs can be used to update the target model. This helps to ensure the reliability of the target model.

[0063] Figure 4 FIG. 4 is a flowchart showing a process 400 of data processing according to some embodiments of the present disclosure. The process 400 may be implemented, for example, at the server 150. Figure 1 Process 400 is described.

[0064] In block 410, the server 150 obtains user feedback information for the target model based on user interactions associated with the target model. The user feedback information includes corresponding user feedback for multiple model outputs, where the multiple model outputs are provided by the target model for the same user input.

[0065] At block 420, the server 150 generates training data based at least on the user feedback information. The training data includes quality indicators for evaluating a plurality of model outputs.

[0066] In block 430 , the server 150 trains a reward model for updating the target model using the training data.

[0067] In some embodiments, process 400 also includes: obtaining reference feedback information for the target model, the reference feedback information includes corresponding reference feedback for multiple model outputs, the corresponding reference feedback is associated with the purpose of use of the target model, and wherein generating training data includes: generating training data based on user feedback information and reference feedback information.

[0068] In some embodiments, generating training data based on user feedback information and reference feedback information includes: determining corresponding user scores for multiple model outputs based on the user feedback information; determining corresponding reference scores for multiple model outputs based on the reference feedback information; and determining corresponding target scores for multiple model outputs as at least part of the quality indicator based on the corresponding user scores and the corresponding reference scores.

[0069] In some embodiments, generating training data based on user feedback information and reference feedback information includes: determining user ranking results for multiple model outputs based on the user feedback information; determining reference ranking results for multiple model outputs based on the reference feedback information; and determining target ranking results for multiple model outputs as at least part of the quality indicator based on the user ranking results and the reference ranking results.

[0070] In some embodiments, generating training data based on user feedback information and reference feedback information includes: determining a user comparison result of the quality of a pair of model outputs among multiple model outputs based on the user feedback information; determining a reference comparison result of the quality of a pair of model outputs based on the reference feedback information; and determining a target comparison result for a pair of model outputs as at least a part of the quality indicator based on the user comparison result and the reference comparison result.

[0071] In some embodiments, the user interaction occurs in a first time period, and process 400 also includes: storing a reward model trained using the training data as a first version of the reward model; generating additional training data to train the reward model based on additional user interactions with the target model occurring in a second time period; storing the reward model trained using the additional training data as a second version of the reward model; and selecting a version for updating the target model from the first version of the reward model and the second version of the reward model.

[0072] In some embodiments, the user input includes input in natural language, and the target model is adapted to process natural language.

[0073] Figure 5A schematic structural block diagram of an apparatus 500 for data processing according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the server 150. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware or any combination thereof.

[0074] As shown in the figure, the device 500 includes a user feedback module 510, which is configured to obtain user feedback information for the target model based on user interactions associated with the target model, and the user feedback information includes corresponding user feedback for multiple model outputs, and the multiple model outputs are provided by the target model for the same user input. The device 500 also includes a data generation module 520, which is configured to generate training data based on at least the user feedback information, and the training data includes a quality indicator for evaluating the outputs of multiple models. The device 500 also includes a model training module 530, which is configured to use the training data to train a reward model for updating the target model.

[0075] In some embodiments, the device 500 also includes: a reference feedback module, configured to obtain reference feedback information for the target model, the reference feedback information includes corresponding reference feedback for multiple model outputs, the corresponding reference feedback is associated with the purpose of use of the target model, and wherein the data generation module 520 is further configured to generate training data based on user feedback information and reference feedback information.

[0076] In some embodiments, the data generation module 520 is further configured to: determine corresponding user scores for multiple model outputs based on user feedback information; determine corresponding reference scores for multiple model outputs based on reference feedback information; and determine corresponding target scores for multiple model outputs as at least part of the quality indicator based on the corresponding user scores and the corresponding reference scores.

[0077] In some embodiments, the data generation module 520 is further configured to: determine user ranking results for multiple model outputs based on user feedback information; determine reference ranking results for multiple model outputs based on reference feedback information; and determine target ranking results for multiple model outputs as at least part of the quality indicator based on the user ranking results and the reference ranking results.

[0078] In some embodiments, the data generation module 520 is further configured to: determine a user comparison result of the quality of a pair of model outputs among multiple model outputs based on user feedback information; determine a reference comparison result of the quality of a pair of model outputs based on reference feedback information; and determine a target comparison result for a pair of model outputs as at least part of the quality indicator based on the user comparison result and the reference comparison result.

[0079] In some embodiments, the user interaction occurs in a first time period, and the apparatus 500 further includes: a first storage module configured to store a reward model trained using the training data as a first version of the reward model; another training module configured to generate additional training data to train the reward model based on additional user interactions with the target model occurring in a second time period; a second storage module configured to store the reward model trained using the additional training data as a second version of the reward model; and a version selection module configured to select a version for updating the target model from the first version of the reward model and the second version of the reward model.

[0080] In some embodiments, the user input includes input in natural language, and the target model is adapted to process natural language.

[0081] Figure 6 6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to implement Figure 1 Server 150.

[0082] like Figure 6 As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0083] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 600.

[0084] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0085] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0086] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0087] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0088] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0091] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0092] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, comprising: Based on user interactions associated with the target model, obtaining user feedback information for the target model, the user feedback information comprising corresponding user feedback for a plurality of model outputs, the plurality of model outputs being provided by the target model for the same user input; generating training data based at least on the user feedback information, the training data comprising quality indicators for evaluating the outputs of the multiple models; as well as Using the training data, a reward model for updating the target model is trained.

2. The method according to claim 1, further comprising: obtaining reference feedback information for the target model, the reference feedback information comprising corresponding reference feedback for the plurality of model outputs, the corresponding reference feedback being associated with a purpose of use of the target model, and Generating the training data comprises: The training data is generated based on the user feedback information and the reference feedback information.

3. The method according to claim 2, wherein generating the training data based on the user feedback information and the reference feedback information comprises: Based on the user feedback information, determining corresponding user scores for the multiple model outputs; Based on the reference feedback information, determining corresponding reference scores for the multiple model outputs; as well as Based on the corresponding user scores and the corresponding reference scores, corresponding target scores for the plurality of model outputs are determined as at least a part of the quality indicator.

4. The method according to claim 2, wherein generating the training data based on the user feedback information and the reference feedback information comprises: Based on the user feedback information, determining user ranking results for the multiple model outputs; Based on the reference feedback information, determining reference ranking results for the multiple model outputs; as well as Based on the user ranking results and the reference ranking results, target ranking results for the multiple model outputs are determined as at least a part of the quality indicator.

5. The method according to claim 2, wherein generating the training data based on the user feedback information and the reference feedback information comprises: determining, based on the user feedback information, a user comparison result of the quality of a pair of model outputs among the plurality of model outputs; Determining a reference comparison result for the quality of the pair of model outputs based on the reference feedback information; as well as Based on the user comparison result and the reference comparison result, a target comparison result for the pair of model outputs is determined as at least a part of the quality indicator.

6. The method of claim 1, wherein the user interaction occurs in a first time period, and the method further comprises: storing a reward model trained using the training data as a first version of the reward model; generating additional training data to train the reward model based on additional user interactions with the target model occurring during a second time period; storing the reward model trained using the additional training data as a second version of the reward model; as well as A version for updating the target model is selected from the first version of the reward model and the second version of the reward model.

7. The method of claim 1, wherein the user input comprises input in a natural language, and the target model is adapted to process natural language.

8. A device for data processing, comprising: A user feedback module is configured to obtain user feedback information for the target model based on user interactions associated with the target model, wherein the user feedback information includes corresponding user feedback for multiple model outputs, where the multiple model outputs are provided by the target model for the same user input; A data generation module, configured to generate training data based at least on the user feedback information, wherein the training data includes quality indicators for evaluating the outputs of the multiple models; as well as The model training module is configured to use the training data to train a reward model for updating the target model.

9. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 7 when executed by the at least one processing unit.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images