Question and answer information processing method and device, model training method and device, electronic equipment and medium

Abstract

The invention provides a question and answer information processing method, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models, intelligent question and answer and the like. According to the specific implementation scheme, at least one piece of initial answer information is generated according to question information provided by an object; at least one piece of feedback information corresponding to the at least one piece of initial answer information is obtained, and the feedback information is used for indicating the preference degree of the object to the initial answer information; and generating a training sample according to the question information, the at least one piece of initial answer information and the at least one piece of feedback information. The invention further provides a dialogue model training method and device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning, large models, intelligent question and answer, etc. More specifically, the present disclosure provides a question and answer information processing method, a model training method, a device, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of conversational models are increasing. Summary of the invention

[0003] The present invention provides a question-answering information processing method, a conversational model training method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a method for processing question and answer information is provided, the method comprising: generating at least one initial answer information based on question information provided by an object; acquiring at least one feedback information corresponding to the at least one initial answer information, wherein the feedback information is used to indicate the object's preference for the initial answer information; and generating a training sample based on the question information, the at least one initial answer information and the at least one feedback information.

[0005] According to another aspect of the present disclosure, a method for training a conversational model is provided, the device comprising: adjusting parameters of the conversational model according to at least one training sample, so that the conversational model generates adjusted answer information according to question information in the training sample, and the adjusted answer information is close to answer information with a high degree of preference in the training sample and away from answer information with a low degree of preference in the training sample, wherein the training sample is generated by the following operations: generating at least one initial answer information according to question information provided by an object; acquiring at least one feedback information corresponding to the at least one initial answer information, wherein the feedback information is used to indicate the object's preference for the initial answer information; and generating the training sample according to the question information, at least one initial answer information and at least one feedback information.

[0006] According to another aspect of the present disclosure, a question and answer information processing device is provided, which includes: a first generating module, used to generate at least one initial answer information according to question information provided by an object; an acquiring module, used to acquire at least one feedback information corresponding to the at least one initial answer information, wherein the feedback information is used to indicate the object's preference for the initial answer information; and a second generating module, used to generate a training sample according to the question information, at least one initial answer information and at least one feedback information.

[0007] According to another aspect of the present disclosure, a training device for a conversational model is provided, the device comprising: an adjustment module, for adjusting parameters of the conversational model according to at least one training sample, so that the conversational model generates adjusted answer information according to question information in the training sample, and the adjusted answer information is close to answer information with a high degree of preference in the training sample and away from answer information with a low degree of preference in the training sample, wherein the training sample is generated by the following modules: a first generation module, for generating at least one initial answer information according to question information provided by an object; an acquisition module, for acquiring at least one feedback information corresponding to the at least one initial answer information, wherein the feedback information is used to indicate the degree of preference of the object for the initial answer information; and a second generation module, for generating a training sample according to question information, at least one initial answer information and at least one feedback information.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram of an exemplary system architecture to which the question-answer information processing method and apparatus can be applied according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of a question and answer information processing method according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of a question-and-answer information processing method according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic flow chart of a method for training a conversational model according to another embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of a model training method according to an embodiment of the present disclosure;

[0018] Figure 6 A block diagram of a question and answer information processing device according to an embodiment of the present disclosure;

[0019] Figure 7 is a block diagram of a training device for a conversational model according to another embodiment of the present disclosure; and

[0020] Figure 8 It is a block diagram of an electronic device to which a question-and-answer information processing method and / or a conversational model training method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Large models can include large language models (LLM), large image models, large video models, etc. Taking the large language model as an example, a large-scale unsupervised corpus can be used for pre-training, and then supervised fine-tuning (SFT) training can be performed using supervised fine-tuning corpora carefully annotated by experts in related fields. Next, alignment training can also be performed using alignment algorithms such as Kahneman-Tversky Optimization (KTO), Direct Preference Optimization (DPO), Simple Preference Optimization (simPO), and Proximal Policy Optimization (PPO). The upper bound of model performance depends on the quantity and quality of the supervised fine-tuning corpus, as well as the effect of the reward model in the alignment training phase.

[0023] In some embodiments, the effect of the model is highly related to the quantity and quality of the training samples. For most models, the corpus can be annotated by experts in related fields to continuously improve the quality of the corpus. However, the cost of annotation is high and the efficiency is low.

[0024] In some embodiments, the data mining process can be customized for the target application scenario to continuously mine questions (queries) that are difficult for the model to answer with high quality in the target application scenario, thereby improving the annotation efficiency of experts in related fields. However, in the mining process, the user's behavior signals are not fully utilized, and the direction of improving the model effect is difficult to be consistent with the direction of improving the user experience.

[0025] Therefore, in order to fully improve the performance of the model and user experience, the present disclosure provides a question and answer information processing method, which will be described below.

[0026] Figure 1 is a schematic diagram of an exemplary system architecture to which the question-answer information processing method and device can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0029] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0030] It should be noted that the question and answer information processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the question and answer information processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The question and answer information processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the question and answer information processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0031] It can be understood that the above describes the system architecture of the present disclosure, and the following will describe the method of the present disclosure.

[0032] Figure 2 It is a flowchart of a question and answer information processing method according to an embodiment of the present disclosure.

[0033] like Figure 2 As shown, the method 200 may include operations S210 to S230.

[0034] In operation S210, at least one initial answer information is generated according to question information provided by the subject.

[0035] In the embodiment of the present disclosure, the object may include a user. The question information may include a question (query) input by the user. For example, the question information may be information in various forms such as text, image, audio, video, etc.

[0036] In the embodiments of the present disclosure, the initial answer information may be generated in various ways. For example, the initial answer information corresponding to the question information may be determined using a preset mapping relationship. The initial answer information corresponding to the question information may also be determined using a conversational model. The initial answer information may also be in various forms of information such as text, image, audio, video, etc.

[0037] In operation S220, at least one piece of feedback information corresponding to at least one piece of initial answer information is acquired.

[0038] In the disclosed embodiment, the feedback information may indicate the object's preference for the initial answer information. For example, the feedback information may be provided by the user. It is understood that after the user authorizes, the question information and feedback information provided by the user may be obtained.

[0039] In operation S230, a training sample is generated according to the question information, the at least one initial answer information, and the at least one feedback information.

[0040] In the embodiment of the present disclosure, a multi-tuple group of data may be determined as a training sample based on question information, at least one initial answer information, and at least one feedback information.

[0041] Through the disclosed embodiments, feedback information provided by the subject for the initial answer information can be obtained, the dependence on experts in related fields is reduced, and the annotation cost of experts can be greatly reduced. By obtaining feedback information, more realistic user preferences can be obtained. The training samples generated based on question information, initial answer information, and feedback information can make the generated answer information closer to the user's real preferences, which can fully improve the user experience.

[0042] It can be understood that the above describes the method of the present disclosure, and some ways of generating initial answer information of the present disclosure will be described below.

[0043] In some embodiments, in some implementations of the above operation S210, at least one initial answer information is generated using a conversational model based on the question information provided by the object.

[0044] For example, a conversational model can be a large model or a lightweight model that is applied to the target application scenario.

[0045] For example, based on the audio type question information provided by the user, the conversational model can perform audio recognition to determine the semantics of the question information. Next, the conversational model can generate initial answer information of the audio type or initial answer information of a type corresponding to the semantics. For another example, based on the text type question information provided by the user, the conversational model can also determine the semantics of the question information to determine the corresponding initial answer information. It can be understood that the way in which the conversational model generates initial answer information based on image type and video type question information is the same or similar to the way in which the initial answer information is generated based on audio type question information, and the present disclosure will not go into details here. Through the embodiments of the present disclosure, the conversational model can be used to more accurately determine the answer information and improve the user experience.

[0046] It can be understood that the above describes some ways of generating initial answer information of the present disclosure, and the following describes the feedback information of the present disclosure.

[0047] In some embodiments, the feedback information may include a preference level value.

[0048] In an embodiment of the present disclosure, the preference value is determined based on one of multiple visual controls triggered by a user, the multiple visual controls include a first visual control and a second visual control, and the preference value corresponding to the first visual control is higher than the preference value corresponding to the second visual control.

[0049] In some embodiments, in some implementations of the above operation S220 and the above operation S230, a preference value corresponding to the initial answer information is obtained; and a training sample is generated according to the question information, the initial answer information and the feedback information.

[0050] For example, a user may provide question information to a conversational model. The conversational model may generate initial answer information corresponding to the question information. If the user believes that the initial answer information is highly correlated with the question information, the first visual control may be triggered to determine the preference value of the initial answer information as the first preference value. Thus, a triplet of data may be generated as a training sample. The triplet of data may include question information, initial answer information, and a first preference value. It is understandable that the first visual control may be a "like" control. The first preference value may be, for example, 1.

[0051] For another example, a user may provide question information to a conversational model. The conversational model may generate initial answer information corresponding to the question information. If the user believes that the initial answer information has a low correlation with the question information, the second visual control may be triggered to determine the preference value of the initial answer information as the second preference value. Thus, a triplet of data may be generated as a training sample. The triplet of data may include question information, initial answer information, and a second preference value. It is understandable that the second visual control may be a "click" control. The second preference value may be, for example, 0.

[0052] For another example, triple data can be represented as<query,response,signal> . response may be initial answer information. signal may be feedback signal information, corresponding to the first preference level value or the second preference level value.

[0053] It can be understood that the triple data including question information, initial answer information and preference value may be pointwise preference data.

[0054] It can be understood that the above description uses the first preference value or the second preference value as an example of feedback information to explain the present disclosure. However, the present disclosure is not limited thereto, and the feedback information may also be a preference evaluation value provided by the object. For example, the preference evaluation value may range from 0 to 100, and the higher the preference evaluation value, the more the initial answer information conforms to the user's true preference.

[0055] It can be understood that the above describes the method for generating point-by-point preference data, and another type of preference data will be described below.

[0056] In some embodiments, in other implementations of the above method 200, multiple initial answer information may be generated based on the question information provided by the subject. Multiple preference values ​​corresponding to the multiple initial answer information are obtained. Training samples may be generated based on the question information, the multiple initial answer information, and the multiple preference values.

[0057] For example, a user may provide question information to a conversational model. The conversational model may generate multiple initial answer information. Alternatively, the user may provide the same question information to the conversational model multiple times to obtain multiple initial answer information. For multiple initial answer information, the user may determine multiple preference evaluation values ​​respectively. Multiple preference evaluation values ​​may indicate the user's preference for different initial answer information. Multiple preference evaluation values ​​may serve as feedback signal information. Thus, taking the example that multiple initial answer information is N, N-tuple data may be generated as a training sample. N-tuple data may include question information, N initial answer information, and feedback signal information. For another example, N-tuple data may be represented as<query,response1, ……,responseN,signal> response1, ..., responseN may be N initial response information. signal may be feedback signal information corresponding to N preference evaluation values. N may be an integer greater than 1.

[0058] It can be understood that the N-tuple data may be pairwise preference data.

[0059] It can be understood that the feedback information above can be generated during the conversation between the object and the conversational model, and can be used as conversational feedback information. In relevant application scenarios, the number of users of the conversational model is huge. The user's usage behavior (providing feedback) plays an important role in improving the performance of the model. Through the embodiments of the present disclosure, the evaluation results of the user on the answer information generated by the conversational model can be obtained to obtain the feedback signal of the user in the real scene, which can effectively improve the performance of the conversational model.

[0060] It can be understood that the above description uses the feedback information being the preference value as an example to explain the present disclosure, but the present disclosure is not limited thereto, and the feedback information may also be the target answer information provided by the object, which will be described below.

[0061] In some embodiments, the target application scenario may be an AI-assisted creation scenario. For example, the target application scenario may be a novel-assisted writing scenario, an intelligent coding scenario, or an intelligent presentation creation scenario. Based on the question information provided by the user, the conversational model may recommend one or more content. The user may directly adopt the recommended content without editing the content, or may not adopt the recommended content or partially adopt the recommended content. The content may be used as initial answer information.

[0062] In the disclosed embodiment, obtaining at least one feedback information corresponding to at least one initial answer information includes: obtaining target answer information provided by the object, corresponding to the question information, and having a correlation index value with the initial answer information lower than a preset correlation threshold. For example, if the user believes that the quality of the recommended content is low, the user may not adopt the recommended content and write the target content on his own. The correlation between the recommended content and the target content is low and may be lower than the preset correlation threshold. The target content may serve as the target answer information corresponding to the question information.

[0063] In the disclosed embodiment, obtaining at least one feedback information corresponding to at least one initial answer information includes: obtaining target answer information provided by the object and obtained by editing the initial answer information. For example, if the user believes that the recommended content can be partially adopted, the user can edit the recommended content to obtain the target content. The target content can be used as the target answer information.

[0064] In the embodiment of the present disclosure, a triplet of data can be generated based on the question information, the initial answer information and the target answer information as a training sample. For example, the triplet of data can be represented as<query,response_before,response_after> . response_before can be the initial answer information. response_after can be the target answer information. The triplet data includes 2 answer information and can also be used as pair-wise preference data.

[0065] It is understandable that when the user directly adopts the recommended content without editing the content, it is difficult to generate pair-wise preference data based on the content. However, the preference level value of the content can be determined as the first preference level value to generate point-wise preference data.

[0066] It can be understood that the feedback information above is the target answer information generated by object editing, which can be used as edit feedback information. In relevant application scenarios, users can modify and polish the results output by the model. Through the embodiment of the present disclosure, the results modified and polished by the user are obtained as the target answer information, and very high-quality user preference data can be obtained, which can fully improve the performance of the conversational model.

[0067] It can be understood that the above description uses the user as an example to explain the present disclosure, but the present disclosure is not limited thereto, and the object may also be a large model, which will be described below.

[0068] In the embodiments of the present disclosure, the feedback information may be generated by the large model according to the preset feedback rules and the initial answer information. For example, the conversational model may be a model on the terminal side, which is smaller in scale. The large model is larger in scale, has better generation capabilities, and also has evaluation capabilities. Based on the preset feedback rules, the large model may be used to determine the preference value as feedback information.

[0069] In the disclosed embodiment, the preset feedback rule may be determined according to the user's preference. For example, the preset feedback rule may include the number of words corresponding to the answer information, the relevance to the question information, etc. The specific values ​​of the number of words and the relevance may be determined based on the user's preference.

[0070] For another example, a user may provide question information to a conversational model. The conversational model may generate multiple initial answer information. Alternatively, the user may provide the same question information to the conversational model multiple times to obtain multiple initial answer information. For multiple initial answer information, based on preset feedback rules, the large model may determine multiple preference evaluation values ​​respectively. Multiple preference evaluation values ​​may also indicate the user's preference for different initial answer information. Multiple preference values ​​may serve as feedback signal information. Thus, taking the example that multiple initial answer information is N, N-tuple data may be generated as a training sample. N-tuple data may include question information, N initial answer information, and feedback signal information. For another example, N-tuple data may be represented as<query,response1, ……,responseN,signal’> response1, ..., responseN may be N initial response information. signal' may be feedback signal information, corresponding to N preference evaluation values ​​generated by the large model.

[0071] It can be understood that the above description uses the large model to determine the preference value as an example to illustrate the present disclosure. However, the present disclosure is not limited thereto. In some embodiments, the large model may also generate target answer information. The target answer information generated by the large model may also be used as feedback information.

[0072] It can be understood that the above description uses the example of a large model determining feedback information to explain the present disclosure, but the present disclosure is not limited thereto, and the large model can also provide question information to the conversational model.

[0073] It can be understood that the feedback information above is determined by the big model and can be used as AI feedback information. Through the embodiments of the present disclosure, the evaluation ability of the big model can be fully utilized to improve the richness of the feedback information source, which helps to further improve the performance of the conversational model. Figure 3 The various feedback information disclosed in the present invention is further explained.

[0074] Figure 3 It is a schematic diagram of a question and answer information processing method according to an embodiment of the present disclosure.

[0075] like Figure 3 As shown, the model can be pre-trained to obtain a pre-trained model Mpret30. The pre-trained model Mpret30 can be fine-tuned by supervision to obtain a fine-tuned model Mst30. The fine-tuned model Mst30 can be used as a conversational model Mchat30. According to the question information provided by the object, the conversational model Mchat30 can generate initial answer information. Based on the initial answer information, conversational feedback information CF30, editing feedback information EF30 and artificial intelligence feedback information AF30 can be obtained. Based on the conversational feedback information CF30, point-by-point preference data pointd30 can be obtained. Based on the conversational feedback information CF30, editing feedback information EF30 and artificial intelligence feedback information AF30, pair-by-pair preference data paird30 can be obtained. The point-by-point preference data pointd30 and the pair-by-pair preference data paird30 can be used as training samples.

[0076] It can be understood that the above describes the method of generating training samples, and the following will describe some methods of model training.

[0077] Figure 4 is a schematic flowchart of a method for training a conversational model according to another embodiment of the present disclosure.

[0078] like Figure 4 As shown, the method 400 may include operation S440.

[0079] In operation S440, parameters of the conversational model are adjusted based on at least one training sample, so that the conversational model generates adjusted answer information based on the question information in the training sample.

[0080] In the disclosed embodiment, alignment training may be performed on the conversational model to adjust parameters of the conversational model.

[0081] In the embodiment of the present disclosure, the adjusted answer information is close to the answer information with a high degree of preference in the training samples and away from the answer information with a low degree of preference in the training samples.

[0082] In the embodiment of the present disclosure, the training sample is generated by the following operations: generating at least one initial answer information according to the question information provided by the subject. Obtaining at least one feedback information corresponding to the at least one initial answer information. The feedback information may indicate the subject's preference for the initial answer information. Generating the training sample according to the question information, the at least one initial answer information and the at least one feedback information. For example, the training sample may be generated by the above method 300.

[0083] Through the disclosed embodiments, the feedback information in the training samples can be at least one of the above-mentioned conversational feedback information, editing feedback information and artificial intelligence feedback information, providing different types of feedback signals for model training. These feedback signals are highly correlated with the user's preferences in real scenarios, and can align the model performance with the user's real preferences, effectively improving the performance ceiling of the model and making the model evolve in the direction of the user's real preferences. In addition, it can also reduce the sample annotation cost and improve the efficiency of obtaining training samples.

[0084] It can be understood that the model training method disclosed in the above text is explained, and the model training method disclosed in the present invention will be further explained below in combination with the above-mentioned various feedback information.

[0085] Figure 5 It is a schematic diagram of a model training method according to an embodiment of the present disclosure.

[0086] like Figure 5As shown, the model can be pre-trained to obtain a pre-trained model Mpret50. The pre-trained conversational model Mpret50 can be fine-tuned through supervised fine-tuning to obtain a fine-tuned model Msft50. The fine-tuned model Msft50 can be used as a conversational model Mchat50. According to the question information provided by the object, the conversational model Mchat50 can generate initial answer information. Based on the initial answer information, conversational counter-information CF50, editing feedback information EF50 and artificial intelligence feedback information AF50 can be obtained. Based on the conversational feedback information CF50, point-by-point preference data pointd50 can be obtained. Based on the conversational feedback information CF50, editing feedback information EF50 and artificial intelligence feedback information AF50, pair-by-pair preference data paird50 can be obtained. The point-by-point preference data pointd50 and the pair-by-pair preference data paird50 can be used as training samples.

[0087] In the disclosed embodiment, the feedback information includes a preference value. According to at least one training sample, adjusting the parameters of the conversational model includes: adjusting the parameters of the conversational model according to the question information, the initial answer information and the preference value of the training sample. The parameters of the conversational model may be adjusted based on the Kahneman-Tversky optimization (KTO) method. For example, the point-by-point preference data pointd50 may include triple data<query,response,signal> The triplet data may include question information, initial answer information, and a preference value. Based on the Kahneman-Tversky optimization method, the parameters of the conversational model may be adjusted. According to the question information in the triplet, the adjusted conversational model may generate adjusted answer information. The preference value of the adjusted answer information may be greater than the preference value of the initial answer information, for example.

[0088] It can be understood that the above description uses point-by-point preference data as an example to illustrate the present disclosure, but the present disclosure is not limited thereto, and the following description will be made using pair-by-pair preference data as an example.

[0089] In the disclosed embodiment, the training sample includes a plurality of initial answer information, and the feedback information includes a preference value. According to at least one training sample, adjusting the parameters of the conversational model includes: adjusting the parameters of the conversational model according to the question information of the training sample, the plurality of initial answer information, and the preference values ​​of the plurality of initial answer information. The parameters of the conversational model may be adjusted based on at least one of direct preference optimization (DPO), simple preference optimization (simPO), and proximal policy optimization (PPO). For example, the pair-wise preference data may include the above-mentioned N-tuple data<query,response1,……,responseN,signal> . The N-tuple data may include question information, N initial answer information and feedback information. The feedback information may correspond to N preference values. For example, based on a direct preference optimization method, the parameters of the conversational model may be adjusted. According to the question information in the N-tuple, the adjusted conversational model may generate adjusted answer information. The preference value of the adjusted answer information may be, for example, close to the highest value among the N preference values.

[0090] It can be understood that the above description of the present disclosure is based on the example that the pair-wise preference data includes N-tuple data, but the present disclosure is not limited thereto, and the pair-wise preference data may also include triplet data, which will be described below.

[0091] In the disclosed embodiment, the feedback information includes target answer information provided by the subject. The subject has a higher preference for the target answer information than for the initial answer information. Adjusting the parameters of the conversational model according to at least one training sample includes: adjusting the parameters of the conversational model according to the question information, the initial answer information, and the target answer information of the training sample. The parameters of the conversational model may be adjusted based on at least one of direct preference optimization, simple preference optimization, and proximal strategy optimization. For example, the pairwise preference data may include the above triple data<query,response_before,response_after> The triplet data may include question information, initial answer information, and target answer information. For example, based on a simple preference optimization method, the parameters of the conversational model may be adjusted. According to the question information in the triplet, the adjusted conversational model may generate adjusted answer information. The subject's preference for the adjusted answer information may be close to the subject's preference for the target answer information, for example.

[0092] It can be understood that the method of the present disclosure is described above, and the device of the present disclosure will be described below.

[0093] Figure 6 A block diagram of a question and answer information processing apparatus according to an embodiment of the present disclosure.

[0094] like Figure 6As shown, the device 600 may include a first generating module 610 , an acquiring module 620 , and a second generating module 630 .

[0095] The first generating module 610 is used to generate at least one initial answer information according to the question information provided by the object.

[0096] The acquisition module 620 is used to acquire at least one feedback information corresponding to at least one initial answer information.

[0097] In the disclosed embodiment, the feedback information is used to indicate the subject's preference for the initial answer information.

[0098] The second generating module 630 is used to generate a training sample according to the question information, at least one initial answer information and at least one feedback information.

[0099] In some embodiments, the first generation module includes: a first generation sub-module, which is used to generate at least one initial answer information based on the question information using a conversational model.

[0100] In some embodiments, the object includes at least one of a user and a large model, and the feedback information includes at least one of a preference level value and target answer information provided by the object.

[0101] In some embodiments, the subject's preference for the target answer information is higher than the preference for the initial answer information. The acquisition module includes at least one of the following: a first acquisition submodule, for acquiring target answer information provided by the subject, corresponding to the question information, and having a correlation index value with the initial answer information lower than a preset correlation threshold. A second acquisition submodule, for acquiring target answer information provided by the subject and edited from the initial answer information.

[0102] In some embodiments, the feedback information is generated by the large model according to preset feedback rules and initial answer information, and the preset feedback rules are determined according to user preferences.

[0103] In some embodiments, the preference value is determined based on one of multiple visual controls triggered by a user, the multiple visual controls include a first visual control and a second visual control, and the preference value corresponding to the first visual control is higher than the preference value corresponding to the second visual control.

[0104] It can be understood that the above describes the question and answer information processing device of the present disclosure, and the model training device of the present disclosure will be described below.

[0105] Figure 7 It is a block diagram of a training device for a conversational model according to another embodiment of the present disclosure.

[0106] like Figure 7As shown, the apparatus 700 may include an adjustment module 740 .

[0107] The adjustment module 740 is used to adjust the parameters of the conversational model according to at least one training sample, so that the conversational model generates adjusted answer information according to the question information in the training sample.

[0108] In the embodiment of the present disclosure, the adjusted answer information is close to the answer information with high preference in the training sample and away from the answer information with low preference in the training sample.

[0109] In the embodiment of the present disclosure, the training sample is generated by the following modules: a first generation module, used to generate at least one initial answer information according to the question information provided by the subject. An acquisition module, used to acquire at least one feedback information corresponding to the at least one initial answer information. The feedback information is used to indicate the subject's preference for the initial answer information. A second generation module, used to generate the training sample according to the question information, the at least one initial answer information and the at least one feedback information. For example, the training sample can be generated by the above-mentioned device 600.

[0110] In some embodiments, the feedback information includes a preference value. The adjustment module includes: a first adjustment submodule, configured to adjust parameters of the conversational model according to question information, initial answer information, and preference value of the training sample.

[0111] In some embodiments, the adjustment module further includes: a first adjustment unit, configured to adjust parameters of the conversational model based on a Kahneman-Tversky optimization method.

[0112] In some embodiments, the training sample includes multiple initial answer information, and the feedback information includes a preference value. The adjustment module includes: a second adjustment submodule, which is used to adjust the parameters of the conversational model according to the question information of the training sample, the multiple initial answer information, and the preference values ​​of the multiple initial answer information.

[0113] In some embodiments, the feedback information includes target answer information provided by the subject, and the subject prefers the target answer information more than the initial answer information. The adjustment module includes: a third adjustment submodule, which is used to adjust the parameters of the conversational model according to the question information, initial answer information and target answer information of the training sample.

[0114] In some embodiments, the adjustment module further includes: a second adjustment unit, configured to adjust parameters of the conversational model based on at least one of direct preference optimization, simple preference optimization, and proximal strategy optimization.

[0115] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0117] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0118] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the question-answering information processing method and / or the training method of the conversational model. For example, in some embodiments, the question-answering information processing method and / or the training method of the conversational model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the question and answer information processing method and / or the conversational model training method described above may be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform the question and answer information processing method and / or the conversational model training method in any other appropriate manner (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0126] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A question-answer information processing method, comprising: Generate at least one initial answer information according to the question information provided by the subject; Acquire at least one piece of feedback information corresponding to at least one piece of the initial answer information, wherein the feedback information is used to indicate the subject's preference for the initial answer information; as well as A training sample is generated according to the question information, at least one of the initial answer information and at least one of the feedback information.

2. The method according to claim 1, wherein: The step of generating at least one initial answer information according to the question information provided by the subject includes: At least one of the initial answer information is generated based on the question information using a conversational model.

3. The method according to claim 1, wherein: The object includes at least one of a user and a large model, and the feedback information includes at least one of a preference degree value and target answer information provided by the object.

4. The method according to claim 3, wherein: The subject's preference for the target answer information is higher than the preference for the initial answer information, The acquiring of at least one feedback information corresponding to at least one of the initial answer information comprises at least one of the following: Acquire target answer information provided by the subject, corresponding to the question information and having a relevance index value with the initial answer information lower than a preset relevance threshold; The target answer information provided by the object and obtained by editing the initial answer information is obtained.

5. The method according to claim 3, wherein: The feedback information is generated by the large model according to a preset feedback rule and the initial answer information, and the preset feedback rule is determined according to the user's preference.

6. The method according to claim 3, wherein: The preference value is determined according to one of a plurality of visual controls triggered by the user, the plurality of visual controls comprising a first visual control and a second visual control, the preference value corresponding to the first visual control being higher than the preference value corresponding to the second visual control.

7. A method for training a conversational model, comprising: According to at least one of the training samples, adjusting the parameters of the conversational model so that the conversational model generates adjusted answer information according to the question information in the training sample, and the adjusted answer information is close to the answer information with a high degree of preference in the training sample and away from the answer information with a low degree of preference in the training sample, The training samples are generated by the following operations: Generate at least one initial answer information according to the question information provided by the subject; Acquire at least one piece of feedback information corresponding to at least one piece of the initial answer information, wherein the feedback information is used to indicate the subject's preference for the initial answer information; and The training sample is generated according to the question information, at least one of the initial answer information and at least one of the feedback information.

8. The method according to claim 7, wherein: The feedback information includes a preference value, The adjusting the parameters of the conversational model according to at least one of the training samples includes: The parameters of the conversational model are adjusted according to the question information, the initial answer information and the preference level value of the training sample.

9. The method according to claim 8, wherein: The adjusting the parameters of the conversational model includes: The parameters of the conversational model are adjusted based on the Kahneman-Tversky optimization method.

10. The method according to claim 7, wherein: The training sample includes a plurality of the initial answer information, and the feedback information includes a preference value. The adjusting the parameters of the conversational model according to at least one of the training samples includes: The parameters of the conversational model are adjusted according to the question information of the training sample, the multiple initial answer information, and the preference level values ​​of the multiple initial answer information.

11. The method according to claim 7, wherein: The feedback information includes target answer information provided by the subject, and the subject's preference for the target answer information is higher than the preference for the initial answer information. The adjusting the parameters of the conversational model according to at least one of the training samples includes: The parameters of the conversational model are adjusted according to the question information, the initial answer information and the target answer information of the training sample.

12. The method according to claim 9 or 10, wherein: The adjusting the parameters of the conversational model includes: Based on at least one of direct preference optimization, simple preference optimization, and proximal strategy optimization, parameters of the conversational model are adjusted.

13. A question-answer information processing device, comprising: A first generating module, used to generate at least one initial answer information according to the question information provided by the subject; An acquisition module, configured to acquire at least one piece of feedback information corresponding to at least one piece of initial answer information, wherein the feedback information is used to indicate the subject's preference for the initial answer information; as well as The second generating module is used to generate a training sample according to the question information, at least one of the initial answer information and at least one of the feedback information.

14. The device according to claim 13, wherein: The first generation module comprises: The first generating submodule is used to generate at least one initial answer information according to the question information by using a conversational model.

15. The device according to claim 13, wherein: The object includes at least one of a user and a large model, and the feedback information includes at least one of a preference degree value and target answer information provided by the object.

16. The device according to claim 15, wherein: The subject's preference for the target answer information is higher than the preference for the initial answer information, The acquisition module comprises at least one of the following: A first acquisition submodule is used to acquire target answer information provided by the subject, corresponding to the question information and having a relevance index value with the initial answer information lower than a preset relevance threshold; The second acquisition submodule is used to acquire target answer information provided by the object and edited from the initial answer information.

17. The device according to claim 15, wherein: The feedback information is generated by the large model according to a preset feedback rule and the initial answer information, and the preset feedback rule is determined according to the user's preference.

18. The method according to claim 15, wherein: The preference value is determined according to one of a plurality of visual controls triggered by the user, the plurality of visual controls comprising a first visual control and a second visual control, the preference value corresponding to the first visual control being higher than the preference value corresponding to the second visual control.

19. A training device for a conversational model, comprising: an adjustment module, configured to adjust parameters of the conversational model according to at least one of the training samples, so that the conversational model generates adjusted answer information according to the question information in the training sample, and the adjusted answer information is close to the answer information with a high degree of preference in the training sample and is far away from the answer information with a low degree of preference in the training sample, The training samples are generated by the following modules: A first generating module, used to generate at least one initial answer information according to the question information provided by the subject; an acquisition module, configured to acquire at least one piece of feedback information corresponding to at least one piece of initial answer information, wherein the feedback information is used to indicate the subject's preference for the initial answer information; and The second generating module is used to generate the training sample according to the question information, at least one of the initial answer information and at least one of the feedback information.

20. The device according to claim 19, wherein The feedback information includes a preference value, The adjustment module comprises: The first adjustment submodule is used to adjust the parameters of the conversational model according to the question information, the initial answer information and the preference level value of the training sample.

21. The device according to claim 20, wherein: The adjustment module also includes: The first adjustment unit is used to adjust the parameters of the conversational model based on the Kahneman-Tversky optimization method.

22. The device according to claim 19, wherein The training sample includes a plurality of the initial answer information, and the feedback information includes a preference value. The adjustment module comprises: The second adjustment submodule is used to adjust the parameters of the conversational model according to the question information of the training sample, the multiple initial answer information and the preference level values ​​of the multiple initial answer information.

23. The device according to claim 19, wherein: The feedback information includes target answer information provided by the subject, and the subject's preference for the target answer information is higher than the preference for the initial answer information. The adjustment module comprises: The third adjustment submodule is used to adjust the parameters of the conversational model according to the question information, the initial answer information and the target answer information of the training sample.

24. The method according to claim 22 or 23, wherein: The adjustment module also includes: The second adjustment unit is used to adjust the parameters of the conversational model based on at least one of direct preference optimization, simple preference optimization and proximal strategy optimization.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images