Large language model optimization method and device based on multimodal feedback and reinforcement learning

Abstract

The embodiments of the present disclosure disclose a large language model optimization method and device based on multimodal feedback and reinforcement learning. A specific implementation of the method includes: obtaining a reply information set and a page feedback set; removing abnormal reply information to obtain a normal reply information set; determining the satisfaction information corresponding to each normal reply information; generating a first feedback data set; screening out a target reply information set; for each target reply information, executing a data generation step: using the corresponding initial reply information as an anchor sample, the reply content with a higher reply quality than the anchor sample as a positive sample, and the reply content with a lower reply quality than the anchor sample as a negative sample; generating second feedback data; and training the large language model. This implementation method can efficiently train the large language model through the multimodal information of page feedback and the performance of multiple rounds of output of the large language model, and obtain a large language model with more accurate output.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method and apparatus for optimizing a large language model based on multimodal feedback and reinforcement learning. Background Art

[0002] With the advent of the intelligent era, large language models are empowering various fields to improve industrial production capacity and work efficiency. The typical approach for training large language models is to directly train an initial large language model using a pre-acquired dataset to obtain the large language model.

[0003] However, when using the above approach to train large language models, the following technical issues often arise:

[0004] Conventionally trained large language models cannot fully capture users' complex attitudes and diverse needs towards model outputs, resulting in responses output by the large language model that are inconsistent with users' actual needs. The large language model has poor accuracy and provides a poor user experience. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a large language model optimization method and device based on multimodal feedback and reinforcement learning to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a large language model optimization method based on multimodal feedback and reinforcement learning, including: obtaining a reply information set output by a large language model and a page feedback set collected by a device; using a target clustering algorithm to remove abnormal reply information in the above reply information set to obtain a normal reply information set; determining the satisfactory information corresponding to each normal reply information based on the page feedback subset corresponding to the above normal reply information set to obtain a satisfactory information set; generating a first feedback data set based on the above satisfactory information set and the normal reply information set; screening out target reply information whose corresponding reply content has multiple rounds of replies from the above normal reply information set to obtain a target reply information set; for each target reply information, executing a data generation step: determining the initial reply information corresponding to the above target reply information as an anchor sample, determining the reply content whose corresponding reply quality in multiple rounds of replies is higher than that of the anchor sample as a positive sample, and determining the reply content whose corresponding reply quality in multiple rounds of replies is lower than that of the anchor sample as a negative sample; generating second feedback data for the above anchor samples, positive sample sets and negative sample sets; performing model training on the above large language model based on the above first feedback data set and the second feedback data set to obtain a trained large language model.

[0008] In the second aspect, some embodiments of the present disclosure provide a large language model optimization device based on multimodal feedback and reinforcement learning, including: an acquisition unit, configured to acquire a reply information set output by the large language model and a page feedback set collected by the device; a removal unit, configured to use a target clustering algorithm to remove abnormal reply information in the above reply information set to obtain a normal reply information set; a determination unit, configured to determine the satisfaction information corresponding to each normal reply information based on a page feedback subset corresponding to the above normal reply information set to obtain a satisfaction information set; a generation unit, configured to generate a first feedback data set based on the above satisfaction information set and the normal reply information set; a screening unit, configured to select the first feedback data set from the above The normal reply information set is filtered out, and the target reply information whose corresponding reply content has multiple rounds of replies is obtained; the execution unit is configured to execute the data generation step for each target reply information: the initial reply information corresponding to the above target reply information is determined as the anchor sample, the reply content whose corresponding reply quality is higher than the anchor sample in the multiple rounds of replies is determined as the positive sample, and the reply content whose corresponding reply quality is lower than the anchor sample in the multiple rounds of replies is determined as the negative sample; the second feedback data of the above anchor samples, positive sample sets and negative sample sets are generated; the training unit is configured to perform model training on the above large language model according to the above first feedback data set and the second feedback data set to obtain the trained large language model.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0011] The above-described embodiments of the present disclosure have the following beneficial effects: Through the large language model optimization method based on multimodal feedback and reinforcement learning in some embodiments of the present disclosure, the large language model can be efficiently trained through the multimodal information of page feedback and the performance of multiple rounds of large language model output, resulting in a large language model with more accurate output. Specifically, the reason for the inaccurate output of related large language models is that conventionally trained large language models cannot fully capture the user's complex attitudes and diverse needs towards the model output, resulting in the replies output by the large language model not matching the user's actual needs, resulting in poor accuracy of the large language model and a poor user experience. Based on this, the large language model optimization method based on multimodal feedback and reinforcement learning in some embodiments of the present disclosure first obtains a reply information set output by the large language model and a page feedback set collected by the device. Here, the reply information set is obtained as the data basis for subsequent model training of the large language model. The page feedback set can reflect the user's effective feedback on the content of the reply during the reply process. By collecting the page feedback set, the subsequent large language model can comprehensively consider the page feedback and achieve accurate training of the corresponding model. Then, using the target clustering algorithm, the abnormal reply information in the above reply information set can be efficiently and accurately removed to obtain the normal reply information set. Then, based on the page feedback subset corresponding to the above normal reply information set, the satisfaction information corresponding to each normal reply information can be accurately determined through various situations of page feedback to obtain the satisfaction information set. Through the satisfaction information, the subsequent large language model can know whether the output reply content meets the user's needs during the training process, so as to achieve accurate training of the large language model. Then, based on the above satisfaction information set and the normal reply information set, an accurate first feedback data set can be generated for the subsequent training of the large language model to generate a more accurate model. Further, the target reply information for which there are multiple rounds of replies to the corresponding reply content is screened out from the above normal reply information set to obtain the target reply information set, so as to focus on the data screening for the performance of the large language model in the multi-round reply process, so that the subsequent large language model can reduce the number of rounds and improve the output accuracy in the multi-round reply process. Then, for each target reply information, the data generation step is performed: the first step is to determine the initial reply information corresponding to the above target reply information as the anchor sample, and the reply content in multiple rounds of replies with a higher reply quality than the anchor sample is determined as a positive sample, and the reply content in multiple rounds of replies with a lower reply quality than the anchor sample is determined as a negative sample, so that the subsequent large language model can learn the reply situation in multiple rounds of replies. The second step is to generate the second feedback data of the above anchor samples, positive sample sets and negative sample sets as training samples, so that the subsequent large language model can improve the output accuracy while reducing the number of replies and continue to learn. Finally, the above large language model is trained based on the above first feedback data set and the second feedback data set to obtain a trained large language model.In summary, by considering the multimodal feature information of page feedback and the output accuracy of multiple replies, the large language model can be continuously improved based on feedback content and multiple replies, thereby improving the output accuracy of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of a large language model optimization method based on multimodal feedback and reinforcement learning according to the present disclosure;

[0014] Figure 2 1 is a schematic diagram of the structure of some embodiments of a large language model optimization device based on multimodal feedback and reinforcement learning according to the present disclosure;

[0015] Figure 3 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] refer to Figure 1 , shows a process 100 of some embodiments of a large language model optimization method based on multimodal feedback and reinforcement learning according to the present disclosure. The large language model optimization method based on multimodal feedback and reinforcement learning includes the following steps:

[0023] Step 101: Obtain the response information set output by the large language model and the page feedback set collected by the device.

[0024] In some embodiments, the execution entity (e.g., an electronic device) of the aforementioned large language model optimization method based on multimodal feedback and reinforcement learning can obtain, via wired or wireless means, a set of response information output by the large language model and a set of page feedback collected by the device. The large language model (LLM) can be a large-scale pre-trained model built on a deep neural network. It learns language statistics and semantic associations from massive amounts of text data, and possesses capabilities such as text generation, comprehension, and reasoning. The response information can be the result of the large language model and the user's response. The response information set can be a large set of response content output by the large language model within a target time period. For example, the target time period can be six months. The number of response information sets can be substantial. The large language model can be a commonly available large language model that supports responses in various fields. Page feedback can be page operations in which users provide quality feedback on the output content of the large language model. In practice, page feedback can be empty information, indicating that the user has not provided any page feedback on the response content. Page feedback can also include, but is not limited to, one of the following: page likes, page content favorites, page content copies, page content downvotes, and page dwell time. Page feedback sets can be acquired through refined event monitoring technology in client-side tracking design. There is a one-to-one correspondence between page feedback in the page feedback set and response information in the response information set. A corresponding model operation page exists for the large language model. The model operation page supports page-related operations such as question input and model response content. Response information may include, but is not limited to, at least one of the following: response user information, response time, response question, and response content.

[0025] Step 102: Using a target clustering algorithm, remove abnormal reply information from the reply information set to obtain a normal reply information set.

[0026] In some embodiments, the execution entity may utilize a target clustering algorithm to remove abnormal response information from the response information set, thereby obtaining a normal response information set. The target clustering algorithm may be a K-Means algorithm or a hierarchical clustering method. Abnormal response information may be response information that is not normally performed during the response process. In other words, abnormal response information may be response information that is the result of a malicious feedback attack. Abnormal response information is noisy response information. Normal response information is response information that is the result of a normal response process. In other words, normal response information may be response information that is not the result of a malicious feedback attack.

[0027] As an example, we first use the K-Means algorithm to cluster the responses in a response set based on their corresponding metrics. We then identify responses whose summed distance from cluster centers exceeds a target value as abnormal responses, yielding an abnormal response set. We then remove the abnormal response set from the response set, yielding a normal response set.

[0028] In some optional implementations of some embodiments, the execution entity may use a target clustering algorithm to remove abnormal response information from the response information set to obtain a normal response information set, including the following steps:

[0029] In the first step, the target clustering algorithm is used to cluster the response information set to obtain a set of response information clusters. The number of clusters corresponding to the response information clusters can be pre-set. Each response information cluster has corresponding cluster label information. The cluster label information includes multiple labels. Each label can represent the information category corresponding to each response information in the response information cluster. For example, the cluster label information may include: "female", "high school", and "campus question type".

[0030] As an example, the execution subject may cluster the reply information set based on features such as "question type" and "user group" as clustering features, and utilize the target clustering algorithm to obtain reply information clusters.

[0031] The second step is to determine the inter-cluster variance between each reply information cluster in the reply information cluster set and the intra-cluster variance corresponding to each reply information cluster. The between-cluster variance (Between-Cluster Variance) can characterize the degree of separation between different reply information clusters. In practice, the between-cluster variance can reflect the degree of difference between the cluster centers and the global mean of different reply information clusters. In practice, the larger the between-cluster variance, the higher the degree of separation between clusters and the better the clustering effect. The within-cluster variance (Within-Cluster Variance) can characterize the degree of dispersion between data points within the same cluster and the cluster center, which can reflect the compactness of the data within the cluster. In practice, the smaller the corresponding value of the within-cluster variance, the more clustered the data points within the cluster and the better the clustering effect.

[0032] The third step is to adaptively adjust each reply information cluster according to the above inter-cluster variance and the obtained intra-cluster variance set to obtain the adjusted reply information cluster set.

[0033] As an example, the execution entity may perform a trial removal of marginal reply information from each reply information cluster to determine the inter-cluster variance and intra-cluster variance sets, thereby maximizing the inter-cluster variance and minimizing the intra-cluster variance for equilibrium, thereby obtaining an adjusted reply information cluster set. Marginal reply information may be reply information within a reply information cluster whose distance from the cluster center exceeds a predetermined value.

[0034] In the fourth step, based on the response information cluster and the adjusted response information cluster, an initial abnormal response information set is filtered from the response information set. The initial abnormal response information may be response information that is preliminarily determined to have a potential abnormality. The abnormality may be an abnormal response. The abnormal response may be an abnormal response.

[0035] As an example, the execution subject compares the reply information cluster in the reply information cluster set with the reply information cluster in the adjusted reply information cluster set to obtain a difference reply information set as the initial abnormal reply information set.

[0036] Step 5: For each initial exception response message, perform the following determination steps:

[0037] Sub-step 1: Obtain the target response content corresponding to the initial exception response information, wherein the target response content may be the actual response content of the initial exception response information.

[0038] Sub-step 2: Determine the response content type and content validity corresponding to the target response content. The response content type can be the content type corresponding to the response content. Content validity can be the percentage of normal content in the response content. Normal content can be non-sensitive content and non-illegal content. In practice, semantic extraction of the response content can be used to determine whether the content is normal, and word extraction can be used to determine whether the response content contains sensitive content.

[0039] Sub-step 3, in response to determining that the above-mentioned content validity does not meet the target validity requirement and / or the reply content type belongs to the predetermined content type set, the above-mentioned initial abnormal reply information is determined as abnormal reply information. Among them, each predetermined content type in the predetermined content type set can be a content type that characterizes that the corresponding reply content is sensitive content. In practice, the predetermined content type can characterize the sensitivity of the corresponding reply question or the sensitivity of the reply content. For example, sensitive content is illegal content. The target validity requirement can be that the corresponding numerical value of the content validity is higher than the target numerical value. The content validity can be information in numerical form. The higher the content validity, the more likely the reply content in the corresponding reply information is to be a normal reply content.

[0040] Step 6: Remove the abnormal response information set from the above response information set to obtain a normal response information set.

[0041] Step 103: Determine the satisfactory information corresponding to each normal reply information based on the page feedback subset corresponding to the normal reply information set, and obtain a satisfactory information set.

[0042] In some embodiments, the execution subject may determine the satisfaction information corresponding to each normal reply information based on the page feedback subset corresponding to the normal reply information set to obtain a satisfaction information set. There is a one-to-one correspondence between the normal reply information in the normal reply information set and the page feedback in the page feedback subset. The satisfaction information may represent the user's satisfaction with the reply content output by the large language model in the normal reply information. In practice, the satisfaction information may be information in the form of a numerical value or in the form of a label. The higher the corresponding numerical value, the more satisfied the user is with the reply content output by the large language model in the normal reply information.

[0043] As an example, for each normal reply message, the execution entity may digitize each page feedback in the page feedback subset to obtain a page feedback value subset. The page feedback value subset is then input into a linear regression model with pre-trained parameter weights to obtain satisfaction information.

[0044] In some optional implementations of some embodiments, the page feedback in the page feedback set includes at least one of the following: an explicit feedback information set after performing an operation on the model operation page, and an implicit feedback information set after performing an operation on the model operation page. Explicit feedback information may be the feedback content resulting from an explicit operation. Explicit operations may be relatively obvious operations on the model operation page. Implicit feedback information may be implicit feedback information that can only be summarized through related operations. In practice, users may perform multiple explicit operations on the model operation page to obtain explicit feedback information sets. Explicit feedback information may include at least one of the following: like feedback and comment feedback. Like feedback indicates a like on a reply on the model operation page. Like feedback may include the like action and the like timestamp. Comment feedback indicates a comment on a reply on the model operation page. Comment feedback may include the comment content and the comment timestamp. The comment content may be pre-processed comment content. That is, text pre-processing is performed immediately after the user completes and submits the input. Text pre-processing may include removing special characters and performing word segmentation. The processed comment data is stored together with meta-information such as user ID and question ID to obtain the comment content. Implicit feedback information includes at least one of the following: page dwell time, context switching frequency, and number of answer modifications. Page dwell time can be the length of time a user stays on the model operation page to view the reply content. The context switching frequency can be counted by monitoring the number of times a user jumps between different question pages or pages related to the same question, and at the same time, the browser's history API can be used to analyze the path of the user switching contexts. The number of answer modifications can be that each time a user edits a generated answer, a background counter automatically accumulates and records the time and content of each modification.

[0045] Optionally, the execution entity may determine the satisfaction information corresponding to each normal response information based on the page feedback subset corresponding to the normal response information set, including the following steps:

[0046] The first step is to determine the target page feedback corresponding to the above normal reply information.

[0047] The second step is to determine the comment sentiment information based on the comment feedback included in the target page feedback. The comment sentiment information can be the user's sentiment towards the comment content. In practice, the comment sentiment information can be one of the following: positive sentiment, neutral sentiment, or negative sentiment.

[0048] As an example, the execution entity can input the review content from the review feedback into a pre-trained sentiment analysis model to obtain information about the sentiment tendency of the reviews. In specific scenarios, the sentiment analysis model can be part of a larger language model, that is, the larger language model supports sentiment analysis processing. The sentiment analysis model can also be a deep learning model independent of the larger language model. The sentiment analysis model can be a common model for sentiment analysis.

[0049] The third step is to quantify the sentiment information of the comments and the likes feedback to obtain sentiment values ​​and likes values. Quantifying the information can be to convert the information into numerical values.

[0050] In practice, a positive sentiment tendency can be converted to a value of "1", a neutral sentiment tendency to a value of "0", and a negative sentiment tendency to a value of "-1". In response to the like content in the like feedback being a like, the like value corresponding to the like feedback is set to a value of "1". In response to the like content in the like feedback being a not-liked, the like value corresponding to the like feedback is set to a value of "-1".

[0051] The fourth step is to obtain the first signal coefficient and first attenuation factor corresponding to the sentiment value at the current time, the second signal coefficient and second attenuation factor corresponding to the number of likes, the third signal coefficient and third attenuation factor corresponding to the page dwell time, the fourth signal coefficient and fourth attenuation factor corresponding to the context switching frequency, and the fifth signal coefficient and fifth attenuation factor corresponding to the number of answer revisions. Each signal coefficient and attenuation factor is periodically updated using a gradient descent algorithm. Initialization is performed based on the importance of the feedback signal type. For example, for explicit likes feedback, since it directly reflects the user's approval of the answer, the signal coefficient can be set to a higher value. For implicit context switching frequency, the signal coefficient is relatively low. The attenuation factor is obtained by training on a large amount of historical data and optimizing it using a gradient descent algorithm to ensure that the time-sensitive attenuation factor accurately reflects the timeliness of user behavior. The signal coefficient represents the importance of each feedback factor. The attenuation factor represents the degree of change in the importance of the corresponding content as the feedback factor changes over time.

[0052] The fifth step is to generate a first dynamic weight corresponding to the above-mentioned emotional tendency value based on the above-mentioned first signal coefficient, the above-mentioned first attenuation factor and the feedback time corresponding to the comment feedback.

[0053] First, determine the time difference between the feedback time corresponding to the comment feedback and the current time. Then, multiply the time difference by the first attenuation factor to obtain a multiplied value. Next, use the opposite value of the multiplied value as the exponent, and the value e as the base to generate an exponential value. Finally, multiply the exponential value by the first signal coefficient to obtain the first dynamic weight.

[0054] Step 6: Generate a second dynamic weight corresponding to the like value based on the second signal coefficient, the second attenuation factor, and the feedback time corresponding to the like feedback. The method for generating the second dynamic weight can be found in the method for generating the first dynamic weight. Details are not repeated here.

[0055] In the seventh step, a third dynamic weight corresponding to the page dwell time is generated based on the third signal coefficient, the third attenuation factor, and the feedback time corresponding to the page dwell time.

[0056] In the eighth step, a fourth dynamic weight corresponding to the context switching frequency is generated based on the fourth signal coefficient, the fourth attenuation factor, and the feedback time corresponding to the context switching frequency.

[0057] In the ninth step, a fifth dynamic weight corresponding to the number of answer revisions is generated based on the fifth signal coefficient, the fifth attenuation factor, and the feedback time corresponding to the number of answer revisions.

[0058] The tenth step is to generate satisfaction information based on the above-mentioned sentiment tendency value, the above-mentioned like value, the above-mentioned target page feedback including the target page stay time, the target context switching frequency, the target answer modification times, the above-mentioned first dynamic weight, the above-mentioned second dynamic weight, the above-mentioned third dynamic weight, the above-mentioned fourth dynamic weight and the fifth dynamic weight.

[0059] As an example, the above-mentioned execution entity can multiply the sentiment tendency value with the first dynamic weight to obtain a first multiplied value. Multiply the like value with the second dynamic weight to obtain a second multiplied value. Multiply the target page stay time with the third dynamic weight to obtain a third multiplied value. Multiply the target context switching frequency with the fourth dynamic weight to obtain a fourth multiplied value. Multiply the target answer modification times with the fifth dynamic weight to obtain a fifth multiplied value. Finally, the first multiplied value, the second multiplied value, the third multiplied value, the fourth multiplied value and the fifth multiplied value are added to obtain the above-mentioned satisfaction information.

[0060] Step 104: Generate a first feedback data set based on the above-mentioned satisfactory information set and normal response information set.

[0061] In some embodiments, the execution entity may generate a first feedback data set based on the satisfaction information set and the normal response information set. Each normal response information has corresponding first feedback data.

[0062] As an example, the execution entity may use the satisfaction information as a label and the normal response information as a training sample, and combine the labels and the training samples one by one to generate the first feedback data, thereby obtaining a first feedback data set.

[0063] Step 105 , filtering out target reply information with corresponding reply contents having multiple rounds of replies from the above-mentioned normal reply information set, and obtaining a target reply information set.

[0064] In some embodiments, the execution entity may filter target response information from the normal response information set to identify response content that has undergone multiple rounds of responses, thereby obtaining a target response information set. A multi-round response may involve a user conducting multiple rounds of questions and answers regarding a question (i.e., a response question). For example, if the content output by the large language model is not what the user desires, the user may continuously refine the question in order to facilitate the large language model's output. This process may result in multiple rounds of responses.

[0065] As an example, the execution entity may filter out target response information corresponding to the response content with multiple rounds of responses from the normal response information set based on the number of questions asked and the number of response contents, and obtain a target response information set.

[0066] Step 106: For each target reply message, execute the data generation step:

[0067] In step 1061, the initial response information corresponding to the target response information is determined as the anchor sample, the response content in multiple rounds of responses whose response quality is higher than that of the anchor sample is determined as the positive sample, and the response content in multiple rounds of responses whose response quality is lower than that of the anchor sample is determined as the negative sample.

[0068] In some embodiments, the execution entity may determine the initial response information corresponding to the target response information as an anchor sample, determine the response content with higher response quality than the anchor sample in multiple rounds of responses as positive samples, and determine the response content with lower response quality than the anchor sample in multiple rounds of responses as negative samples. The initial response information may include the user's first question and the response content of the large language model's first response. Response quality may refer to the degree to which the content output by the large language model matches the user's requirements.

[0069] In some optional implementations of some embodiments, the execution entity may determine the initial response information corresponding to the target response information as an anchor sample, determine the response content in multiple rounds of responses with higher response quality than the anchor sample as a positive sample, and determine the response content in multiple rounds of responses with lower response quality than the anchor sample as a negative sample, including the following steps:

[0070] The first step is to extract the response content sequence and question-and-answer sequence for each response phase from the target response information. The response content in the response content sequence corresponds to a response phase. The response content can be the response result of the large language model. The question-and-answer question can be the question entered by the user on the model operation page during each response phase. Each response content has a unique corresponding question-and-answer question.

[0071] The second step is to determine the target page feedback corresponding to the above target response information.

[0072] The third step is to extract the corresponding feedback sub-information from the target page feedback for each response in the response content sequence. The target page feedback contains interface feedback (i.e., feedback sub-information) for each response stage within multiple response rounds. For example, the multiple response rounds include: a first response stage, a second response stage, and a third response stage. The target page feedback includes: feedback sub-information corresponding to the first response stage (which may include explicit feedback and implicit feedback), feedback sub-information corresponding to the second response stage, and feedback sub-information corresponding to the third response stage.

[0073] The fourth step is to extract the question semantic information of each question in the question-answering sequence and the question association semantic information between each two adjacent questions, thereby obtaining a question semantic information sequence and a question association semantic information sequence. The question semantic information can be information in vector form that represents the semantic content of the question features corresponding to the question-answering question. The question association semantic information can be the association between the question semantic information corresponding to two adjacent questions. The question association semantic information can also be information in vector form.

[0074] The fifth step is to cross-fuse the above question semantic information sequence and question-related semantic information sequence to obtain a semantic information sequence.

[0075] As an example, the execution entity may cross-fuse the question semantic information sequence and the question-related semantic information sequence in order of rounds to obtain a semantic information sequence.

[0076] The sixth step is to perform feature dimensionality reduction under the adaptive scale dimensionality reduction format on the spliced ​​semantic information corresponding to the above semantic information sequence to obtain feature semantic dimensionality reduction information, wherein the semantic information at the later position in the sequence in the semantic information sequence. The higher the proportion of feature content in the above feature semantic dimensionality reduction information. Among them, the adaptive scale dimensionality reduction format can be a more delicate scale dimensionality reduction of the semantic content corresponding to the time closest to the current time in the spliced ​​semantic information, that is, the features extracted from the semantic content corresponding to the time closest to the current time are more detailed and more important. These contents are also the key contents that need to be learned in the subsequent large language model. Feature semantic dimensionality reduction information can be information in the form of a vector with a vector dimension of a preset dimension that represents the feature semantic content. Spliced ​​semantic information can be information in the form of a vector after the semantic information in the semantic information sequence is sequentially spliced ​​together.

[0077] As an example, the execution entity may first perform downsampling convolution processing on the spliced ​​semantic information using at least one convolution kernel of a different scale to obtain at least one piece of downsampled convolution information. Then, vector dimension completion may be performed on each piece of the at least one piece of downsampled convolution information (e.g., vector completion according to a target value) to obtain at least one completed vector having the same vector dimension. Finally, the at least one completed vector may be horizontally concatenated to obtain feature semantic dimensionality reduction information.

[0078] In the seventh step, a generative model is used to generate a comprehensive question corresponding to the aforementioned feature semantic dimensionality reduction information. The generative model can be a deep learning model that generates summary questions. The summary question can be a question without redundant content. In other words, it is a comprehensive question that is a summary of the questions from each round. In practice, the generative model can be a component of a larger language model or a deep learning model independent of the larger language model. For deep learning models independent of the larger language model, the generative model can be a Transformer-based generative model.

[0079] As an example, the above execution entity can directly input the feature semantic dimensionality reduction information into the generative model to obtain a comprehensive question.

[0080] Step 8: Generate high-quality responses to the above comprehensive questions. High-quality responses can be highly accurate.

[0081] As an example, the execution entity may first input a comprehensive question into at least one large language model of different model types to obtain at least one response. The execution entity may then aggregate the individual responses within the at least one response to obtain a high-quality response.

[0082] The ninth step is to generate high-quality content semantic information corresponding to the above-mentioned high-quality reply content, and generate a reply content semantic sequence corresponding to the reply content sequence. The high-quality content semantic information can be information in vector form that represents the content semantic information corresponding to the reply content. There is a one-to-one correspondence between the reply content semantics in the reply content semantic sequence and the reply content in the reply content sequence. The reply content semantics can be information in vector form that represents the content semantic information. As an example, the above-mentioned execution subject can use a content semantic extraction model to realize the generation of high-quality content semantic information and reply content semantic sequence.

[0083] Step 10: Determine the semantic similarity between each piece of reply content semantic information in the reply content semantic sequence and the high-quality content semantic information, obtaining a semantic similarity information sequence. The semantic similarity information can represent the similarity in semantic content between the reply content semantic information and the high-quality content semantic information. The semantic similarity information can be a numerical value between 0 and 1, with higher values ​​indicating greater semantic similarity between the two. In practice, the semantic similarity information can be cosine similarity.

[0084] In the eleventh step, based on the above semantic similarity information sequence, the initial response information corresponding to the above target response information is used as the anchor sample, the response content in multiple rounds of responses with higher response quality than the anchor sample is used as the positive sample, and the response content in multiple rounds of responses with lower response quality than the anchor sample is used as the negative sample.

[0085] As an example, first, at least one first reply content whose corresponding semantic similarity information is higher than the corresponding semantic similarity of the anchor sample is filtered out from the reply content sequence. Each first reply content is combined with the first label to generate a positive sample, and at least one positive sample is obtained. The first label can represent that the reply quality is higher than the corresponding reply quality of the anchor sample. Then, at least one second reply content whose corresponding semantic similarity information is lower than or equal to the corresponding semantic similarity of the anchor sample is filtered out from the reply content sequence. Each second reply content is combined with the second label to generate a negative sample, and at least one negative sample is obtained. The second label can represent that the reply quality is not higher than the corresponding reply quality of the anchor sample.

[0086] Optionally, the execution subject may perform feature dimensionality reduction in an adaptive scale dimensionality reduction format on the concatenated semantic information corresponding to the semantic information sequence to obtain feature semantic dimensionality reduction information, including the following steps:

[0087] The first step is to obtain different semantic information division methods. Different semantic information division methods correspond to different semantic information division nodes. The semantic information division method can be a method of intercepting and dividing semantic information. For example, the semantic information division method can be a division method based on the information length of "1:2:3" or a division method based on rounds. The division methods corresponding to different semantic information division methods are different. Different semantic information division methods can be manually set on the relevant pages.

[0088] In the second step, for each semantic information partitioning method, the following second generation step is performed:

[0089] Sub-step 1: Divide the concatenated semantic information into semantic information sequences according to the semantic information division method.

[0090] Sub-step 2, determining the down-sampling convolution block corresponding to each semantic information in the above semantic information sequence, and obtaining a down-sampling convolution block sequence. The down-sampling convolution block is a convolution block used for down-sampling processing. The later the down-sampling convolution block is in the sequence, the smaller the matrix dimension of the corresponding down-sampling convolution block is. For example, the down-sampling convolution block sequence includes: a first down-sampling convolution block, a second down-sampling convolution block, and a third down-sampling convolution block. The dimension of the convolution kernel corresponding to the first down-sampling convolution block is greater than the dimension of the convolution kernel corresponding to the second down-sampling convolution block. The dimension of the convolution kernel corresponding to the second down-sampling convolution block is greater than the dimension of the convolution kernel corresponding to the third down-sampling convolution block. In practice, the dimensions of the convolution kernels corresponding to each down-sampling convolution block can be set in an arithmetic difference form.

[0091] Sub-step 3: For each piece of semantic information, use the corresponding downsampling convolution block to perform feature dimensionality reduction on the semantic information to obtain semantic dimensionality reduction information.

[0092] Sub-step 4: sequentially concatenate the semantic dimensionality reduction information sequences to obtain semantic concatenation semantic information.

[0093] The third step is to semantically fuse the semantic information of each semantic splicing to obtain feature semantic dimensionality reduction information.

[0094] As an example, the execution entity may supplement and stitch together the semantic splicing information to obtain semantic splicing information, and input the semantic splicing information into a fully connected layer to output feature semantic dimension reduction information.

[0095] Step 1062: Generate second feedback data of the above-mentioned anchor samples, positive sample sets, and negative sample sets.

[0096] In some embodiments, the execution entity may generate second feedback data for the anchor sample, positive sample set, and negative sample set.

[0097] As an example, for an anchor sample, each sample in the positive sample set and the negative sample set has a corresponding user-input question. The anchor sample, the positive sample set, the negative sample set, and the corresponding user-input question can be combined in a key-value pair format to obtain the second feedback data. For example, the second feedback data can be {anchor sample: question 1, positive sample 1: question 2, positive sample 2: question 3, negative sample 1: question 4, negative sample 2: question 5}.

[0098] Step 107 : Perform model training on the large language model based on the first feedback data set and the second feedback data set to obtain a trained large language model.

[0099] In some embodiments, the execution entity may perform model training on the large language model based on the first feedback data set and the second feedback data set to obtain a trained large language model.

[0100] As an example, the execution entity may use the first feedback data set and the second feedback data set as training data sets, and perform model training on the large language model using a training method based on a gradient descent method to obtain a trained large language model.

[0101] In some optional implementations of some embodiments, the execution entity may perform model training on the large language model based on the first feedback data set and the second feedback data set to obtain a trained large language model, including the following steps:

[0102] The first step is to obtain a cluster label information set corresponding to the obtained abnormal reply information cluster set. There is a one-to-one correspondence between the abnormal reply information clusters in the abnormal reply information cluster set and the cluster label information in the cluster label information set. There is a cluster correspondence between the abnormal reply information clusters in the abnormal reply information cluster set and the reply information clusters in the reply information cluster set. Specifically, the abnormal reply information cluster may be the cluster obtained by removing the normal reply information from the reply information cluster. The cluster label information may be the label information corresponding to the reply information cluster.

[0103] In the second step, for each exception reply message, execute the first generation step:

[0104] Sub-step 1: Obtain the exception response information cluster corresponding to the above-mentioned exception response information.

[0105] Sub-step 2, according to the cluster label information corresponding to the above-mentioned abnormal reply information cluster, obtain the corresponding knowledge graph as the first knowledge graph. Among them, the knowledge graph can be a knowledge graph of general knowledge related to the cluster label information. In practice, the cluster label information includes multiple labels. The multiple labels may include: feature labels, scene labels and object labels. The corresponding knowledge graph can be a graph of basic knowledge related to multiple labels. For example, the cluster label information includes: "boys", "campus", and "basketball skills". The corresponding knowledge graph can be a graph related to "boys", "campus", and "basketball skills". In practice, the knowledge graph corresponding to the cluster label information can be a knowledge graph output by a related model. The main nodes in the knowledge graph are the important related content corresponding to the labels, and the edges are the association relationships between the labels.

[0106] Sub-step 3: Multiple adjustments are made to the exception response information based on the key knowledge nodes and key knowledge edges in the first knowledge graph to obtain an adjusted exception response information set. A key knowledge node can be a graph node corresponding to a label or a node with the largest number of edge connections. A key knowledge edge can be an edge corresponding to a key knowledge node.

[0107] In practice, we can adjust the content of abnormal response information based on the correspondence between key knowledge nodes and key knowledge edges and conventional knowledge connections, thereby obtaining an adjusted abnormal response information set. For example, if the key knowledge node and key knowledge edge are "Boys often play basketball on campus," the content of the abnormal response information can be changed to "Boys cannot play basketball on campus" to obtain an adjusted abnormal response information set.

[0108] In a third step, in response to the total number of information corresponding to the exception response information set and the adjusted exception response information set reaching a target data volume, the exception response information set and the adjusted exception response information set are fused to obtain an exception fused response information set. The target data volume may be a preset data volume.

[0109] The fourth step is to generate third feedback data for each abnormal fusion reply information according to the corresponding content validity, reply content type and the above abnormal fusion reply information.

[0110] As an example, the execution entity may generate a triple corresponding to content validity, reply content type, and the abnormal fusion reply information as the third feedback data.

[0111] In the fifth step, the large language model is trained based on the first feedback data set, the second feedback data set, and the third feedback data set to obtain a trained large language model.

[0112] As an example, the above-mentioned execution entity can use the first feedback data set as the first training set, the second feedback data set as the second training set, and the third feedback data set as the third training set, and perform model training on the above-mentioned large language model through back propagation to obtain a trained large language model.

[0113] In some optional implementations of some embodiments, after step 107, the steps further include:

[0114] The first step is to, in response to receiving a target question input by a target subject on a model operation page, decompose the target question into multiple semantic units and determine a corresponding response information cluster as a target response information cluster. The target subject may be a user who inquires about the response content corresponding to the target question. The multiple semantic units may be semantic units corresponding to sub-questions within the target question. Semantic units may be vector-based information representing the semantic content of the sub-questions.

[0115] The second step is to determine the corresponding second knowledge graph based on the cluster label information corresponding to the target response information cluster. The details will not be repeated here.

[0116] The third step is to determine the semantic knowledge concepts and semantic knowledge relationships corresponding to each of the multiple semantic units based on the second knowledge graph.

[0117] As an example, the execution subject may query the second knowledge graph for semantic knowledge concepts and semantic knowledge relationships corresponding to multiple keywords for each semantic unit by keyword query. The semantic knowledge relationship may be an association relationship between semantic knowledge concepts.

[0118] The fourth step is to generate a triple for the above semantic knowledge concept, the above semantic knowledge relationship and the target question.

[0119] The fifth step is to generate prompt information representing the response information, evidence chain, and confidence level generated based on the triples. The evidence chain can be a chain-like representation of the basis for determining the correctness of the response information. The basis for determination can be the content of the basis for determining the correctness of the logic. The confidence level can be a value between 0 and 1. The confidence level can represent the correctness probability of each basis for determination.

[0120] The sixth step is to input the above prompt information into the above trained large language model to obtain the optimized reply information, the reply evidence chain corresponding to the above optimized reply information, and the confidence of each basis content corresponding to the above reply evidence chain.

[0121] Step 7: Display the above optimized response information, the above response evidence chain and each confidence level on the above model operation page.

[0122] In some optional implementations of some embodiments, the large language model includes: a first large language sub-model for explicit feedback information and a second large language sub-model for implicit feedback information. The first large language sub-model can be a large language model that responds to the response question during the initial response to improve response satisfaction and encourage the target user to provide as much explicit feedback as possible. In practice, the first large language sub-model can be a temporal neural network model. The model input for the first large language sub-model is the response question, and the output is the response content and satisfaction information. The satisfaction information can be the satisfaction level of the response content. For example, the first large language sub-model can be a language model based on the Transformer model. The second large language sub-model can be a language model that improves the accuracy of the large language model's responses based on user feedback during multiple rounds of responses. The second large language sub-model is trained to accurately understand user feedback in multi-round dialogue scenarios, reduce response rounds based on the target user's questioning habits, and output accurate responses. In other words, the first large language sub-model is a large language model that outputs accurate responses to improve user satisfaction in the first round of dialogue. The second-largest language sub-model is a large language model that reduces the number of response turns for multi-turn dialogue scenarios. For example, the second-largest language model can also be a language model based on the Transformer model.

[0123] Optionally, the execution entity may perform model training on the large language model based on the first feedback data set and the second feedback data set to obtain a trained large language model, including the following steps:

[0124] The first step is to query the first feedback data set for a subset of feedback data corresponding to the target subject, using this subset as the target feedback data set. The target feedback data set is generated based on the target subject's questioning behavior. The target feedback data set can be an empty dataset. An empty dataset can indicate that this is the first time the target subject has asked a question using the large language model.

[0125] In the second step, in response to determining that the data volume corresponding to the above-mentioned target feedback data subset has not reached the target data volume and receiving support for page information acquisition from the target object, user operation behavior information is obtained from the operation page or operation application corresponding to the above-mentioned target object. The page information can be information on a web page or application operation information in an operation application. In practice, user operation behavior information can be obtained by crawling or local acquisition. The user operation behavior information is the behavior trajectory information of the user performing page operation behavior.

[0126] The third step is to generate the user feature information set based on the user operation behavior information, wherein the user feature information can represent the feature preference information of the target object under the target feature.

[0127] As an example, the execution entity may determine the user feature information set by means of word frequency statistics.

[0128] The fourth step is to crawl the question set of the target object from the web page or related applications. For example, the web page can be a search page. The related application can be a shopping application or a short video application.

[0129] Step 5: Generate a page key content information set corresponding to the page click information set based on the page click information set corresponding to the question information set. The page click information may be information about the webpage or app clicked by the target user in response to the question information. The page key content information may be the main content of the clicked page. There is a one-to-one correspondence between the question information in the question information set and at least one page click information in the page click information set. That is, multiple pages may be clicked in response to the question information.

[0130] As an example, for each page click information in the page click information set, the execution entity may extract the page key content information from the page content based on the page content corresponding to the page click information and the user characteristic information set. The page key content information may be the key content in the page associated with the user characteristic information set.

[0131] As an example, the execution entity may match the page local semantic information with the user feature information set to filter out the page local content closely related to at least one user feature information from the page content, thereby obtaining at least one page local content. The at least one page local content is then aggregated to obtain the page key content information.

[0132] The sixth step is to generate a page response information set for the page key content information set and the question information set, wherein the page response information includes: question information and at least one corresponding page key content information.

[0133] Step 7: For each page response, perform the following feedback data generation steps:

[0134] Sub-step 1: determining at least one page key content information and question information corresponding to the above page reply information, and using them as at least one target page key content information and target question information respectively.

[0135] Sub-step 2: Generate question feature semantic information corresponding to the target question information, and generate at least one page key content semantic information corresponding to at least one page key content information. The question feature semantic information can be information in vector form that represents the question semantic content corresponding to the target question information. Each page key content information has corresponding page key content semantic information. The page key content semantic information can be information in vector form that represents the semantics of the page key content.

[0136] Sub-step 3: Input the at least one page key content semantic information into a same page content extraction layer based on a multi-layer residual layer to generate common semantic information of the page key content. The common semantic information of the page key content may be the common semantic content corresponding to each of the at least one page key content semantic information. The common semantic information of the page key content may be information in vector form that represents the common page content of the at least one page key content information.

[0137] Sub-step 4: input the above-mentioned question feature semantic information and the common semantic information of the above-mentioned page key content into the generative layer based on the Transformer model to obtain the page response content.

[0138] Sub-step 5: generating page feedback data corresponding to the page response information based on the page response content, the page response information and the click information.

[0139] In the eighth step, the obtained page feedback dataset and the target feedback data subset are combined to obtain the initial explicit feedback dataset corresponding to the target object.

[0140] In a ninth step, in response to determining that the data volume corresponding to the target feedback data subset reaches the target data volume, the execution entity may directly determine the target feedback data subset as the explicit feedback data set.

[0141] In step 10, in response to determining that the data volume corresponding to the initial explicit feedback dataset reaches the target data volume, the execution entity may directly determine the initial explicit feedback dataset as the explicit feedback dataset.

[0142] In step 11, in response to receiving a signal that the target object does not support page information acquisition or the data volume of the initial explicit feedback data set does not reach the target data volume, the target object is determined to be associated with a target group based on the user feature information set, wherein the features of the target group are similar to those of the target object.

[0143] In the twelfth step, the feedback dataset corresponding to the above-mentioned target group, which is stored in advance on the data end, is combined with the target feedback data subset or the initial explicit feedback dataset to obtain a combined feedback dataset as the explicit feedback dataset. The feedback dataset corresponding to the target group may be a dataset under explicit feedback.

[0144] Step 13: Set the weight of the first dataset for the initial explicit feedback dataset, the weight of the second dataset for the feedback dataset, and the weight of the third dataset for the target feedback data subset. The weight of the third dataset is higher than the weight of the first dataset. The weight of the first dataset is higher than the weight of the second dataset. For example, the weight of the third dataset is 1, the weight of the first dataset is 0.7, and the weight of the second dataset is 0.4.

[0145] In the fourteenth step, based on the above-mentioned explicit feedback data set, the corresponding weights of each data set and the second feedback data set, the first large language sub-model and the second large language sub-model are trained using the back propagation method to obtain a trained large language model.

[0146] As an example, the execution subject can perform model training on the first large language sub-model according to the explicit feedback data set and the corresponding weights of each data set by back propagation to obtain the trained first large language sub-model. Then, for each feedback data in the second feedback data set, the trained first large language model is used to generate a response content set corresponding to the question set corresponding to each feedback data. The second feedback data set is used as input data and the response content set is used as the output label to perform model training on the second large language sub-model to obtain the trained second large language sub-model. Finally, the first large language sub-model and the second large language sub-model in the large language model are replaced with the trained first large language sub-model and the trained second large language sub-model respectively to obtain the trained large language model.

[0147] It should be noted that the above-mentioned "steps 1 to 14" as one of the inventive points of the present disclosure solves at least one other technical problem: "the large language model cannot achieve accurate responses in the first round for answering questions, and in the process of multiple rounds of responses, there is often ambiguity in understanding the questions, resulting in the subject asking multiple questions and the large language model giving multiple responses, which not only wastes a lot of computing resources but also leads to a poor user experience." It also solves the problem that "the large language model cannot output customized content based on the preferences of the target object, resulting in the large language model outputting content that is not what the target object wants, resulting in ambiguous content output." Based on this, the present disclosure first extracts the data set corresponding to the target object from the first feedback data set. Then, when it is determined that the data volume corresponding to the data set is small and the subsequent first large language sub-model cannot fully learn the relevant feature information of the target object's corresponding response preferences, with the permission of the target object, the explicit response information is filled in through historical page browsing records and application browsing records, so that more initial explicit feedback data sets matching the above-mentioned target object can be obtained. In the case that the initial explicit feedback data set has not yet reached the target data volume, the object group corresponding to the target object is determined based on the user feature information set to further supplement the explicit reply information. Here, through the same page content extraction layer and the generative layer based on the Transformer model, accurate generation of page reply content can be achieved. In addition, by setting the weights of each data set, the importance of the corresponding data of the true and false data sets can be judged, so that the subsequent first large language sub-model can achieve more accurate model training. Based on this, through the above-mentioned explicit feedback data set, the corresponding weights of each data set and the second feedback data set, the back propagation method can be used to achieve accurate training of the large language model.

[0148] In some optional implementations of some embodiments, the execution entity may perform model training on the first large language sub-model and the second large language sub-model using back propagation based on the explicit feedback dataset, the corresponding weights of each dataset, and the second feedback dataset to obtain a trained large language model, including the following steps:

[0149] The first step is to train the first large language sub-model using backpropagation based on the explicit feedback dataset and the weights of each dataset. This yields the trained first large language sub-model. During the training of the first large language sub-model, feature semantic information is selectively learned based on the weights of the datasets corresponding to each dataset. (That is, the higher the dataset weight, the more emphasis the first large language sub-model places on learning. Similarly, the lower the dataset weight, the more superficial learning the first large language sub-model places on learning.)

[0150] The second step is to extract the second feedback data from the second feedback data set as the extracted feedback data.

[0151] In the third step, for the extracted feedback data, perform the following training steps:

[0152] Sub-step 1: determine the feedback question information set under multiple rounds of replies corresponding to the above-extracted feedback data.

[0153] Sub-step 2: For each round of responses, perform the following question-setting steps:

[0154] The first sub-step is to determine the feedback question information corresponding to the current round of responses and the set of historical feedback question information prior to the current round of responses. The feedback question information may be questions raised by the target audience during the current round of responses. The set of historical feedback question information may be a set of questions raised by the target audience during the previous round of responses.

[0155] In the second sub-step, the feedback question information set and the historical feedback question information set are combined to obtain combined question information.

[0156] Sub-step 3: Using the trained first language sub-model, generate a set of responses and satisfaction information corresponding to the obtained comprehensive question information set, which serve as the feedback response content set and the feedback satisfaction information set, respectively. Each feedback response has corresponding feedback satisfaction information. Each comprehensive question has corresponding feedback response content and feedback satisfaction information.

[0157] Sub-step 4: for each round of replies in the multiple rounds of replies, combine the reply content corresponding to the extracted feedback data corresponding to the round of replies with the corresponding feedback reply content to obtain a content group.

[0158] Sub-step 5: Based on the obtained multiple content groups, multiple content loss differences corresponding to the multiple rounds of responses are generated. The content loss difference may represent the difference between the feedback response content in the content group and the response content corresponding to the second feedback data. Each content group has a corresponding content loss difference. A higher content loss difference indicates a smaller difference between the feedback response content and the response content corresponding to the second feedback data.

[0159] As an example, first, content feature semantic information corresponding to each content in the content group is generated to obtain a content feature semantic information group. Then, the content feature semantic information group is input into the cross entropy loss function to obtain a content loss difference.

[0160] Sub-step 6: For the positive sample set, negative sample set and anchor sample corresponding to the second feedback data, randomly combine the negative samples, positive samples and anchor samples to obtain a comparative learning sample set, wherein each comparative learning sample includes: a positive sample, a negative sample and an anchor sample.

[0161] Sub-step 7: Using the contrastive loss function, generate contrastive loss information corresponding to each contrastive learning sample in the contrastive learning sample set to obtain a contrastive loss information set. For example, the contrastive loss function can be a triplet loss function.

[0162] Sub-step 8: Perform weighted summation on the contrast loss information set and multiple content loss differences to obtain weighted loss information.

[0163] Sub-step 9: In response to determining that the weighted loss information corresponds to the loss information sequence tending to the target loss value, the second largest language sub-model is determined as the second largest language sub-model after training.

[0164] Sub-step 10: replacing the first large language sub-model and the second large language sub-model in the large language model with the trained first large language sub-model and the trained second large language sub-model respectively.

[0165] In the fourth step, in response to determining that the loss information sequence corresponding to the weighted loss information does not tend to the target loss value, the second largest language sub-model is trained according to the weighted loss information using a backpropagation training method to obtain an initial second largest language sub-model.

[0166] In the fifth step, the initial second-largest language sub-model is used as the second-largest language model, and the feedback data extracted again is used as the extracted feedback data, and the above training steps are continued.

[0167] It should be noted that the "first to fifth steps" described above, as another inventive feature of this disclosure, address another technical problem: "How to accurately train the first and second largest language sub-models to ensure that the trained second largest language sub-model can quickly and accurately respond to questions in a relatively small number of response rounds?" Based on this, this disclosure firstly achieves accurate responses to questions in a single round by training with a large number of the aforementioned explicit feedback datasets for the target subject and the corresponding weights for each dataset. Furthermore, the first largest language sub-model may not be adept at accurately responding to questions over multiple rounds based on adjustments to the target subject's questions. Furthermore, for each round of responses, the first largest language model generates labels for the response content in a single round, thereby generating information about the content loss for each round of responses. Furthermore, by setting positive, negative, and anchor samples, comparative learning samples can be constructed, allowing the second largest language sub-model to quickly grasp the quality differences between responses across rounds. By using comparative loss information, the second largest language sub-model can accurately respond over multiple rounds. Thus, a large language model can be accurately obtained.

[0168] In some optional implementations of some embodiments, the large language model further includes a response evidence chain and confidence generation model. The response evidence chain and confidence generation model can also be a temporal neural network model based on the Transformer model. The response evidence chain and confidence generation model can generate the response evidence chain and confidence based on a prompt word.

[0169] The above-described embodiments of the present disclosure have the following beneficial effects: Through the large language model optimization method based on multimodal feedback and reinforcement learning in some embodiments of the present disclosure, the large language model can be efficiently trained through the multimodal information of page feedback and the performance of multiple rounds of large language model output, resulting in a large language model with more accurate output. Specifically, the reason for the inaccurate output of related large language models is that conventionally trained large language models cannot fully capture the user's complex attitudes and diverse needs towards the model output, resulting in the replies output by the large language model not matching the user's actual needs, resulting in poor accuracy of the large language model and a poor user experience. Based on this, the large language model optimization method based on multimodal feedback and reinforcement learning in some embodiments of the present disclosure first obtains a reply information set output by the large language model and a page feedback set collected by the device. Here, the reply information set is obtained as the data basis for subsequent model training of the large language model. The page feedback set can reflect the user's effective feedback on the content of the reply during the reply process. By collecting the page feedback set, the subsequent large language model can comprehensively consider the page feedback and achieve accurate training of the corresponding model. Then, using the target clustering algorithm, the abnormal reply information in the above reply information set can be efficiently and accurately removed to obtain the normal reply information set. Then, based on the page feedback subset corresponding to the above normal reply information set, the satisfaction information corresponding to each normal reply information can be accurately determined through various situations of page feedback to obtain the satisfaction information set. Through the satisfaction information, the subsequent large language model can know whether the output reply content meets the user's needs during the training process, so as to achieve accurate training of the large language model. Then, based on the above satisfaction information set and the normal reply information set, an accurate first feedback data set can be generated for the subsequent training of the large language model to generate a more accurate output model. Further, the target reply information with multiple rounds of replies corresponding to the reply content is filtered out from the above normal reply information set to obtain the target reply information set, so as to focus on the data screening for the performance of the large language model in the multi-round reply process, so that the subsequent large language model can reduce the number of rounds and improve the output accuracy in the multi-round reply process. Then, for each target reply information, the data generation step is performed: the first step is to determine the initial reply information corresponding to the above target reply information as the anchor sample, and the reply content in multiple rounds of replies with a higher reply quality than the anchor sample is determined as a positive sample, and the reply content in multiple rounds of replies with a lower reply quality than the anchor sample is determined as a negative sample, so that the subsequent large language model can learn the reply situation in multiple rounds of replies. The second step is to generate the second feedback data of the above anchor samples, positive sample sets and negative sample sets as training samples, so that the subsequent large language model can improve the output accuracy while reducing the number of replies and continue to learn. Finally, the above large language model is trained based on the above first feedback data set and the second feedback data set to obtain a trained large language model.In summary, by considering the multimodal feature information of page feedback and the output accuracy of multiple replies, the large language model can be continuously improved based on feedback content and multiple replies, thereby improving the output accuracy of the large language model.

[0170] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a large language model optimization device based on multimodal feedback and reinforcement learning. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the large language model optimization device based on multimodal feedback and reinforcement learning can be specifically applied to various electronic devices.

[0171] like Figure 2 As shown, a large language model optimization device 200 based on multimodal feedback and reinforcement learning includes: an acquisition unit 201, a removal unit 202, a determination unit 203, a generation unit 204, a screening unit 205, an execution unit 206 and a training unit 207. Among them, the acquisition unit 201 is configured to obtain the reply information set output by the large language model and the page feedback set collected by the device; the removal unit 202 is configured to use the target clustering algorithm to remove the abnormal reply information in the above reply information set to obtain the normal reply information set; the determination unit 203 is configured to determine the satisfaction information corresponding to each normal reply information based on the page feedback subset corresponding to the above normal reply information set to obtain the satisfaction information set; the generation unit 204 is configured to generate a first feedback data set based on the above satisfaction information set and the normal reply information set; the screening unit 205 is configured to screen out the corresponding reply information from the above normal reply information set. There is target reply information for multiple rounds of replies to the content, and a target reply information set is obtained; the execution unit 206 is configured to execute the data generation step for each target reply information: the initial reply information corresponding to the above target reply information is determined as the anchor sample, the reply content with higher reply quality than the anchor sample in the multiple rounds of replies is determined as the positive sample, and the reply content with lower reply quality than the anchor sample in the multiple rounds of replies is determined as the negative sample; the second feedback data of the above anchor samples, positive sample sets and negative sample sets are generated; the training unit 207 is configured to perform model training on the above large language model according to the above first feedback data set and the second feedback data set to obtain the trained large language model.

[0172] It is understandable that the various units and references recorded in the large language model optimization device 200 based on multimodal feedback and reinforcement learning Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the large language model optimization device 200 based on multimodal feedback and reinforcement learning and the units contained therein, and will not be repeated here.

[0173] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0174] like Figure 3 As shown, electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 302 or programs loaded from a storage device 308 into a random access memory (RAM) 303. RAM 303 also stores various programs and data required for the operation of electronic device 300. Processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0175] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0176] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0177] It should be noted that in some embodiments of the present disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0178] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0179] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the reply information set output by the large language model and the page feedback set collected by the device; uses the target clustering algorithm to remove the abnormal reply information in the reply information set to obtain a normal reply information set; determines the satisfactory information corresponding to each normal reply information based on the page feedback subset corresponding to the normal reply information set to obtain a satisfactory information set; generates a first feedback data set based on the satisfactory information set and the normal reply information set; filters out the target reply information whose corresponding reply content has multiple rounds of replies from the normal reply information set to obtain a target reply information set; for each target reply information, executes a data generation step: determines the initial reply information corresponding to the target reply information as an anchor sample, determines the reply content whose corresponding reply quality in multiple rounds of replies is higher than that of the anchor sample as a positive sample, and determines the reply content whose corresponding reply quality in multiple rounds of replies is lower than that of the anchor sample as a negative sample; generates second feedback data for the anchor sample, positive sample set and negative sample set; performs model training on the large language model based on the first feedback data set and the second feedback data set to obtain a trained large language model.

[0180] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] The units described in some embodiments of the present disclosure may be implemented in software or in hardware. The described units may also be provided in a processor, for example, they may be described as: a processor comprising an acquisition unit, a removal unit, a determination unit, a generation unit, a screening unit, an execution unit, and a training unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the acquisition unit may also be described as a "unit for acquiring a response information set output by a large language model and a page feedback set collected by a device."

[0183] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0184] The above descriptions are merely some preferred embodiments of the present disclosure and illustrate the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A large language model optimization method based on multimodal feedback and reinforcement learning, characterized in that: include: Obtain the response information set output by the large language model and the page feedback set collected by the device; Using a target clustering algorithm, removing abnormal reply information from the reply information set to obtain a normal reply information set; Determining, based on the page feedback subset corresponding to the normal response information set, satisfactory information corresponding to each normal response information, to obtain a satisfactory information set; generating a first feedback data set according to the satisfaction information set and the normal response information set; Filtering target reply information with multiple rounds of replies corresponding to the reply content from the normal reply information set to obtain a target reply information set; For each target response message, perform the data generation steps: The initial response information corresponding to the target response information is determined as an anchor sample, the response content corresponding to the response quality higher than the anchor sample in multiple rounds of responses is determined as a positive sample, and the response content corresponding to the response quality lower than the anchor sample in multiple rounds of responses is determined as a negative sample; generating a second feedback data set of the anchor sample, the positive sample set, and the negative sample set; The large language model is trained according to the first feedback data set and the second feedback data set to obtain a trained large language model.

2. The method according to claim 1, characterized in that The method of using a target clustering algorithm to remove abnormal reply information from the reply information set to obtain a normal reply information set includes: Using the target clustering algorithm, clustering the reply information set to obtain a reply information cluster set; Determining the inter-cluster variance between each reply information cluster in the reply information cluster set and the intra-cluster variance corresponding to each reply information cluster; Adaptively adjusting each reply information cluster according to the inter-cluster variance and the obtained intra-cluster variance set to obtain an adjusted reply information cluster set; Filtering an initial abnormal reply information set from the reply information set according to the reply information cluster set and the adjusted reply information cluster set; For each initial exception reply message, the following determination steps are performed: Obtaining target response content corresponding to the initial exception response information; Determining the response content type and content validity corresponding to the target response content; In response to determining that the content validity does not meet a target validity requirement and / or the reply content type belongs to a predetermined set of content types, determining the initial abnormal reply information as abnormal reply information; The abnormal reply information set is removed from the reply information set to obtain a normal reply information set.

3. The method according to claim 2, characterized in that The performing model training on the large language model according to the first feedback data set and the second feedback data set to obtain a trained large language model includes: Obtaining a cluster label information set corresponding to the obtained abnormal response information cluster set; For each exception reply message, the first generation step is performed: Obtaining an exception response information cluster corresponding to the exception response information; According to the cluster label information corresponding to the abnormal reply information cluster, a corresponding knowledge graph is obtained as a first knowledge graph; According to the key knowledge nodes and key knowledge edges in the first knowledge graph, the exception response information is adjusted multiple times to obtain an adjusted exception response information group; In response to the total number of information corresponding to the exception reply information set and the adjusted exception reply information set reaching the target data volume, the exception reply information set and the adjusted exception reply information set are fused to obtain an exception fused reply information set; For each abnormal fusion reply information, generating third feedback data according to the corresponding content validity, reply content type and the abnormal fusion reply information; The large language model is trained according to the first feedback data set, the second feedback data set, and the third feedback data set to obtain a trained large language model.

4. The method according to claim 1, wherein The page feedback in the page feedback set includes at least one of the following: an explicit feedback information set after an operation is performed on the model operation page, and an implicit feedback information set after an operation is performed on the model operation page. The explicit feedback information includes at least one of the following: like feedback and comment feedback. The implicit feedback information includes at least one of the following: page dwell time, context switching frequency, and answer modification times. as well as The determining, based on the page feedback subset corresponding to the normal response information set, the satisfaction information corresponding to each normal response information includes: Determine the target page feedback corresponding to the normal reply information; Determining comment sentiment information based on the comment feedback included in the target page feedback; Quantifying the sentiment information of the comments and the corresponding likes feedback to obtain sentiment values ​​and likes values; Obtaining a first signal coefficient and a first attenuation factor corresponding to the sentiment tendency value at the current time, a second signal coefficient and a second attenuation factor corresponding to the like value, a third signal coefficient and a third attenuation factor corresponding to the page dwell time, a fourth signal coefficient and a fourth attenuation factor corresponding to the context switching frequency, and a fifth signal coefficient and a fifth attenuation factor corresponding to the number of answer modifications, wherein each signal coefficient and each attenuation factor is periodically updated using a gradient descent method; generating a first dynamic weight corresponding to the sentiment tendency value according to the first signal coefficient, the first attenuation factor, and a feedback time corresponding to the comment feedback; generating a second dynamic weight corresponding to the like value according to the second signal coefficient, the second attenuation factor, and the feedback time corresponding to the like feedback; generating a third dynamic weight corresponding to the page dwell time according to the third signal coefficient, the third attenuation factor, and a feedback time corresponding to the page dwell time; generating a fourth dynamic weight corresponding to the context switching frequency according to the fourth signal coefficient, the fourth attenuation factor, and a feedback time corresponding to the context switching frequency; generating a fifth dynamic weight corresponding to the number of answer modifications according to the fifth signal coefficient, the fifth attenuation factor, and a feedback time corresponding to the number of answer modifications; Satisfaction information is generated based on the sentiment tendency value, the like value, the target page feedback including the target page stay time, the target context switching frequency, the target answer modification times, the first dynamic weight, the second dynamic weight, the third dynamic weight, the fourth dynamic weight and the fifth dynamic weight.

5. The method according to claim 1, wherein The step of determining the initial response information corresponding to the target response information as an anchor sample, determining the response content in multiple rounds of responses whose response quality is higher than that of the anchor sample as a positive sample, and determining the response content in multiple rounds of responses whose response quality is lower than that of the anchor sample as a negative sample, includes: Extracting the response content sequence and the question-answering question sequence in each response stage from the target response information; Determine the target page feedback corresponding to the target reply information; For each reply content in the reply content sequence, extracting feedback sub-information corresponding to the reply content from the target page feedback; Extracting question semantic information of each question and answer in the question and answer sequence and question association semantic information between every two adjacent question and answer questions to obtain a question semantic information sequence and a question association semantic information sequence; Cross-fusing the question semantic information sequence and the question-related semantic information sequence to obtain a semantic information sequence; Performing feature dimensionality reduction in an adaptive scale dimensionality reduction format on the concatenated semantic information corresponding to the semantic information sequence to obtain feature semantic dimensionality reduction information, wherein the semantic information at a later sequence position in the semantic information sequence has a higher proportion of feature content in the feature semantic dimensionality reduction information; Generate a comprehensive question corresponding to the feature semantic dimensionality reduction information using a generative model; Generate high-quality responses to the comprehensive questions; Generating high-quality content semantic information corresponding to the high-quality reply content, and generating a reply content semantic sequence corresponding to the reply content sequence; Determining semantic similarity information between each reply content semantic information in the reply content semantic sequence and the high-quality content semantic information to obtain a semantic similarity information sequence; According to the semantic similarity information sequence, the initial reply information corresponding to the target reply information is taken as the anchor sample, the reply content in multiple rounds of replies whose quality is higher than that of the anchor sample is taken as the positive sample, and the reply content in multiple rounds of replies whose quality is lower than that of the anchor sample is taken as the negative sample.

6. The method according to claim 5, characterized in that The step of performing feature dimensionality reduction in an adaptive scale dimensionality reduction format on the spliced ​​semantic information corresponding to the semantic information sequence to obtain feature semantic dimensionality reduction information includes: Obtaining different semantic information division methods, wherein different semantic information division methods correspond to different semantic information division nodes; For each semantic information partitioning method, the following second generation step is performed: According to the semantic information division method, the spliced ​​semantic information is divided into semantic information sequences; Determine a down-sampling convolution block corresponding to each semantic information in the semantic information sequence to obtain a down-sampling convolution block sequence, wherein the later the down-sampling convolution block in the sequence is, the smaller the matrix dimension corresponding to the down-sampling convolution block is; For each piece of semantic information, using the corresponding downsampling convolution block, the feature dimension reduction of the semantic information is performed to obtain semantic dimension reduction information; Sequentially splice the semantic dimension reduction information sequences to obtain semantic splicing semantic information; The semantic information of each semantic concatenation is semantically fused to obtain feature semantic dimension reduction information.

7. The method according to claim 1, characterized in that The method further comprises: In response to receiving a target question input by a target subject, decomposing the target question into a plurality of semantic units, and determining an answer information cluster corresponding to the target question as a target answer information cluster; Determining a corresponding second knowledge graph according to cluster label information corresponding to the target reply information cluster; Determining, based on the second knowledge graph, a semantic knowledge concept and a semantic knowledge relationship corresponding to each semantic unit in the plurality of semantic units; generating a triple for the semantic knowledge concept, the semantic knowledge relationship, and the target question; Generate representation to generate response information, evidence chain and confidence prompt information based on triples; Inputting the prompt information into the trained large language model to obtain optimized reply information, a reply evidence chain corresponding to the optimized reply information, and confidence levels of each basis content corresponding to the reply evidence chain; The optimized response information, the response evidence chain and each confidence level are displayed on the model operation page.

8. A large language model optimization device based on multimodal feedback and reinforcement learning, characterized in that: include: an acquisition unit configured to acquire a response information set output by the large language model and a page feedback set collected by the device; a removal unit configured to remove abnormal reply information from the reply information set by using a target clustering algorithm to obtain a normal reply information set; a determining unit configured to determine, based on the page feedback subset corresponding to the normal response information set, satisfactory information corresponding to each normal response information, to obtain a satisfactory information set; a generating unit configured to generate a first feedback data set according to the satisfaction information set and the normal response information set; A screening unit is configured to screen out target reply information corresponding to reply contents with multiple rounds of replies from the normal reply information set, to obtain a target reply information set; An execution unit is configured to, for each target response information, perform a data generation step: determine the initial response information corresponding to the target response information as an anchor sample, determine the response content corresponding to the response quality higher than the anchor sample in the multiple rounds of responses as a positive sample, and determine the response content corresponding to the response quality lower than the anchor sample in the multiple rounds of responses as a negative sample; generating a second feedback data set of the anchor sample, the positive sample set, and the negative sample set; The training unit is configured to perform model training on the large language model based on the first feedback data set and the second feedback data set to obtain a trained large language model.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images