Text processing method, training method of text processing model and electronic equipment

Abstract

The invention provides a text processing method, a training method of a text processing model and electronic equipment, and relates to the field of artificial intelligence technology and model processing. The method comprises the following steps: acquiring an inquiry text; inputting the inquiry text into a text processing model, and generating a plurality of candidate texts by utilizing the text processing model; the inquiry text and the multiple candidate texts are input into a text processing model, the multiple candidate texts are evaluated through the text processing model, text evaluation results corresponding to the multiple candidate texts are obtained, and the text evaluation results are used for representing evaluation values of the corresponding candidate texts and winning rates relative to evaluation values of other candidate texts; and based on the text evaluation results corresponding to the plurality of candidate texts, determining a reply text corresponding to the inquiry text from the plurality of candidate texts. The technical problem that the text processing effect of the model in the related technology is poor is solved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology and model processing, specifically to a text processing method, a training method for a text processing model, and an electronic device. Background Technology

[0002] Currently, a major problem facing models in open-domain tasks is the lack of clear standard answers, which makes it difficult to apply traditional training methods based on validation rewards. While industry attempts have attempted to guide model iteration through human feedback or dedicated reward models, these solutions are costly, have limited generalization ability, and perform poorly, especially in long-tail scenarios. Furthermore, existing self-boosting learning methods either do not adequately consider the training of the evaluator role or employ offline methods, failing to achieve online policy iteration, thus resulting in poor text processing performance.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a text processing method, a text processing model training method, and an electronic device to at least solve the technical problem of poor text processing performance of models in related technologies.

[0005] According to one aspect of the embodiments of this application, a text processing method is provided, comprising: acquiring an inquiry text; inputting the inquiry text into a text processing model and generating multiple candidate texts using the text processing model; inputting the inquiry text and the multiple candidate texts into the text processing model, evaluating the multiple candidate texts using the text processing model respectively, and obtaining text evaluation results corresponding to the multiple candidate texts respectively, wherein the text evaluation results are used to characterize the evaluation value of the corresponding candidate text, the win rate relative to the evaluation values ​​of other candidate texts, and the evaluation value is the degree of matching between the corresponding candidate text and the inquiry text; and determining the response text corresponding to the inquiry text from the multiple candidate texts based on the text evaluation results corresponding to the multiple candidate texts respectively.

[0006] According to another aspect of the embodiments of this application, a method for training a text processing model is provided, comprising: acquiring training text; inputting the training text into an initial processing model and generating multiple predicted texts using the initial processing model; inputting the training text and the multiple predicted texts into the initial processing model, evaluating the multiple predicted texts using the initial processing model respectively, and obtaining a first evaluation result corresponding to each of the multiple predicted texts, wherein the first evaluation result is used to characterize the evaluation value of the corresponding predicted text, the win rate relative to the evaluation values ​​of other predicted texts, and the evaluation value is the degree of matching between the corresponding predicted text and the training text; training the initial processing model based on the first evaluation results corresponding to the multiple predicted texts respectively, to obtain a text processing model, wherein the text processing model is used to execute the method of any one of the above embodiments.

[0007] According to another aspect of the embodiments of this application, a computing device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0008] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0009] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0010] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0011] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0012] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0013] In this embodiment, an inquiry text is obtained; the inquiry text is input into a text processing model, and multiple candidate texts are generated using the text processing model; the inquiry text and multiple candidate texts are input into the text processing model, and the text processing model evaluates the multiple candidate texts respectively, obtaining text evaluation results corresponding to the multiple candidate texts. The text evaluation results represent the evaluation value of the corresponding candidate text, the win rate relative to the evaluation values ​​of other candidate texts, and the degree of matching between the corresponding candidate text and the inquiry text; based on the text evaluation results corresponding to the multiple candidate texts, the response text corresponding to the inquiry text is determined from the multiple candidate texts. By comparing the evaluation results of candidate texts in pairs, the relative advantage frequency of each candidate text when compared with other candidate texts is counted to determine the comprehensive evaluation value of each candidate text; the response text corresponding to the inquiry text is determined from the multiple candidate texts, i.e., the candidate text with the higher relative advantage frequency is preferentially selected as the response. This process improves the selection strategy of the text processing model, improves the text generation quality, and thus solves the technical problem of poor text processing performance of models in related technologies.

[0014] The above general description and the following detailed description are for illustrative and explanatory purposes only and do not constitute a limitation thereof. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of a text processing method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a text processing method according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the framework of a text processing method according to an embodiment of this application;

[0019] Figure 4 This is a flowchart of a training method for a text processing model according to an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of a text processing apparatus according to an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of a training apparatus for a text processing model according to an embodiment of this application;

[0022] Figure 7This is a structural block diagram of a computing device according to an embodiment of this application;

[0023] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in other orders. Wherein, "other orders" refers to orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that comprises a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed, or inherent to such processes, methods, products, or apparatus.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0027] The technical solution provided in this application is mainly implemented using a deep learning model. Deep learning models can be widely applied in fields such as Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and image generation, as well as to natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios of this application include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0028] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0029] Actor: During training, the large language model takes on the role of generating responses and is responsible for generating diverse candidate texts for specific inputs.

[0030] Judge: The role of evaluator during the training of the same model, who compares multiple responses generated by the generator in pairs and determines which response is better.

[0031] A ranking algorithm based on pairwise comparison win rate (Copeland Ranking Method, or simply Copeland Ranking Method): This method treats each generated response as a "candidate" and determines its global ranking by counting the number of wins it achieves in all pairwise comparisons.

[0032] Self-consistency Reward Mechanism: This mechanism is used to evaluate whether the judge's judgment is consistent with the final global ranking, aiming to improve the reliability and consistency of the judge's evaluation.

[0033] The Length-Control Module (LCM) is a mechanism designed to prevent models from generating longer text. In pairwise comparisons, it limits the difference in response length and gives extra weight to the wins of shorter texts, ensuring that responses of different lengths are evaluated fairly.

[0034] Position Bias Mitigation Mechanism (PBMM): To address the potential position bias problem that may arise when large language models act as evaluators, this mechanism randomly swaps the display order of two responses in pairwise comparisons, making the model evaluation more impartial and reducing evaluation errors caused by response position.

[0035] Length-Controlled Win Rate (LC Win Rate): This is the model win rate calculated during the automatic evaluation process, taking into account the influence of the output text length. Through the adjustment function of the length control module, the interference of length factors on the evaluation results is eliminated, making the win rate calculation more accurate.

[0036] According to an embodiment of this application, a text processing method is provided. The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than that shown here.

[0037] The technical solutions provided in this application can employ deep learning models with relatively large parameter scales, such as large models containing billions or even more model parameters. Here, "large model" is just one example; this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based language models (LM) or multimodal models (MM).

[0038] Considering the limited computing resources of mobile terminals, the methods described above in this application embodiment can be applied to, for example... Figure 1 The application scenarios shown. Figure 1 This is a schematic diagram illustrating an application scenario of a text processing method according to an embodiment of this application. Figure 1 In the application scenario shown, the deep learning model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the deep learning model, thereby implementing the method provided in this embodiment.

[0039] In this embodiment, the system consisting of a client device and a server can perform the following steps: The client device acquires the query text. The server inputs the query text into a text processing model and generates multiple candidate texts using the text processing model; the query text and the multiple candidate texts are input into the text processing model, and the text processing model evaluates the multiple candidate texts respectively to obtain text evaluation results corresponding to the multiple candidate texts; based on the text evaluation results corresponding to the multiple candidate texts, the response text corresponding to the query text is determined from the multiple candidate texts.

[0040] With the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-the-loop machines in other application scenarios. In one optional embodiment, the model-in-the-loop machine has multiple built-in models. Users can select a model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-the-loop machine can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the deep learning model-in-the-loop machine has a pre-trained model built-in. The high-performance computing unit built into the model-in-the-loop machine can then directly call this model to execute the methods provided in this application embodiment.

[0041] Furthermore, when users need to train their own models, they can upload their own datasets via the client. This dataset is sent from the client to the server. The server can then use this dataset to fine-tune the pre-trained model, resulting in the user's customized model, which can then be deployed to the production environment. To facilitate user adjustments, the server provides complete adjustment tools, development frameworks, and processes, supporting various adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0042] Under the aforementioned operating environment, this application provides the following: Figure 2 The text processing method shown. Figure 2 This is a flowchart of a text processing method according to an embodiment of this application. For example... Figure 2 As shown, the method may include the following steps:

[0043] Step S202: Obtain the query text.

[0044] The query text mentioned above is text input issued by the user or system to obtain information or answers. For example, a user might input, "I want to know the reviews of this restaurant." Query text can be any form of language input, such as questions, statements, or instructions, designed to guide the large language model to understand the task requirements and formulate corresponding generation strategies.

[0045] The process of obtaining the query text also involves preliminary processing of the input information, which may include, but is not limited to, standardization, cleaning, and format conversion, to ensure that the text can be correctly interpreted and responded to by the model. This is the foundation for the efficient operation of the Self-Examining (SERL) framework and a bridge connecting user needs with model-generated responses, laying a solid information foundation for subsequent multi-response generation and self-evaluation.

[0046] Step S204: Input the query text into the text processing model and use the text processing model to generate multiple candidate texts.

[0047] The text processing model described above can be a language model that can understand and generate text.

[0048] The candidate texts mentioned above are multiple possible response texts generated by the text processing model based on the query text.

[0049] The above text evaluation results are obtained by the text processing model after evaluating each candidate text, and are used to measure the quality of each candidate text.

[0050] The query text can be fed into a text processing model, which generates a diverse range of candidate texts to cover different answer angles and styles.

[0051] The acquired query text can be input into a pre-trained text processing model, triggering the model's generation mode and generating a series of diverse candidate texts as potential responses. This process leverages the generative characteristics of large language models, adjusting internal strategies to stimulate the model to generate N different but closely related responses to the query text. This diverse generation not only demonstrates the model's capabilities but also prepares for subsequent self-evaluation, ensuring sufficient comparative samples during the evaluation phase and providing rich material for model improvement.

[0052] Subsequently, these candidate texts are fed back into the model, which then switches to the role of Judge, evaluating the quality of these responses through pairwise comparisons and generating K judgments. Notably, to eliminate positional bias, half of the comparisons reverse the response order (i.e., the PBMM mechanism) to ensure impartiality in the evaluation. This mechanism not only avoids potential "first impression" biases in the model but also improves its flexibility and accuracy in handling complex tasks, enabling it to improve itself in unsupervised environments, unaffected by predetermined order, and focused on the essential value of the response content.

[0053] Step S206: Input the query text and multiple candidate texts into the text processing model, and use the text processing model to evaluate the multiple candidate texts respectively to obtain the text evaluation results corresponding to the multiple candidate texts.

[0054] The text evaluation result is used to characterize the evaluation value of the corresponding candidate text, which is the win rate relative to the evaluation values ​​of other candidate texts. The evaluation value is the degree of matching between the corresponding candidate text and the query text.

[0055] The query text, along with the generated candidate texts, can be fed back into the text processing model. The model then acts as an evaluator, assessing each candidate text and calculating its match with the query text to determine the win rate. The win rate reflects how often a candidate text is considered the better answer among all generated options, rather than simply its direct match with the query text.

[0056] The query text, along with several previously generated candidate texts, can be fed back into the text processing model. The model then acts as a judge, performing in-depth analysis and value assessment of each candidate text. Through a detailed evaluation of the matching degree between the candidate texts and the query text, the model generates a series of text evaluation results. These results not only reflect the independent quality of each candidate text, but more importantly, they reveal the relative strengths and weaknesses of the candidate texts in their competition, i.e., their win rates.

[0057] This evaluation process employs the Copeland method, treating each candidate text as a competing "player" and determining its relative position in the set through pairwise comparisons. The self-consistent reward mechanism (RJ) ensures the stability and reliability of the Judge's judgments, avoiding potential bias drift or decision inconsistency, thus making the evaluation results more fair and credible. The length control module (LCM) further guarantees the fairness of the evaluation, preventing the model from becoming biased due to differences in text length and ensuring that the evaluation focuses on text content rather than form.

[0058] Through this series of operations, this application can automatically distinguish responses without external supervision, driving continuous model improvement. This not only reduces the need for expensive manual annotation and lowers training costs, but also accelerates the model's learning speed, enabling it to quickly adapt and improve in a wider range of open-domain tasks. Ultimately, SERL achieves self-iterative upgrades of large language models, ensuring that each generated response more accurately and efficiently meets the user's actual needs, while eliminating common length and position biases.

[0059] Step S208: Based on the text evaluation results corresponding to the multiple candidate texts, determine the response text corresponding to the query text from the multiple candidate texts.

[0060] The above response text is a better candidate text determined based on the text evaluation results, and serves as a formal response to the inquiry text.

[0061] Based on the text evaluation results, candidate texts with high win rates are selected as the final response texts. This process is not based on a single criterion, but rather comprehensively considers the relative quality of all candidate texts, ensuring that the response text not only closely matches the query text but also stands out from the candidate set, meeting the needs of the user or task.

[0062] Based on the text evaluation results generated in the previous stage, the model determines candidate texts suitable as formal responses. This selection process relies on a refined evaluation mechanism that ranks each candidate response according to its matching strength with the query text, and then selects the text with the highest win rate as the final response text.

[0063] This application utilizes the win rate under length control as an evaluation criterion, calculated using the Copeland ranking method, ensuring the comprehensiveness and objectivity of the evaluation. The LC win rate not only considers the matching degree of text content but also balances the potential impact of different text lengths on the evaluation results through the length control module LCM, avoiding the model's tendency to favor excessively long or short texts. Furthermore, the position bias mitigation mechanism PBMM ensures that the quality evaluation remains consistent regardless of the order of candidate texts in the comparison, eliminating unfairness caused by position bias.

[0064] This application enables the output of high-quality response texts that have undergone rigorous screening and improvement. This approach not only improves the model's response quality and efficiency but also reduces the high costs associated with traditional reinforcement learning methods. In particular, it avoids reliance on manual annotation, enabling the model to exhibit stronger versatility and self-iterative capabilities in open-domain tasks. Ultimately, the response texts generated by this application are more closely aligned with user needs, addressing the challenge of lacking standard answers in open-domain tasks, providing users with a more accurate and personalized service experience, and simultaneously promoting the development and performance improvement of large language models in practical application scenarios.

[0065] The core of this application's embodiments is to ensure the quality and matching degree of the response text by evaluating and selecting the generated candidate texts through the text processing model itself. This method avoids dependence on external reward models or manual annotation, reduces training and maintenance costs, and improves the system's autonomy and adaptability. Simultaneously, the internal loop mechanism of the evaluation process enhances the text processing model's generation and evaluation capabilities, enabling it to continuously update itself without additional supervision, making it suitable for a wide range of open-domain task environments.

[0066] Through the above steps, the query text is obtained; the query text is input into a text processing model, and multiple candidate texts are generated using the text processing model; the query text and multiple candidate texts are input into the text processing model, and the text processing model evaluates the multiple candidate texts respectively, obtaining text evaluation results for each candidate text. The text evaluation results represent the evaluation value of the corresponding candidate text, its win rate relative to the evaluation values ​​of other candidate texts, and the degree of matching between the corresponding candidate text and the query text; based on the text evaluation results corresponding to the multiple candidate texts, the response text corresponding to the query text is determined from the multiple candidate texts. By comparing the evaluation results of candidate texts in pairs, the relative advantage frequency of each candidate text when compared with other candidate texts is counted, thereby determining the comprehensive evaluation value of each candidate text. The response text corresponding to the query text is then determined from the multiple candidate texts, i.e., the candidate text with the higher relative advantage frequency is preferentially selected as the response. This process improves the selection strategy of the text processing model, enhances the quality of text generation, and thus solves the technical problem of poor text processing performance in related technologies.

[0067] In the above embodiments of this application, a text processing model is used to evaluate multiple candidate texts to obtain text evaluation results corresponding to each candidate text. This includes: matching multiple candidate texts with query texts using a text processing model to obtain evaluation values ​​corresponding to each candidate text; determining multiple candidate text sequences based on any two candidate texts; determining a sequence evaluation result corresponding to any candidate text sequence based on the evaluation values ​​corresponding to the two candidate texts in any candidate text sequence, wherein the sequence evaluation result is used to characterize the relative merits of the evaluation values ​​corresponding to the two candidate texts; and determining the text evaluation results corresponding to each candidate text based on the sequence evaluation results corresponding to each of the multiple candidate text sequences.

[0068] The Evaluation Value (EV) mentioned above is a quantitative assessment of how well the text processing model matches the query text for each candidate text. A higher EV means that the text responds to the query more accurately and appropriately.

[0069] The candidate text sequence described above is formed by pairing any two candidate texts for pairwise comparison. The construction of the candidate text sequence ensures that the model can systematically evaluate the merits of the candidate texts, rather than evaluating each text individually.

[0070] The sequence evaluation results described above represent the winner judgment generated by the text processing model for each pair of candidate texts in the candidate text sequence. The sequence evaluation results determine which response is considered better and form the basis for constructing the global win rate.

[0071] The text evaluation result described above, a single value combining the evaluation results of all sequences, is used to characterize the average win rate of the candidate text across all comparisons. The text evaluation result, derived using the Copeland ranking method, is the final basis for determining the better response text, reflecting the advantage of the candidate text relative to other texts. By constructing and evaluating candidate text sequences, one or more text evaluation results are ultimately determined to select the response text that best matches the query text. This process not only improves the quality of the model's generated responses but also enhances the model's own evaluation mechanism, ensuring its continuous iteration and self-improvement in open-domain tasks without external supervision or guidance. This significantly reduces training costs and enhances the model's generalization and dynamic decision-making capabilities.

[0072] First, after receiving the query text, the model generates a series of candidate texts, each representing a different interpretation and answer to the query. Next, the model, acting as an evaluator, performs deep matching between each candidate text and the query text. This process, based on the model's inherent understanding and semantic analysis capabilities, assigns an evaluation value to each candidate text, intuitively reflecting the relevance and quality of the text. Subsequently, to eliminate positional bias and enhance the comprehensiveness of the evaluation, the model constructs a sequence of candidate texts, where each sequence contains pairwise comparisons of any two candidate texts. Within the sequence, the evaluation mechanism ensures that all candidate texts are examined equally regardless of their position within the sequence. By comparing their respective evaluation values, the model clarifies the relative quality of the two texts, forming a sequence evaluation result. This result reveals the relative quality of the texts, providing a direct basis for subsequent selection. Finally, by integrating all sequence evaluation results, the model calculates and sorts the text evaluation result for each candidate text, representing its overall performance among all candidate texts. This evaluation result reflects the competitiveness of the candidate text relative to the entire set, avoiding the bias that may arise from a single comparison, and ensuring that the final selected response text best addresses the needs of the query text.

[0073] By employing a model self-evaluation method, the need for additional manual annotation or dedicated reward models is eliminated, significantly reducing training costs while improving the model's response quality and generalization ability in open-domain tasks. In particular, through multiple pairwise comparisons and comprehensive evaluations, position and length biases are effectively eliminated, ensuring the fairness and accuracy of the evaluation. This further improves the model's self-learning and iteration path, enabling the model to adapt to various open-domain tasks more quickly and accurately, providing users with a more satisfactory service experience.

[0074] In the above embodiments of this application, determining multiple candidate text sequences based on any two candidate texts from multiple candidate texts includes: obtaining any one candidate text from multiple candidate texts to obtain a first candidate text; obtaining any one candidate text from candidate texts other than the first candidate text to obtain a second candidate text; and constructing two candidate text sequences based on the first candidate text and the second candidate text, wherein the first candidate text and the second candidate text correspond to different orders in the two candidate text sequences.

[0075] The first candidate text mentioned above is a text randomly selected from multiple candidate texts and used as a benchmark for pairwise comparison.

[0076] The second candidate text mentioned above is another text randomly selected from the candidate text set, in addition to the first candidate text, and is used in pairwise comparisons with the first candidate text.

[0077] During the evaluation phase, this application employs a Position Bias Mitigation Mechanism (PBMM) to ensure the model's impartiality in evaluating candidate texts. Specifically, two texts—a first candidate text and a second candidate text—are randomly selected from the candidate texts to construct two candidate text sequences. One sequence maintains the original order, while the other sequence reverses the order of the first and second candidate texts. This strategy ensures that the model is not affected by the response order during evaluation, thereby avoiding position bias.

[0078] When a text processing model acts as a judge, the PBMM mechanism requires the model to evaluate each pair of candidate texts twice: once in their original order and once in reverse order. This process effectively balances the model's evaluation bias, ensuring that each candidate text has the opportunity to be evaluated in different positions, eliminating the influence of positional bias on the evaluation results. The implementation of the PBMM mechanism not only improves the credibility of the evaluation results but also enhances the model's evaluation capabilities, enabling it to make fairer and more accurate judgments when facing open-domain tasks.

[0079] The PBMM mechanism is a key component of the SERL framework in this application, designed to address the potential positional bias problem when a large language model is used as a judge. By swapping the order of responses in pairwise comparisons, this scheme ensures that the model's judgment is not influenced by the position of the text in the sequence during evaluation. This is particularly important in open-domain tasks without standard answers, helping to eliminate irrational factors in evaluation and improve overall evaluation quality.

[0080] The PBMM mechanism ensures the fairness and consistency of model evaluation, avoiding positional bias. This allows the model, when acting as a judge, to focus more on the quality of the text content itself, rather than being influenced by the presentation order. This not only improves the evaluation reliability of the SERL framework but also promotes the self-improvement of the model's generative capabilities, enhancing its generalization and adaptability in open-domain tasks.

[0081] In this embodiment, to ensure the comprehensiveness and accuracy of the large language model's self-evaluation, a strategy of exchanging the order of candidate texts is adopted. Technically, this mechanism first randomly selects one text from multiple candidate texts generated by the model as the first candidate text, and then randomly selects another text from the remaining texts to form the second candidate text. Based on these two texts, the model constructs two sets of comparison sequences, one set maintaining the original order of the first and second texts, and the other set reversing their order to form a "mirror image" of the sequence.

[0082] The above approach effectively overcomes positional bias, preventing the model from assigning higher scores to texts that appear earlier in the sequence during evaluation. By comparing evaluation results under different orders, the model can more objectively judge text quality, ensuring fairness in the evaluation. For open-domain tasks, this mechanism improves the model's reliability as a judge, reduces reliance on external supervision signals, and enables the model to improve itself based on its intrinsic capabilities, significantly enhancing the quality of generated response text and its ability to solve problems in complex environments.

[0083] In the above embodiments of this application, constructing two candidate text sequences based on a first candidate text and a second candidate text includes: obtaining the ratio between the text length of the first candidate text and the text length of the second candidate text to obtain a target ratio; and constructing two candidate text sequences based on the first candidate text and the second candidate text when the target ratio is within a preset ratio range.

[0084] The text lengths of the first and second candidate texts can be compared. A target ratio is obtained by calculating the ratio between the lengths of the first and second candidate texts. If the target ratio falls within a preset range, the two texts are allowed to form two candidate text sequences for subsequent pairwise comparison and evaluation. Conversely, if the length difference is too large, the pair of responses is discarded to avoid being included in the evaluation, thus ensuring the fairness and effectiveness of the evaluation.

[0085] The large language model in this embodiment can automatically identify and filter out candidate text pairs with excessively large length differences during evaluation, thereby avoiding evaluation bias that may be caused by long texts. Through the constraint of LCM, the model no longer favors long texts, but instead judges based on the quality of the text content itself and its relevance to the query text, significantly improving the accuracy of the evaluation and the quality of the selected response.

[0086] The model evaluation in this embodiment incorporates length control, effectively avoiding irrational biases that the model might exhibit due to text length differences in the absence of supervision signals. This improves the fairness and efficiency of the evaluation when the model acts as the judge. This improvement not only enhances the quality of the generated responses but also strengthens the model's adaptability and generalization ability in open-domain tasks, significantly reduces evaluation errors caused by length bias, and promotes the co-evolution of generation and evaluation capabilities.

[0087] In this embodiment, the large language model effectively controls the interference of text length during self-evaluation and response selection, avoiding misjudgment of long content as high-quality responses. This enhances the model's evaluation capabilities and lays a solid foundation, enabling the model to focus more on the quality of text content and its relevance to the task. Consequently, it generates more accurate and user-responsive text in open-domain tasks. This improvement provides strong support for the model's introspective reinforcement learning path, further enhancing the model's response quality and evaluation reliability in complex open-domain tasks.

[0088] In the above embodiments of this application, the sequence evaluation result corresponding to the candidate text sequence is determined based on the evaluation values ​​corresponding to the two candidate texts in any candidate text sequence, including: determining the first candidate text and the second candidate text in any candidate text sequence; if the evaluation value corresponding to the first candidate text is greater than the matching degree corresponding to the second candidate text, determining the corresponding sub-evaluation result to indicate that the first candidate text is better than the second candidate text; if the matching degree corresponding to the second candidate text is greater than the matching degree corresponding to the first candidate text, determining the corresponding sub-evaluation result to indicate that the second candidate text is better than the first candidate text.

[0089] The first step involves randomly selecting two texts from the multiple candidate texts generated by the model, referred to as the first candidate text and the second candidate text, forming a candidate text sequence. This random selection ensures the diversity and comprehensiveness of the evaluation process, laying the foundation for subsequent pairwise comparisons.

[0090] The evaluation value described above is a numerical representation assigned by the model based on the fit and quality of the first candidate text with the query text. This value reflects the text's overall performance in terms of expressing the query's intent, information completeness, and language fluency.

[0091] The matching degree mentioned above is similar to the concept of the evaluation value. The matching degree is an indicator that measures the relevance and consistency between the second candidate text and the query text. It is also given in numerical form and used for subsequent comparison and evaluation.

[0092] The sub-evaluation results described above indicate that when the evaluation value of the first candidate text is higher than that of the second candidate text, it means that the first candidate text is of higher quality than the second candidate text, and vice versa. This comparison result will be recorded as the sub-evaluation result and used for subsequent integration and analysis to determine the final text evaluation result.

[0093] By performing in-depth analysis and pairwise comparisons of the generated candidate texts themselves, the model determines the relative merits of the candidate texts based on their fit and quality with the query text. This process requires no external supervision or manual annotation; based on the model's internal mechanisms, it significantly improves the efficiency and effectiveness of the evaluation, providing data support for the model's self-improvement. Through introspective comparison, the model can clearly identify which texts are superior and which need improvement, thereby promoting the improvement of the quality of the generated texts. The originality of this mechanism lies in its independence from external standard answers, enhancing the model's ability to self-improve in the absence of explicit guidance, reducing training costs, and strengthening the model's generalization and adaptability.

[0094] The embodiments of this application realize the ability of a large language model to self-evaluate and improve generated text without human intervention. The random selection and pairwise comparison in this process are designed to overcome the potential length bias and positional bias of the model, ensuring that the evaluation process is fair and reasonable, and the final output text is not only of high quality, but also more in line with the diverse needs of open domain tasks.

[0095] This application's embodiments implement text self-evaluation and comparison through an internal mechanism, ensuring the model's response quality in open-domain tasks. This process not only avoids reliance on expensive human preference data and reduces training costs, but also ensures the fairness and accuracy of model evaluation, accelerating the model's self-improvement and iteration in complex open-domain tasks, and providing users with more accurate and valuable information responses.

[0096] In the above embodiments of this application, determining the text evaluation results corresponding to multiple candidate text sequences based on the sequence evaluation results respectively includes: for any one candidate text among multiple candidate texts, determining at least one candidate text sequence containing the candidate text; determining the number of times the candidate text wins based on the sub-evaluation results corresponding to the at least one candidate text sequence, wherein the number of times the candidate text wins is used to characterize the number of times the candidate text is better than other candidate texts; and obtaining the text evaluation result corresponding to the candidate text based on the number of times the candidate text wins and the number of at least one candidate text sequence.

[0097] The aforementioned candidate text sequence refers to a sequence of any two texts selected from a series of candidate texts generated by the model. In this process, for each candidate text, at least one candidate text sequence containing that text is identified for pairwise comparison and evaluation.

[0098] The sub-evaluation results described above determine the superiority of one text over the other within each candidate text sequence by comparing the evaluation values ​​of the first and second candidate texts. These sub-evaluation results are directly derived from pairwise comparisons and reflect the matching status of the two texts with their corresponding query texts.

[0099] The number of wins mentioned above refers to the total number of times a particular text is judged to be the winner in all candidate text sequences containing that text. The number of wins quantifies the advantage of a candidate text relative to other texts and is an important basis for determining the text evaluation result.

[0100] The text evaluation results described above are calculated based on the relationship between the number of wins and the total number of candidate text sequences containing that text, reflecting the relative ranking and quality level of the candidate text within the entire generated set. The calculation of the text evaluation results incorporates the model's self-evaluation, providing an objective basis for the final text selection.

[0101] By constructing candidate text sequences containing arbitrary text and performing pairwise comparisons, a series of sub-evaluation results are obtained to reflect the superiority or inferiority of the texts. Subsequently, the number of wins for each text is calculated, that is, the number of times it wins in all comparisons. Finally, the text evaluation result for each text is determined by combining the number of wins with the total number of comparisons. Texts with higher evaluation results indicate that they perform better in the model's self-evaluation and are more likely to become the final preferred text.

[0102] This application's embodiments utilize introspective reinforcement learning to achieve efficient evaluation of generated text without external supervision signals or manual annotation. In this process, calculating the number of wins and text evaluation results not only helps the model identify its own strengths and weaknesses but also promotes the co-evolution of generation and evaluation capabilities, significantly improving the accuracy and satisfaction of output text in open-domain tasks. By calculating the number of wins and text evaluation results, the large language model can autonomously judge the quality of text without explicit annotations or reference answers, promoting self-learning and capability improvement.

[0103] In this embodiment, facing an open-domain task, the large language model generates a series of candidate texts to respond to user queries or instructions. To accurately and in real-time evaluate these texts, the model selects a candidate text and constructs multiple candidate text sequences containing that text. This aims to obtain a series of sub-evaluation results reflecting the relative quality of the texts through pairwise comparisons with other texts. Next, based on these sub-evaluation results, the model cumulatively calculates the number of times the selected candidate text wins in each sequence—that is, the total number of times the text is considered superior to other texts in the sequence. This calculation quantifies the quality level of the candidate text, particularly its superiority relative to other generated texts, thus providing the model with an inherent self-feedback mechanism. Finally, by linking the number of wins to the total number of candidate text sequences, the model derives an important comprehensive indicator: the text evaluation result. This result reveals the relative ranking of the candidate text within the entire evaluation pool. A higher text evaluation result means that the text performs better in the model's self-evaluation and is more likely to be selected as a high-quality response in the final output.

[0104] This application's model autonomously identifies and improves text quality without external supervision, manual annotation, or pre-defined answer standards. The introspective reinforcement learning framework, through built-in length control and positional bias mitigation mechanisms, ensures the model's impartiality as an evaluator, avoiding irrational biases towards long texts or specific positions within sequences, thus making the evaluation results more objective and accurate. This approach significantly reduces training costs and iteration cycles, enhances the model's generalization ability, and provides more personalized, high-quality responses, especially when handling complex and varied open-domain tasks. For example, in e-commerce data analysis scenarios, the model-generated text about the reasons for sales fluctuations, through pairwise comparisons and the calculation of the number of wins, can more accurately identify which analyses truly meet user needs, thereby enabling self-adjustment and improvement.

[0105] In the above embodiments of this application, determining the number of wins for any candidate text based on the sub-evaluation results corresponding to at least one candidate text sequence includes: determining a target text sequence from at least one candidate text sequence, wherein any text is superior to other candidate texts in the target text sequence; obtaining a weight value corresponding to the target text sequence based on the text lengths of other candidate texts and the text length of any candidate text; and obtaining the number of wins for any candidate text based on the weight value corresponding to the target text sequence.

[0106] The aforementioned determination of the target text sequence refers to randomly selecting one of the generated candidate texts as a comparison benchmark and constructing multiple sequences containing that text. These sequences are the target text sequences. For example, if the model generates texts A, B, C, and D for "e-commerce operation data analysis," then target text sequences can be constructed, such as sequence 1 containing texts A and B, sequence 2 containing A and C, and so on, for subsequent pairwise comparisons.

[0107] The aforementioned weighting based on text length refers to the introduction of a length control mechanism in this application to prevent the model from irrationally favoring long texts. A weight value is calculated for each text in the target text sequence. The weight calculation considers the length difference between text A and other texts in the sequence, ensuring the fairness of the evaluation results. For example, if text A is shorter than text B, the weight value will favor text A to balance the impact of length on the evaluation results.

[0108] The determination of the number of wins mentioned above is based on the sub-evaluation results of each target text sequence, that is, the text that wins in pairwise comparisons. The number of times each text is the winner is counted, which is called the number of wins. This step quantifies the quality advantage of a text relative to other candidate texts and is crucial for the model's self-evaluation.

[0109] The model first randomly selects a target text sequence, then calculates weights based on differences in text length, adjusting the evaluation weights of each text in the sequence. Through a series of pairwise comparisons, it collects sub-evaluation results and counts the number of wins for each text. The number of wins reflects the text's performance in different scenarios and is the foundational data for the model's final text quality assessment.

[0110] This application's embodiments realize the ability of a large language model to self-improve without human annotation. Through the above steps, the model can accurately identify high-quality text in open-domain tasks without standard answers, promoting the co-evolution of generation and evaluation capabilities, reducing training costs, and improving the model's generalization performance. The length control module and weight calculation ensure the model's fairness in evaluating text, avoiding bias caused by differences in text length. The statistics of the number of wins provide data for the model's self-improvement, enabling the model to continuously improve the quality of generated text through multiple iterations, achieving higher user satisfaction.

[0111] The strategy in this application effectively addresses the potential length bias issue that may exist in models when evaluating text generated for open-domain tasks. By calculating weight values, it ensures the fairness and consistency of the evaluation. The statistics of the number of wins provide the model with a clear self-improvement path, achieving text quality improvement under unsupervised conditions and promoting the model's performance in open-domain tasks.

[0112] In this embodiment, the model answers open-domain questions by generating a series of text responses and then searching for target text sequences that stand out from these responses. Specifically, for any candidate text, the model constructs multiple sequences, in which the text is considered a "target" that surpasses all other candidate texts in quality. This means that it is frequently considered a better option in pairwise comparisons within the sequence. Through this filtering, the model can identify texts that demonstrate better matching and quality in various contexts.

[0113] Next, the model further considers the length of the response text to ensure fairness in the evaluation. In the target text sequence, the length difference between each pair of texts is converted into a series of weight values. This weighting strategy aims to correct for potential length bias in the model, ensuring that whether the text is long or short, as long as its content quality is higher, it receives due recognition in the comparison.

[0114] Finally, the model counts the number of times each candidate text wins in the target text sequence based on the aforementioned weight values. The number of wins reflects how frequently a text is judged as better in a length-controlled sequence. This metric quantifies the text's advantage over other texts in the same sequence, thus providing the model with an inherent feedback mechanism to guide it in generating higher-quality responses.

[0115] This application's embodiments effectively overcome evaluation biases that models may encounter when processing open-domain tasks through a series of self-evaluation steps. First, by determining the target text sequence, the model can identify responses that perform well in various comparisons. Second, based on the calculation of weight values, the model corrects evaluation unfairness caused by differences in text length. Finally, by statistically analyzing the number of wins, the model establishes the relative advantage and quality level of the generated text, promoting continuous improvement of its output text in open-domain tasks.

[0116] This process not only allows the model to improve itself without external supervision but also significantly enhances its decision-making ability when facing open-domain problems. Through internal evaluation and adjustment, the model can generate text that better matches user intent and has higher content quality, thereby improving user experience and increasing response satisfaction. Furthermore, since the entire process does not require manual annotation or a dedicated reward model, it significantly reduces training costs, accelerates model iteration, and enhances the model's adaptability and generalization ability to various open-domain tasks.

[0117] In the above embodiments of this application, the method further includes: for any candidate text sequence, determining a target candidate text in the candidate text sequence based on the sequence evaluation result corresponding to the candidate text sequence, wherein the target candidate text is superior to another candidate text in the candidate text sequence; sorting the target candidate text and the other candidate text based on the text evaluation result corresponding to the target candidate text and the text evaluation result corresponding to the other candidate text to obtain a sorting result; and determining an evaluation quality index corresponding to any candidate text sequence based on the sorting result, wherein the evaluation quality index is used to characterize whether the sequence evaluation result corresponding to any candidate text sequence is consistent with the sorting result.

[0118] The aforementioned candidate text sequence refers to the set of multiple response texts generated by the large language model in response to a given input prompt.

[0119] The sequence evaluation results described above are a summary of the evaluations performed on each text in the candidate text sequence, reflecting the relative quality of each text. This result stems from the model's self-scoring of each text and the pairwise comparison process.

[0120] The aforementioned target candidate text is the text that is identified as superior to other candidate texts in the candidate text sequence in a specific dimension (such as: information richness, logical coherence, user satisfaction).

[0121] The above text evaluation results are quantitative scores for each target candidate text, based on the model's self-evaluation and performance in pairwise comparisons.

[0122] The above ranking results are the result of rearranging the relative positions of each candidate text in the sequence based on the text evaluation results, reflecting the different levels of text quality.

[0123] The aforementioned evaluation quality metrics are used to measure the consistency between the sequence evaluation results and the ranking results, ensuring that the model's self-evaluation mechanism can accurately reflect the true quality of the text. This metric helps improve the consistency and reliability of the model as an evaluator, avoiding text selection errors caused by evaluation bias.

[0124] Starting with any candidate text sequence, the model identifies at least one target candidate text based on the sequence evaluation results. This target text exhibits a higher quality level than another candidate text in the sequence. The model further uses the evaluation results of both the target candidate text and the other text in the sequence to rank them, generating a ranking result. This process ensures that all candidate texts in the sequence are arranged from highest to lowest quality. Finally, the model calculates an evaluation quality index to verify the consistency between the sequence evaluation results and the ranking results. If the target candidate text is indeed ranked before the other text, the model's evaluation mechanism is effective; otherwise, the model needs to adjust its evaluation strategy to reduce evaluation bias.

[0125] This application implements a self-evaluation and text quality improvement process that requires no external supervision. In open-domain tasks, the text generated by the model often faces the challenge of not having a definitive standard answer. The evaluation mechanism proposed in this application ensures that the model can accurately identify and prioritize higher-quality text even under unsupervised conditions, thereby providing users with more accurate and satisfactory responses. Evaluation quality metrics monitor the rationality of the model's self-evaluation, avoiding potential evaluation errors and ensuring the accuracy and reliability of the sequence evaluation results. By combining the ranking results with the evaluation quality metrics, the model can more fairly select high-quality text suitable for open-domain tasks, reducing selection bias caused by irrational preferences. This application relies on the model's self-evaluation, requiring no manual intervention or external verification, reducing the cost of training and iteration, and accelerating the model's self-improvement speed. Because the evaluation mechanism is not limited to specific scenarios or tasks, the model's performance in open-domain tasks is significantly improved, enabling it to more broadly adapt to various queries and instructions.

[0126] For example, in an e-commerce data analysis scenario, suppose a user asks "how to improve store conversion rate". The model then generates four candidate texts: Text 1 analyzes in detail strategies for improving product detail pages; Text 2 discusses price strategy adjustments; Text 3 mentions the importance of customer review management; and Text 4 briefly describes methods for improving the logistics experience. In the subsequent sequence evaluation, the model identifies Text 1 as a target candidate text because its performance in the sequence is significantly better than Text 4. Based on the text evaluation results of Text 1 and Text 4, the model ranks them, and the results show that Text 1 is indeed superior to Text 4 in quality. Finally, when calculating the evaluation quality index, the model finds that the sequence evaluation results are consistent with the ranking results, confirming the effectiveness of the evaluation mechanism. This process demonstrates the ability of this application embodiment to automatically identify and improve text quality without human intervention, providing users with more accurate and effective strategies for improving store conversion rates.

[0127] In any given sequence of candidate texts, the model identifies at least one target candidate text based on the sequence evaluation results. These texts are considered superior in quality to the other candidate texts in the sequence. The sequence evaluation results are a comprehensive score derived from the model's self-comparison and evaluation of the candidate texts, reflecting the relative strength of the texts. Next, the model ranks the target candidate text and another candidate text in the sequence based on their text evaluation results, i.e., their scores in the model's self-evaluation system. By comparing these scores, the model arrives at a ranking indicating which text should be placed earlier in the sequence. Finally, the model calculates an evaluation quality metric based on the ranking results. This metric measures the consistency between the sequence evaluation results and the ranking results. In short, if a target candidate text is judged to be superior in the sequence evaluation and is indeed ranked ahead of another text, the evaluation quality metric reflects the effectiveness and consistency of the model's self-evaluation mechanism.

[0128] By identifying target candidate texts through sequence evaluation and then ranking them in conjunction with text evaluation results, the model can more accurately identify high-quality texts, avoiding errors that may be introduced by evaluating only a single text. The calculation of evaluation quality metrics ensures that the model's evaluation results match the ranking results, thus verifying the fairness of the model's evaluation mechanism and avoiding evaluation bias caused by irrational preferences (such as text length, frequency of occurrence, etc.). This entire process provides the model with an adaptive improvement path, enabling it to continuously adjust and improve its self-evaluation capabilities under unsupervised conditions, thereby improving the overall quality of the generated text and more accurately meeting the needs of open-domain tasks.

[0129] In the above embodiments of this application, determining the evaluation quality index corresponding to any candidate text sequence based on the ranking result includes: when the target candidate text is located before another candidate text in the ranking result, determining the evaluation quality index corresponding to any candidate text sequence indicates that the sequence evaluation result corresponding to any candidate text sequence is consistent with the ranking result; when the target candidate text is located after another candidate text in the ranking result, determining the evaluation quality index corresponding to any candidate text sequence indicates that the sequence evaluation result corresponding to any candidate text sequence is inconsistent with the ranking result.

[0130] The ranking results above are based on the text evaluation results of the model after comparing two candidate texts. If the model determines that one text is of higher quality than the other, the former will occupy a higher position in the ranking results.

[0131] The aforementioned evaluation quality metrics are used to quantify the consistency between the sequence evaluation results and the ranking results, reflecting the effectiveness of the model's self-evaluation mechanism. When the target candidate text is ranked before another candidate text, the evaluation quality metrics indicate that the sequence evaluation results are consistent with the ranking results; conversely, when the target candidate text is ranked lower, the evaluation quality metrics indicate that the two results are inconsistent.

[0132] After completing the sequence evaluation of candidate texts, the model compares the target candidate text in pairs with other candidate texts and determines the ranking based on their text evaluation results, i.e., which text should be ranked first. Based on the ranking results, this application embodiment clarifies the definition of the evaluation quality index. If the target candidate text is indeed ranked higher in the ranking results, i.e., before another candidate text, this indicates that the sequence evaluation result is consistent with the ranking result, and the evaluation quality index will reflect that the model's self-evaluation mechanism is working properly; conversely, if the latter is not the case, it indicates that the mechanism may have a bias, and the evaluation quality index will reflect this inconsistency.

[0133] Through the above steps, the embodiments of this application ensure that the model's self-evaluation in open-domain tasks is both accurate and fair. The setting of evaluation quality metrics provides the model with an inherent feedback mechanism to monitor and adjust its evaluation algorithm, preventing irrational biases such as excessive preference for long texts or text in specific locations. This self-monitoring and adjustment capability helps the model continuously improve the quality of generated text without human intervention, better meeting user needs and expectations, while reducing resource waste caused by evaluation bias and improving the efficiency of model self-learning and iteration.

[0134] This application provides a self-evaluation quality control mechanism that requires no external intervention. By evaluating quality indicators, the model can automatically verify the rationality and consistency of its evaluation results, thereby continuously improving the quality of generated text in open-domain tasks and meeting diverse user needs. This effect is independent of other features and technical steps in this application, and is mainly reflected in the model's ability to self-regulate, ensuring the fairness and accuracy of the evaluation process, avoiding unnecessary resource consumption, and accelerating the model's learning and improvement process.

[0135] This application's embodiments, by setting evaluation quality indicators, achieve real-time monitoring and adjustment of the accuracy and consistency of the model's self-evaluation mechanism, ensuring the quality of the generated text during open-domain task processing. This mechanism is independent of specific task types or scenarios, providing a universal quality control standard for the model, which helps improve the model's performance and adaptability in various applications. It also reduces the cost of manual annotation and supervision, promoting the automation and intelligence of model self-improvement.

[0136] In this embodiment, considering the characteristics of open-domain tasks, the model employs an innovative self-evaluation mechanism to ensure the optimal ranking of the target text within the generated candidate text sequence. This mechanism verifies the degree of fit between the model's internal evaluation results and the ranking results by calculating evaluation quality indicators, thereby ensuring the fairness and accuracy of the evaluation process.

[0137] First, the model self-evaluates the generated candidate texts, obtaining a sequence evaluation result. Then, the model compares the target candidate text pairwise with other texts in the sequence, determining the ranking based on their respective evaluation results—that is, which text should be ranked first. The key to this process is that when the target candidate text ranks ahead of another candidate text, the evaluation quality metric reflects the consistency between the sequence evaluation result and the ranking result, meaning the model accurately selected the higher-quality text. Conversely, if the target text ranks lower, the model recognizes a bias in its evaluation mechanism and needs to correct it by adjusting parameters or algorithms; in this case, the evaluation quality metric indicates an inconsistency between the two results.

[0138] By evaluating quality metrics, the model can automatically detect and correct potential biases in the evaluation process, such as length bias or positional effects, ensuring more accurate evaluation results. Even in open-domain environments lacking external supervision or standard answers, evaluation quality metrics help the model maintain evaluation consistency and stability, avoiding irrational choices and improving the overall quality of the response text. The model can autonomously adjust its generation and evaluation strategies based on the indicators provided by the evaluation quality metrics, achieving self-improvement. This mechanism reduces reliance on externally labeled data, lowers training costs, and simultaneously improves the model's generalization ability and its ability to quickly adapt to new scenarios.

[0139] After receiving a user query, the model generates multiple candidate responses. To identify the target candidate text, the model first performs sequence evaluation, assigning a quality score to each response. Then, the model performs pairwise comparisons, determining the order of the target text relative to another text based on the text evaluation results. Finally, by calculating an evaluation quality metric, the model checks the consistency between the sequence evaluation results and the ranking results; this step ensures the optimal position of the target text. If the target text is indeed ranked highly, the evaluation mechanism is effective, and the evaluation quality metric reflects this positive outcome. Conversely, if the target text lags behind in the ranking, the model recognizes an evaluation bias, and the evaluation quality metric reveals an inconsistency, prompting the model to make necessary self-adjustments to improve future evaluations.

[0140] For example, consider an e-commerce diagnostic scenario where a user asks, "How can I improve my store's visitor conversion rate?" The model generates three candidate texts: Text 1 details strategies for improving product detail pages; Text 2 provides suggestions for improving customer service response speed; and Text 3 focuses on improving advertising. The model self-evaluates and determines Text 1 as the preferred target. Next, the model compares the evaluation results of Text 1 and Text 2, and the ranking shows that Text 1 is superior to Text 2. In this case, the evaluation quality metric reflects the consistency between the sequence evaluation results and the ranking results, proving the effectiveness of the model's evaluation mechanism. On the other hand, if the ranking results show that Text 1 is inferior to Text 2, the evaluation quality metric will reveal an inconsistency between the model's internal evaluation and the ranking results, triggering the model to self-adjust in order to more accurately evaluate text quality in the future. Through this mechanism, even without explicit standard answers or human preference labeling, the model can generate and select the high-quality text that best meets user needs in open-domain tasks.

[0141] This application presents a training framework for introspective reinforcement learning (SERL) suitable for Large Language Models (LLMs) in open-domain tasks. By allowing the model to simultaneously act as both a generator (Actor) and a judge, SERL achieves the co-evolution of model generation and self-evaluation capabilities without the need for external supervision, human feedback (HF), or specialized reward models (SRM). This application reduces training costs and data dependencies, improves model iteration efficiency, and enables LLMs to exhibit superior performance in open-domain tasks without standard answers, such as data analysis and open-ended writing. Furthermore, SERL's versatility allows it to cover long-tail scenarios, enhancing the model's generalization ability and flexibility.

[0142] Figure 3This is a schematic diagram of the framework of a text processing method according to an embodiment of this application, such as... Figure 3 As shown, during the generation phase (Actor), for each input cue, the model can sample N diverse responses from Q. , … , In the judging phase, for each response, the model generates K pairwise comparison judgments. , … , The algorithm employs a Position Bias Mitigation Mechanism (PBMM) to eliminate position bias. Specifically, during the calculation of each response pair, if the lengths of the two responses in a pair differ significantly, the entire pair is discarded without further processing. Furthermore, the win rate of each response is calculated based on the length control module. This is used to adjust the win rate calculation results; if the winner of a certain judgment ranks higher than the loser in the global ranking, then that judgment receives a positive consistency reward. The global ranking refers to the results sorted according to the win rate of each response. This is specifically achieved through the following formula:

[0143] ;

[0144] ;

[0145] in, For generator reward correction function, For length balance factor, For the above response, To win the response, For failure response, The total number of responses in the response set, The number of pairwise comparisons. Indicates the total number of comparisons.

[0146] like Figure 3 As shown, further, an improved Group Reward Policy Optimization (GRPO) function is adopted, which simultaneously improves the probabilistic strategies for generating responses and making judgments, eliminating the need for the current divergence constraint (KL divergence constraint). The pairwise comparison ranking method (Copeland) can be used to determine the win rate of each response in pairwise comparisons. , … , }, and sort them, so that the sorting result { , … , Improvements were made to the generation phase; during the evaluation phase, when one response is determined to be superior to another in pairwise comparisons, a consistency check verifies whether this determination is consistent with the global Copeland ranking. , … , If the winner is indeed ranked higher, then this judgment will be considered "consistent"; otherwise, it will be considered "inconsistent," and the ranking will proceed accordingly, thus sorting the results based on consistency. , … , The evaluation phase has been improved. This is specifically represented by the following formula:

[0147] ;

[0148] in, For the joint objective function of the SER1 framework, Represents the parameters of the model. The expected value operator is used to calculate the average value of a random variable under a certain probability distribution. q is a sample drawn from the input prompt distribution Q, representing a specific task input. It is the nth generated response sampled from the Actor's old policy πold on the input q, where N is the number of responses generated by the Actor, and 1 / N represents the average contribution of the N responses. It is the agent objective function of the Actor, used to calculate the generated response. The degree of improvement compared to the original generation strategy πold, where K / / represents the number of comparisons required for each pair of responses, and M is the number of response pairs that the Judge needs to evaluate. It is the surrogate objective function of Judge, used to quantify the consistency and accuracy of Judge's judgments in pairwise comparisons.

[0149] ;

[0150] in, This represents a proxy objective function for the model's role (R) when processing a specific output (o). Choosing the smaller value between the two is typically used to control the magnitude of the gradient and avoid gradient explosion. It is the ratio of the probability of generating or judging a specific token under the new parameters θ to the probability under the old parameters θ_{old}. It reflects the change in the new policy's preference for a specific action and represents the relative probability between the new and old policies when performing a specific action. This involves subtracting the average reward of all responses in the group from Ri,t, and then dividing by the standard deviation. The purpose of this is to eliminate absolute differences in rewards and highlight the strength of each response / judgment relative to other members of the group. Clipping function, used to limit probability ratio In parameter 1- and 1+ In between, further stabilize the training process.

[0151] ;

[0152] in, This represents the normalized Actor advantage function. The actor's original reward, and Let represent the mean and standard deviation of the rewards for the N Actors, respectively, which are used to normalize the rewards and ensure the stability of the advantage function.

[0153] ;

[0154] in, The representation is the normalized dominance function of Judge at time t, based on the current pairwise judgment J(i,j),k,t, when performing the k-th pairwise comparison of the i-th and j-th generated responses. For Iudge's original consistency reward, Let represent the mean of the consistency rewards for M×K Judges. Let represent the standard deviation of the M×K Judge consistency rewards.

[0155] ;

[0156] in, This represents the relative probability ratio of the generator for the nth response at time step t, based on the current parameter θ. This represents the input q and the previously generated text. At that time, generate the next word under the current parameter θ. The probability, . represents the input q and the previously generated text. At that time, the next word is generated under the historical parameter θold. The probability of.

[0157] ;

[0158] in, This represents the probability ratio of the Judge module (the evaluator) at the current time step t to generate a specific evaluation decision (i.e., determining which response is better) when making the k-th pairwise comparison between the i-th and j-th responses, using the new parameter θ and the old parameter θold. This indicates that given the input q and the judgments made previously... When the current parameter θ is θ, make the next judgment. The probability, This indicates that given the input q and the judgments made previously... At that time, make the next judgment based on the historical parameters θ. The probability of.

[0159] In the introspective reinforcement learning framework, three key mechanisms are employed to improve the model's performance on open-domain tasks, ensuring that the generated responses are not only of high quality but also that the evaluation process is fair and reliable. First, the Position Bias Mitigation Mechanism (PBMM) effectively mitigates position bias by randomly swapping the order of responses in pairwise comparisons, thus preventing the model from being influenced by the response order when evaluating responses and ensuring that each response receives a fair evaluation. Second, the introduction of the Length-Controlled Module (LCM) controls the output length of the generated path, preventing excessively long responses from being mistakenly considered higher quality. This step ensures that the model's evaluation is not solely based on the length of the response but focuses more on its actual semantic content and structure, thereby avoiding the model's tendency to generate redundant information.

[0160] Finally, the SERL framework employs a pure reinforcement learning training approach, eliminating the need for a cold-start chain of thought (CoT) or external reward models. This means the model can evolve solely based on tool feedback and self-evaluation, significantly reducing training costs and reliance on manual annotation, and enhancing its ability to handle highly generalized tasks in open-domain environments. The core of this mechanism is to allow the model to simultaneously act as a generator and evaluator, co-evolving within a self-contained closed loop. It improves its performance on open-domain tasks through a self-reinforcement learning path, without requiring an additional cold-start phase to train inference capabilities; instead, it relies on the model's ability to generate and evaluate responses to guide its evolutionary direction.

[0161] In summary, the combination of PBMM, LCM, and pure RL training mechanisms forms a key strategy within the SERL framework for improving the performance of LLM in open-domain tasks. By systematically addressing positional and length bias issues and achieving efficient self-supervised training, it significantly enhances the model's generation and evaluation capabilities, realizing the co-evolution of generation and evaluation abilities. This enables SERL to provide more accurate, efficient, and cost-effective solutions for open-domain tasks such as data analysis, summary generation, and open-ended writing.

[0162] SERL cleverly integrates multiple innovative mechanisms, significantly improving the performance and training efficiency of large language models on open-domain tasks. By introducing self-consistent reward (RJ), SERL significantly improves the decision consistency of the judge, enhances feedback quality, and solves the stability problem in the evaluation process. Simultaneously, the use of the length control module (LCM) and the position bias mitigation mechanism (PBMM) eliminates potential length bias and positional bias in the model's response evaluation, ensuring the fairness and impartiality of the evaluation system and preventing the model from obtaining inappropriate scores by producing lengthy text or relying on output position. SERL's self-contained nature eliminates the need for expensive manual annotation during training, significantly reducing costs and broadening its applicability to various open-domain tasks. Employing the improved GRPO strategy with within-group advantage probability, SERL achieves efficient online training, demonstrating significant performance improvements even within a limited number of training steps.

[0163] This application utilizes the same model to alternately act as both Actor and Judge, achieving collaborative online evolution of generation and evaluation capabilities. Copeland's pairwise comparison and self-consistent reward mechanism can construct reliable, differentiable bi-objective reward signals without supervision, requiring no external preference data. The Position Bias Mitigation Mechanism (PBMM) and Length Control Module (LCM) systematically address common bias issues when LLMs act as judges, ensuring evaluation fairness. This general self-boosting paradigm, applicable to any open-domain task, does not rely on reference answers, human annotations, or validators, exhibiting strong generalization and scalability.

[0164] For the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. This is because, according to this application, certain steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, or by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0166] Figure 4 This is a flowchart of a training method for a text processing model according to an embodiment of this application, such as... Figure 4 As shown, the method includes:

[0167] Step S402: Obtain the training text.

[0168] The steps described above are used to collect input data for model training, i.e., training text. This text can come from various open-domain scenarios, including but not limited to user questions, dialogue records, and article paragraphs. Training text is the foundation for the model to generate predicted text and is crucial for improving the model's generalization ability and response diversity.

[0169] Step S404: Input the training text into the initial processing model and use the initial processing model to generate multiple predicted texts.

[0170] After receiving the training text, the model enters the generation phase. In the initial processing, the model generates a series of predicted texts based on the training text; each predicted text represents an attempt at a possible response to the input text. This process aims to explore multiple response strategies, ensuring that the generated text covers a wide range of possibilities and provides diverse samples for the subsequent evaluation phase.

[0171] Step S406: Input the training text and multiple predicted texts into the initial processing model, and use the initial processing model to evaluate the multiple predicted texts respectively to obtain the first evaluation results corresponding to the multiple predicted texts.

[0172] The first evaluation result is used to characterize the evaluation value of the corresponding predicted text, which is the win rate relative to the evaluation values ​​of other predicted texts. The evaluation value is the degree of matching between the corresponding predicted text and the training text.

[0173] In the above steps, the model performs self-evaluation using training text and multiple predicted texts. The initial processing model acts as a judge, generating a first evaluation result for each predicted text. This result reflects the degree of matching between the predicted text and the training text, as well as its relative quality compared to other predicted texts. This evaluation process is based on the model's own judgment criteria, requiring no external verification or manual annotation, demonstrating the autonomy and efficiency of the embodiments in this application.

[0174] Step S408: Based on the first evaluation results corresponding to multiple predicted texts, train the initial processing model to obtain the text processing model.

[0175] The text processing model is used to execute any of the methods in the above embodiments.

[0176] After collecting the initial evaluation results of all predicted texts, the model enters the training phase. Through backpropagation, the model adjusts its parameters based on these evaluation results to improve its ability to generate and evaluate text. The ultimate goal of this process is to obtain a text processing model that can more accurately generate high-quality text that meets user needs and can more reliably self-evaluate the quality of the generated text, thus achieving co-evolution of text generation and quality evaluation.

[0177] Through the steps described above, this application establishes a closed-loop self-improvement mechanism. The model can generate, evaluate, and improve predicted text under unsupervised conditions, which significantly reduces training costs and improves the model's iterative efficiency. The introduction of evaluation quality metrics ensures that the model continuously improves its quality when generating text and maintains consistency and fairness when evaluating text.

[0178] Through the above steps, training text is obtained; the training text is input into the initial processing model, and multiple predicted texts are generated using the initial processing model; the training text and multiple predicted texts are input into the initial processing model, and the initial processing model is used to evaluate the multiple predicted texts respectively, obtaining the first evaluation result corresponding to each of the multiple predicted texts. The first evaluation result is used to characterize the evaluation value of the corresponding predicted text, which is the win rate relative to the evaluation values ​​of other predicted texts. The evaluation value is the degree of matching between the corresponding predicted text and the training text; based on the first evaluation results corresponding to the multiple predicted texts respectively, the initial processing model is trained to obtain a text processing model. By comparing the evaluation results of candidate texts in pairs, the relative advantage frequency of each candidate text when compared with other candidate texts is counted to determine the comprehensive evaluation value of each candidate text. The response text corresponding to the query text is determined from multiple candidate texts, that is, the candidate text with the higher relative advantage frequency is selected as the response. This process improves the selection strategy of the text processing model, improves the quality of text generation, and thus solves the technical problem of poor text processing performance of the model in related technologies.

[0179] In the above embodiments of this application, an initial processing model is trained based on the first evaluation results corresponding to multiple predicted texts to obtain a text processing model, including: obtaining multiple first reward values ​​based on the first evaluation results corresponding to multiple predicted texts; obtaining multiple second reward values ​​based on the first evaluation results corresponding to multiple predicted texts and the second evaluation results corresponding to multiple predicted text sequences, wherein the multiple predicted text sequences are constructed based on any two predicted texts among the multiple predicted texts, and the second evaluation results are used to characterize the superiority or inferiority between the evaluation values ​​corresponding to the two predicted texts in the corresponding predicted text sequences; determining the target loss function value corresponding to the initial processing model based on the multiple first reward values ​​and the multiple second reward values; and adjusting the model parameters of the initial processing model based on the target loss function value to obtain the text processing model.

[0180] The first evaluation result described above is an individual evaluation of each predicted text by the primary model, used to quantify the degree and quality of the match between the text and the training text. This result is part of the model's self-evaluation and is used to guide the generation of the first reward value.

[0181] The aforementioned first reward value is a reward signal calculated based on the first evaluation result. It is used to represent the model's direct evaluation of the predicted text and reflects the individual quality of the predicted text.

[0182] The aforementioned predicted text sequence consists of two predicted texts, used for subsequent pairwise comparison evaluation. This sequence is constructed by randomly selecting two predicted texts and combining them as the basis for evaluation and comparison, reflecting the relative merits of the multiple texts.

[0183] The second evaluation result mentioned above is the evaluation obtained by the model after performing a pairwise comparison of two texts in the predicted text sequence. It is used to judge the relative quality and superiority between the two texts and is the basis for constructing the second reward value.

[0184] The aforementioned second reward value is a reward signal calculated based on the second evaluation result. It is used to represent the model's judgment on the relative quality of the two predicted texts and to further guide the adjustment of model parameters.

[0185] The aforementioned target loss function value, which comprehensively considers the results of the first and second reward values, is used to guide the adjustment of model parameters. This function value reflects the overall performance of the model in generating predicted text from the training text under the current parameter settings, serving as an improvement target during the training process.

[0186] Based on the target loss function value, this embodiment of the application adjusts the model's parameters using machine learning techniques such as backpropagation to improve the model's text generation ability and the consistency of its self-evaluation. After adjustment, the initial processing model gradually evolves into a text processing model with more efficient self-improvement capabilities.

[0187] This application provides a method for training a text processing model that can improve itself without external supervision or manual annotation. First, through a first evaluation result and a second evaluation result, the model can evaluate the quality of the predicted text from multiple perspectives, considering both the quality of individual texts and the relative merits between texts. Second, the introduction of first and second reward values ​​allows the model to receive feedback at both the individual and group levels, guiding parameter adjustments. Finally, the calculation of the objective loss function ensures the overall guidance of the training process, helping the model maintain text quality while improving the diversity and adaptability of the generated text.

[0188] This application's embodiments are geared towards open-domain tasks, aiming to enable Large Language Models (LLMs) to improve themselves without additional supervision. For example, given the training text "improve store conversion rate," the model generates multiple predicted texts, including improving product detail pages, enhancing customer service quality, and refining advertising strategies. During the evaluation phase, the model generates a first evaluation result for each text, reflecting its quality. Simultaneously, it performs pairwise comparisons of predicted text sequences constructed from any two texts, generating a second evaluation result to determine their relative merits. By calculating the first and second reward values, the model establishes a target loss function value, guiding parameter adjustments and ultimately forming a text processing model better suited for generating high-quality text that meets user needs.

[0189] In the above embodiments of this application, multiple second reward values ​​are obtained based on the first evaluation results corresponding to multiple predicted texts and the second evaluation results corresponding to multiple predicted text sequences, including: for any predicted text sequence, determining the target predicted text in the predicted text sequence based on the second evaluation result corresponding to the predicted text sequence, wherein the target predicted text is superior to another predicted text in the predicted text sequence; sorting the target predicted text and the other predicted text based on the first evaluation result corresponding to the target predicted text and the first evaluation result corresponding to the other predicted text to obtain a training sorting result; and determining the second reward value corresponding to any predicted text sequence based on the training sorting result, wherein the second reward value is used to characterize whether the second evaluation result corresponding to any predicted text sequence is consistent with the training sorting result.

[0190] The predicted text sequence mentioned above consists of two predicted texts generated by the model, which are used for subsequent pairwise comparison evaluation.

[0191] The second evaluation result mentioned above is the evaluation result obtained by the model after performing a pairwise comparison of two texts in the predicted text sequence, reflecting the superiority or inferiority between the two texts, i.e., which text is better.

[0192] The target predicted text mentioned above is the superior predicted text determined based on the second evaluation result within any predicted text sequence. This text is considered to have a higher evaluation value and is of higher quality compared to another predicted text in the sequence.

[0193] The first evaluation result mentioned above is the model's evaluation of a single predicted text, used to measure the individual quality of that text, its matching degree with the training text, and its advantage relative to other predicted texts.

[0194] The training ranking results described above are the result of the model ranking the target predicted text and another predicted text in the sequence based on the first evaluation results. This ranking reflects the model's own judgment on the quality of the predicted text, i.e., which text should be ranked first.

[0195] The second reward value mentioned above is used to measure the consistency between the second evaluation result and the training ranking result, i.e., whether the target predicted text ranks higher in the sequence. This reward value provides feedback to the model, guiding it on how to adjust to more accurately evaluate text quality.

[0196] For any given sequence of predicted text, the model first determines which predicted text in the sequence is better based on the second evaluation result, thus identifying the target predicted text. This step ensures that the model can identify and prioritize higher-quality text. Then, the model ranks the target predicted text and another predicted text in the sequence using the first evaluation result, generating a training ranking. This ranking process is based on the model's own quality evaluation criteria and aims to reflect the quality level of individual texts. Finally, the model determines a second reward value based on the training ranking result. If the ranking result shows that the target predicted text is indeed ranked higher, the second reward value indicates that the model's evaluation process is consistent with the ranking result; conversely, if the target predicted text is ranked lower, the second reward value will reflect an inconsistency between the evaluation and the ranking, suggesting that the model needs to adjust its evaluation strategy.

[0197] The method provided in this application enables the model to self-evaluate the quality of generated text under unsupervised conditions and continuously adjust and improve the evaluation accuracy through a second reward value feedback mechanism. This method not only reduces dependence on external labeled data and training costs, but also significantly improves the model's generalization ability and self-improvement efficiency, ensuring the quality and diversity of generated text in open-domain tasks.

[0198] The method described in this application enables the model to autonomously determine the quality of predicted text and continuously improve the evaluation strategy through a second reward mechanism. For example, in response to the query "improve conversion rate," the model generates the aforementioned three predicted texts. By comparing the predicted text sequence in pairs, the model identifies the first text as the target predicted text. The ranking result based on the first evaluation shows the priority of the first text, and the second reward confirms the effectiveness of the model's evaluation mechanism. The entire process allows the model to automatically identify and prioritize high-quality predicted texts without human intervention, improving the accuracy of the generated text, enhancing the model's performance and generalization ability on open-domain tasks, and significantly reducing the cost of training and iteration.

[0199] According to embodiments of this application, a text processing apparatus for implementing the above-described text processing method is also provided. Figure 5 This is a schematic diagram of a text processing apparatus according to an embodiment of this application, such as... Figure 5 As shown, the device 500 includes: an acquisition module 502, a generation module 504, an evaluation module 506, and a determination module 508.

[0200] The system comprises the following modules: an acquisition module for acquiring the query text; a generation module for inputting the query text into a text processing model and generating multiple candidate texts using the model; an evaluation module for inputting the query text and multiple candidate texts into the text processing model, evaluating each candidate text using the model, and obtaining text evaluation results for each candidate text. The text evaluation results represent the evaluation value of the corresponding candidate text, its win rate relative to the evaluation values ​​of other candidate texts, and the degree of matching between the candidate text and the query text; and a determination module for determining the response text corresponding to the query text from the multiple candidate texts based on the text evaluation results.

[0201] The acquisition module 502, generation module 504, evaluation module 506, and determination module 508 described above correspond to steps S202 to S208 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. These modules may also be part of a device and may run in the server 10 provided in the above embodiments.

[0202] In the above embodiments of this application, the evaluation module is used to match multiple candidate texts with the query text using a text processing model to obtain evaluation values ​​corresponding to the multiple candidate texts respectively; to determine multiple candidate text sequences based on any two candidate texts among the multiple candidate texts; to determine the sequence evaluation result corresponding to any candidate text sequence based on the evaluation values ​​corresponding to the two candidate texts respectively in any candidate text sequence, wherein the sequence evaluation result is used to characterize the superiority or inferiority between the evaluation values ​​corresponding to the two candidate texts respectively; and to determine the text evaluation result corresponding to the multiple candidate texts respectively based on the sequence evaluation results corresponding to the multiple candidate text sequences respectively.

[0203] In the above embodiments of this application, the evaluation module is used to obtain any one candidate text from multiple candidate texts to obtain a first candidate text; to obtain any one candidate text from candidate texts other than the first candidate text to obtain a second candidate text; and to construct two candidate text sequences based on the first candidate text and the second candidate text, wherein the first candidate text and the second candidate text correspond to different orders in the two candidate text sequences.

[0204] In the above embodiments of this application, the evaluation module is used to obtain the ratio between the text length of the first candidate text and the text length of the second candidate text to obtain a target ratio; when the target ratio is within a preset ratio range, two candidate text sequences are constructed based on the first candidate text and the second candidate text.

[0205] In the above embodiments of this application, the determining module is used to determine a first candidate text and a second candidate text in any candidate text sequence; if the evaluation value corresponding to the first candidate text is greater than the matching degree corresponding to the second candidate text, the corresponding sub-evaluation result is determined to indicate that the first candidate text is better than the second candidate text; if the matching degree corresponding to the second candidate text is greater than the matching degree corresponding to the first candidate text, the corresponding sub-evaluation result is determined to indicate that the second candidate text is better than the first candidate text.

[0206] In the above embodiments of this application, the determining module is used to determine at least one candidate text sequence containing any one of the multiple candidate texts; based on the sub-evaluation results corresponding to the at least one candidate text sequence, determine the number of times any candidate text wins, wherein the number of times a candidate text wins is used to characterize the number of times a candidate text is better than other candidate texts; and based on the number of times any candidate text wins and the number of at least one candidate text sequence, obtain the text evaluation result corresponding to any candidate text.

[0207] In the above embodiments of this application, the determining module is used to determine a target text sequence from at least one candidate text sequence, wherein any one text is superior to other candidate texts in the target text sequence; based on the text lengths of other candidate texts and the text length of any one candidate text, a weight value corresponding to the target text sequence is obtained; based on the weight value corresponding to the target text sequence, the number of times any one candidate text wins is obtained.

[0208] In the above embodiments of this application, the device is further configured to, for any candidate text sequence, determine a target candidate text in the candidate text sequence based on the sequence evaluation result corresponding to the candidate text sequence, wherein the target candidate text is superior to another candidate text in the candidate text sequence; sort the target candidate text and the other candidate text based on the text evaluation result corresponding to the target candidate text and the text evaluation result corresponding to the other candidate text to obtain a sorting result; and determine an evaluation quality index corresponding to any candidate text sequence based on the sorting result, wherein the evaluation quality index is used to characterize whether the sequence evaluation result corresponding to any candidate text sequence is consistent with the sorting result.

[0209] In the above embodiments of this application, the device is further configured to, when the target candidate text is located before another candidate text, determine that the evaluation quality index corresponding to any candidate text sequence indicates that the sequence evaluation result corresponding to any candidate text sequence is consistent with the ranking result; and when the target candidate text is located after another candidate text in the ranking result, determine that the evaluation quality index corresponding to any candidate text sequence indicates that the sequence evaluation result corresponding to any candidate text sequence is inconsistent with the ranking result.

[0210] According to an embodiment of this application, a training apparatus for a text processing model for implementing the above-described text processing model training method is also provided. Figure 6 This is a schematic diagram of a training apparatus for a text processing model according to an embodiment of this application, as shown below. Figure 6 As shown, the device 600 includes: an acquisition module 602, a generation module 604, an evaluation module 606, and a training module 608.

[0211] The acquisition module is used to acquire training text; the generation module is used to input the training text into the initial processing model and generate multiple predicted texts using the initial processing model; the evaluation module is used to input the training text and multiple predicted texts into the initial processing model, evaluate the multiple predicted texts using the initial processing model, and obtain a first evaluation result corresponding to each of the multiple predicted texts, wherein the first evaluation result is used to characterize the evaluation value of the corresponding predicted text, the win rate relative to the evaluation values ​​of other predicted texts, and the evaluation value is the degree of matching between the corresponding predicted text and the training text; the training module is used to train the initial processing model based on the first evaluation results corresponding to the multiple predicted texts to obtain a text processing model, wherein the text processing model is used to execute the method of any one of claims 1 to 9.

[0212] The acquisition module 602, generation module 604, evaluation module 606, and training module 608 described above correspond to steps S402 to S408 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. These modules may also be part of a device and may run in the server 10 provided in the above embodiments.

[0213] In the above embodiments of this application, the training module is used to obtain multiple first reward values ​​based on the first evaluation results corresponding to multiple predicted texts respectively; to obtain multiple second reward values ​​based on the first evaluation results corresponding to multiple predicted texts respectively and the second evaluation results corresponding to multiple predicted text sequences, wherein the multiple predicted text sequences are constructed based on any two predicted texts among the multiple predicted texts, and the second evaluation results are used to characterize the superiority or inferiority between the evaluation values ​​corresponding to the two predicted texts in the corresponding predicted text sequences; to determine the target loss function value corresponding to the initial processing model based on the multiple first reward values ​​and the multiple second reward values; and to adjust the model parameters of the initial processing model based on the target loss function value to obtain the text processing model.

[0214] In the above embodiments of this application, the training module is used to determine, for any given predicted text sequence, a target predicted text in the given predicted text sequence based on the second evaluation result corresponding to the given predicted text sequence, wherein the target predicted text is superior to another predicted text in the given predicted text sequence; sort the target predicted text and the other predicted text based on the first evaluation result corresponding to the target predicted text and the first evaluation result corresponding to the other predicted text to obtain a training sorting result; and determine a second reward value corresponding to any given predicted text sequence based on the training sorting result, wherein the second reward value is used to characterize whether the second evaluation result corresponding to any given predicted text sequence is consistent with the training sorting result.

[0215] The preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, and will not be repeated here.

[0216] Embodiments of this application may provide a computing device. Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 7 As shown, the computing device 100 may include one or more (one shown in the figure) processors 102, memory 104, memory controller, and peripheral interfaces.

[0217] The aforementioned computing device can be understood as an integrated intelligent terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), and all-in-one model machines. Furthermore, the computing device may have the model described in the above embodiments of this application pre-installed.

[0218] Specifically, this computing device can pre-install various types of models, including but not limited to models in fields such as natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model choices. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities. Models can be called into created applications through API interfaces, and application management tools are provided to control and manage applications.

[0219] Furthermore, this computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technologies), and basic control capabilities (providing enterprise-level basic control capabilities to ensure system security and efficient operation). Through these functions, it provides a comprehensive, integrated device for AI development, training, deployment, and application.

[0220] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0221] The processor can invoke an executable program stored in memory via a transmission device to execute any of the methods described in the above embodiments.

[0222] Embodiments of this application may provide an electronic device. Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 8 As shown, the electronic device may include: an input / output device 112; a memory 114; and a processor 116, wherein the processor 116 is connected to the input / output device 112 and the memory 114 via a bus 118.

[0223] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0224] The processor can invoke an executable program stored in memory via a transmission device to execute any of the methods described in the above embodiments.

[0225] Those skilled in the art will understand that, Figure 8 The structure shown is illustrative. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. This diagram does not limit the structure of the aforementioned electronic devices. For example, electronic devices may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the diagram, or have a different configuration than shown in the diagram.

[0226] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: a flash drive, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0227] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0228] Optionally, in this embodiment, the storage medium may be located in a computing device or an electronic device.

[0229] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program. When the executable program runs, it controls the device where the computer-readable storage medium is located to perform the method described in any of the above embodiments.

[0230] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program. When executed by a processor, the computer program implements the methods provided in the above embodiments.

[0231] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium can be used to store a computer program. When the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0232] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0233] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are illustrative; for example, the division of units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined, integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling, direct coupling, or communication connection shown or discussed may be through some interfaces, indirect coupling of units or modules, or communication connection, and may be electrical or other forms.

[0235] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment.

[0236] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0237] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0238] The above description represents the preferred embodiments of this application. For those skilled in the art, various improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A text processing method, characterized in that, include: Get the query text; The query text is input into a text processing model, and multiple candidate texts are generated using the text processing model. The query text and the plurality of candidate texts are input into the text processing model, and the text processing model is used to evaluate the plurality of candidate texts respectively to obtain the text evaluation results corresponding to the plurality of candidate texts. The text evaluation results are used to characterize the evaluation value of the corresponding candidate text and the win rate relative to the evaluation values ​​of other candidate texts. Based on the text evaluation results corresponding to the multiple candidate texts, the response text corresponding to the query text is determined from the multiple candidate texts.

2. The method according to claim 1, characterized in that, The step of evaluating the multiple candidate texts using the text processing model to obtain text evaluation results for each candidate text includes: The text processing model is used to match the multiple candidate texts with the query text to obtain the evaluation values ​​corresponding to the multiple candidate texts respectively; Based on any two candidate texts from the plurality of candidate texts, a plurality of candidate text sequences are determined; Based on the evaluation values ​​corresponding to two candidate texts in any candidate text sequence, a sequence evaluation result corresponding to any candidate text sequence is determined, wherein the sequence evaluation result is used to characterize the superiority or inferiority between the evaluation values ​​corresponding to the two candidate texts. Based on the sequence evaluation results corresponding to the multiple candidate text sequences, the text evaluation results corresponding to the multiple candidate text sequences are determined.

3. The method according to claim 2, characterized in that, The step of determining multiple candidate text sequences based on any two candidate texts from the plurality of candidate texts includes: From the plurality of candidate texts, any one candidate text is obtained to obtain the first candidate text; Obtain any candidate text from the candidate texts other than the first candidate text to obtain the second candidate text; Based on the first candidate text and the second candidate text, two candidate text sequences are constructed, wherein the first candidate text and the second candidate text are in different orders.

4. The method according to claim 3, characterized in that, The construction of two candidate text sequences based on the first candidate text and the second candidate text includes: Obtain the ratio between the text length of the first candidate text and the text length of the second candidate text to get the target ratio; When the target ratio is within a preset ratio range, the two candidate text sequences are constructed based on the first candidate text and the second candidate text.

5. The method according to claim 2, characterized in that, The step of determining the sequence evaluation result corresponding to the candidate text sequence based on the evaluation values ​​corresponding to two candidate texts in any candidate text sequence includes: Determine the first and second candidate texts in any candidate text sequence; If the evaluation value corresponding to the first candidate text is greater than the matching degree corresponding to the second candidate text, the corresponding sub-evaluation result is determined to indicate that the first candidate text is better than the second candidate text. If the matching degree of the second candidate text is greater than that of the first candidate text, the corresponding sub-evaluation result is determined to indicate that the second candidate text is superior to the first candidate text.

6. The method according to claim 2, characterized in that, The step of determining the text evaluation results corresponding to the multiple candidate text sequences based on the sequence evaluation results corresponding to the multiple candidate text sequences includes: For any one of the plurality of candidate texts, determine at least one candidate text sequence that contains the candidate text. Based on the sub-evaluation results corresponding to the at least one candidate text sequence, the number of times any candidate text wins is determined, wherein the number of times the candidate text is superior to other candidate texts; Based on the number of times any candidate text wins and the number of at least one candidate text sequence, the text evaluation result corresponding to any candidate text is obtained.

7. The method according to claim 6, characterized in that, The step of determining the number of wins for any candidate text based on the sub-evaluation results corresponding to the at least one candidate text sequence includes: A target text sequence is determined from the at least one candidate text sequence, wherein any one text is preferred over the other candidate texts in the target text sequence; Based on the text lengths of other candidate texts and the text length of any one candidate text, the weight value corresponding to the target text sequence is obtained; Based on the weight values ​​corresponding to the target text sequence, the number of times any candidate text wins is obtained.

8. The method according to any one of claims 2 to 7, characterized in that, The method further includes: For any candidate text sequence, based on the sequence evaluation result corresponding to any candidate text sequence, a target candidate text in any candidate text sequence is determined, wherein the target candidate text is superior to another candidate text in any candidate text sequence; Based on the text evaluation results corresponding to the target candidate text and the text evaluation results corresponding to the other candidate text, the target candidate text and the other candidate text are sorted to obtain a sorting result; Based on the ranking results, an evaluation quality index is determined for any candidate text sequence, wherein the evaluation quality index is used to characterize whether the sequence evaluation result corresponding to any candidate text sequence is consistent with the ranking results.

9. The method according to claim 8, characterized in that, Based on the sorting results, the evaluation quality index corresponding to any candidate text sequence is determined, including: In the ranking results, if the target candidate text is located before the other candidate text, the evaluation quality index corresponding to any candidate text sequence is determined to indicate that the sequence evaluation result corresponding to any candidate text sequence is consistent with the ranking results; In the ranking results, if the target candidate text is located after the other candidate text, the evaluation quality index corresponding to any candidate text sequence is determined to indicate that the sequence evaluation result corresponding to any candidate text sequence is inconsistent with the ranking results.

10. A method for training a text processing model, characterized in that, include: Obtain the training text; The training text is input into the initial processing model, and the initial processing model is used to generate multiple predicted texts; The training text and the plurality of predicted texts are input into the initial processing model, and the initial processing model is used to evaluate the plurality of predicted texts respectively to obtain the first evaluation result corresponding to the plurality of predicted texts respectively. The first evaluation result is used to characterize the evaluation value of the corresponding predicted text and the win rate relative to the evaluation values ​​of other predicted texts. Based on the first evaluation results corresponding to the plurality of predicted texts, the initial processing model is trained to obtain a text processing model, wherein the text processing model is used to perform the method described in any one of claims 1 to 9.

11. The method according to claim 10, characterized in that, The initial processing model is trained based on the first evaluation results corresponding to the multiple predicted texts to obtain a text processing model, including: Based on the first evaluation results corresponding to the multiple predicted texts, multiple first reward values ​​are obtained; Based on the first evaluation results corresponding to the multiple predicted texts and the second evaluation results corresponding to the multiple predicted text sequences, multiple second reward values ​​are obtained. The multiple predicted text sequences are constructed based on any two predicted texts among the multiple predicted texts. The second evaluation results are used to characterize the superiority or inferiority between the evaluation values ​​corresponding to the two predicted texts in the corresponding predicted text sequences. Based on the plurality of first reward values ​​and the plurality of second reward values, the target loss function value corresponding to the initial processing model is determined; The model parameters of the initial processing model are adjusted based on the target loss function value to obtain the text processing model.

12. The method according to claim 11, characterized in that, Based on the first evaluation results corresponding to the multiple predicted texts respectively, and the second evaluation results corresponding to the multiple predicted text sequences, multiple second reward values ​​are obtained, including: For any predicted text sequence, based on the second evaluation result corresponding to any predicted text sequence, a target predicted text in any predicted text sequence is determined, wherein the target predicted text is superior to another predicted text in any predicted text sequence; Based on the first evaluation result corresponding to the target predicted text and the first evaluation result corresponding to the other predicted text, the target predicted text and the other predicted text are sorted to obtain the training sorting result; Based on the training ranking results, a second reward value is determined for any predicted text sequence, wherein the second reward value is used to characterize whether the second evaluation result for any predicted text sequence is consistent with the training ranking results.

13. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor, connected to a memory via a bus, is used to run the program, wherein the program, when running, executes the method described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 12.

Images