Data processing methods, devices, equipment and media

By using multi-path data processing and evaluation criteria to train a labeling model, the problem that the quality of answers in intelligent body search depends on the document set is solved, thus improving the accuracy and coverage of query results.

CN123115366APending Publication Date: 2026-11-17TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611630870.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In intelligent body search, existing technologies cannot effectively address the issue that the quality of answers depends on the set-level utility of the document collection, resulting in low accuracy of query results, repeated coverage of the same aspect while ignoring key aspects.

Method used

Multiple response texts are generated through at least two data processing paths. A candidate set of recall results is generated using evaluation criteria. The model is then trained through imitation learning and reinforcement learning in conjunction with a reference model to obtain a labeling model. This model is used to generate document utility labels for the recall and ranking training phases of query information.

Benefits of technology

It improves the quality of responses to queries, ensures that the recall and ranking stages are aligned with the same answer quality goal, and enhances the accuracy and coverage of query results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115366A_ABST
    Figure CN123115366A_ABST
Patent Text Reader

Abstract

This application provides a data processing method, apparatus, device, and medium. The method includes: generating at least two first response texts for a first query information according to multiple independent data processing paths; obtaining candidate entries from the at least two first response texts; generating a first evaluation criterion for the first query information based on the candidate entries and a candidate set of recall results for the first query information; the first evaluation criterion is used to generate a set-level reward for the first query information during a selection training phase; obtaining a first training corpus set; performing supervised fine-tuning on an initial annotation model based on the first training corpus set and reference utility labels output by a reference model to obtain a candidate annotation model; performing reinforcement learning on the candidate annotation model according to the first evaluation criterion to obtain a label annotation model; and using the label annotation model to generate document utility labels for the first query information during a recall / ranking training phase. This application embodiment can improve the quality of query information responses.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] In current search scenarios, intelligent agent-based search eliminates the need for manual browsing of sorted result lists to obtain answers. Instead, it directly receives a single answer synthesized by a large model across a document set (containing only a few documents). The quality of this answer depends on the set-level utility of the document set. However, attributes affecting answer quality, such as coverage, content contradictions, and informational contradictions, only become apparent after the documents are grouped into a set, exceeding the expressive power of document-by-document relevance scoring. Therefore, even if the relevance score of each document in the document set is accurate, the answer synthesized by the large model may still repeatedly cover the same aspects while ignoring crucial aspects, resulting in excessively low accuracy of the query results. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and medium that can improve the quality of responses to query information.

[0004] One embodiment of this application provides a data processing method, including: Obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths; there is no data interaction between at least two data processing paths. Candidate entries are obtained from at least two first response texts, and a candidate set of recall results for the first query information is obtained. Based on the candidate entries and the candidate set of recall results, a first evaluation criterion for the first query information is generated. The first evaluation criterion is used to generate a set-level reward for the first query information to select the training phase. Obtain the first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criteria and the reference utility label to obtain the candidate annotation model; The candidate annotation model is trained using reinforcement learning based on the first evaluation criterion to obtain the label annotation model; the label annotation model is used to generate document utility labels for the recall training phase and the ranking training phase for the first query information.

[0005] One embodiment of this application provides a data processing apparatus, including: The first response generation module is used to obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths; there is no data interaction between the at least two data processing paths. The evaluation criteria generation module is used to obtain candidate entries from at least two first response texts, obtain a candidate set of recall results for the first query information, and generate a first evaluation criterion for the first query information based on the candidate entries and the candidate set of recall results; the first evaluation criterion is used to generate a set-level reward for the first query information to select the training stage; The imitation learning module is used to obtain the first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criteria and the reference utility label to obtain the candidate annotation model. The reinforcement learning module is used to train the candidate annotation model using reinforcement learning based on the first evaluation criterion to obtain the label annotation model; the label annotation model is used to generate document utility labels for the first query information during the recall training phase and the ranking training phase.

[0006] Among them, at least two data processing paths correspond one-to-one with at least two pre-trained language models, and each data processing path uses an independent pre-trained language model; the at least two pre-trained language models include the i-th pre-trained language model, where i is a positive integer; The first response generation module generates at least two first response texts for the first query information according to at least two data processing paths, which are used to perform the following steps: The first query information is input into the i-th pre-trained language model, and the first query information is converted into a query embedding vector through the embedding layer of the i-th pre-trained language model. The semantic encoding layer of the i-th pre-trained language model is used to semantically encode the query embedding vector to obtain the semantic encoding vector of the first query information. Perform text prediction processing on the semantic encoding vector to generate the first response text of the i-th pre-trained language model for the first query information.

[0007] The evaluation criteria generation module obtains the candidate set of recall results for the first query information, and generates the first evaluation criteria for the first query information based on the candidate entries and the candidate set of recall results, which is used to perform the following steps: For each candidate item in the first response text, deduplication is performed to obtain the evaluation items of the first query information, and the evaluation items are constructed into an evaluation dimension architecture; Obtain the recall result candidate set of the first query information from the production log, perform semantic matching between the evaluation dimension architecture and the recall result candidate set, determine the matching content of the evaluation item in the recall result candidate set, and convert the matching content into verifiable sub-items of the evaluation item. If the verifiable sub-entry passes the fact check, the evaluation entry and the verifiable sub-entry will be combined into the first evaluation criterion of the first query information.

[0008] The imitation learning module acquires the first training corpus set and outputs the reference utility label of the first sample document in the first training corpus set through the reference model, which is used to perform the following steps: Obtain a second training corpus set, and determine the first training corpus set from the second training corpus set according to a preset ratio parameter; Obtain document annotation rules, and through the reference model, annotate the first sample document in the first training corpus according to the document annotation rules to obtain the reference utility label of the first sample document.

[0009] The imitation learning module trains the initial annotation model using the first training corpus, the first query information, the first evaluation criteria, and the reference utility label to obtain a candidate annotation model, which is then used to perform the following steps: Using the initial annotation model, based on the first query information and the first evaluation criteria, the utility of the first sample document in the first training corpus is evaluated to obtain the first predicted utility category of the first sample document; Based on the first predicted utility category and the reference utility label, the imitation learning loss of the initial annotation model is determined. The network parameters of the initial annotation model are then corrected based on the imitation learning loss to obtain the candidate annotation model.

[0010] The reinforcement learning module trains the candidate labeling model using reinforcement learning based on the first evaluation criterion to obtain a labeling model, which is then used to perform the following steps: Obtain the second sample document from the second training corpus set, evaluate the utility of the second sample document using the candidate annotation model based on the first query information and the first evaluation criteria, and output the second predicted utility category of the second sample document; Based on the second predicted utility category and the first evaluation criterion, the policy reward for the second predicted utility category is determined. The network parameters of the candidate labeling model are then modified according to the policy reward to obtain the labeling model.

[0011] The second sample document contains N documents, where N is an integer greater than 1. The reinforcement learning module determines the policy reward for the second predicted utility category based on the second predicted utility category and the first evaluation criterion, which is used to perform the following steps: Based on the first evaluation criterion, determine the utility accuracy corresponding to the second predicted utility category for each second sample document; Based on the document annotation rules, determine the structural matching degree corresponding to the second predicted utility category of each second sample document; The true ranking relationship between every two second sample documents in the N second sample documents is determined according to the first evaluation criterion, and the predicted ranking relationship between every two second sample documents is determined according to the second predicted utility category of every two second sample documents. Based on the actual ranking relationship and the predicted ranking relationship, the pairwise ranking accuracy is determined. The utility accuracy, structure matching accuracy and pairwise ranking accuracy are combined to determine the policy reward for the second predicted utility category of each second sample document.

[0012] The data processing device also includes: The recall training module is used to obtain the recall training set of the first query information from the production log, and divide the recall training documents in the recall training set into the first positive sample and the first negative sample; the first positive sample is any recall training document in the recall training set. The recall training module is also used to compare and train the initial encoder using the first positive sample and the first negative sample to obtain candidate encoders; The recall training module is also used to output the document utility tags corresponding to the recalled training documents in the recall training set through the label annotation model, and to determine the second positive sample and the second negative sample in the recall training set based on the document utility tags; the priority of the document utility tags corresponding to the second positive sample is higher than the priority of the document utility tags corresponding to the second negative sample. The recall training module is also used to perform utility alignment training on the candidate encoder using the second positive sample and the second negative sample to obtain the recall encoder.

[0013] The recall training module uses the first positive sample and the first negative sample to perform comparative training on the initial encoder to obtain candidate encoders, which are then used to perform the following steps: The first positive sample is encoded by the initial encoder to obtain the first sample encoding vector of the first positive sample; The first negative sample is encoded by the initial encoder to obtain the second sample encoding vector of the first negative sample; The first query encoding vector of the first query information is output by the initial encoder, the first similarity between the first query encoding vector and the first sample encoding vector is obtained, and the second similarity between the first query encoding vector and the second sample encoding vector is obtained. Based on the first and second similarities, the contrast loss of the initial encoder is determined, and the network parameters of the initial encoder are corrected according to the contrast loss to obtain the candidate encoder.

[0014] The data processing device also includes: The sorting training module is used to obtain the sorting training set of the first query information from the production logs. Through the shared encoder in the initial sorting model, the sorting training documents in the sorting training set are encoded to obtain the third sample encoding vector of the sorting training documents. The ranking training module is also used to map the third sample encoding vector to at least two prediction evaluation values ​​through at least two prediction components in the initial ranking model; the at least two prediction components include a main prediction component and at least one auxiliary prediction component, and each prediction component corresponds to a prediction evaluation value; The ranking training module is also used to determine the multi-task loss of the initial ranking model based on the prediction evaluation value output by the main prediction component and the document utility label of the ranking training document, as well as the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component. The ranking training module is also used to correct the network parameters of the initial ranking model based on the multi-task loss to obtain the ranking stage model; the ranking stage model includes the trained shared encoder and the master prediction component.

[0015] The ranking training module determines the multi-task loss of the initial ranking model based on the prediction evaluation value output by the main prediction component and the document utility labels of the ranking training documents, as well as the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component. This loss is used to perform the following steps: The utility loss of the main prediction component is determined based on the prediction evaluation value output by the main prediction component and the document utility labels of the ranked training documents. Based on the predicted evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component, determine the auxiliary loss of each auxiliary prediction component; Obtain the control parameters corresponding to the auxiliary loss of each auxiliary prediction component. Based on the control parameters corresponding to the auxiliary loss of each auxiliary prediction component, perform a weighted summation of the auxiliary losses of each auxiliary prediction component to obtain the regularization term. The utility loss and regularization term are combined into the multi-task loss of the initial ranking model.

[0016] Among them, the evaluation items in the first evaluation criterion carry item weights; The data processing device also includes: The selection training module is used to obtain the selection training set of the first query information. Through the initial policy model, M candidate subsets are sampled from the selection training set. The hit items of each candidate subset are determined in the first evaluation criterion. The set-level reward of each candidate subset is determined according to the item weight of the hit items; M is a positive integer. The selection training module is also used to determine the mean reward of M candidate subsets based on the ensemble-level reward of each candidate subset, and to determine the standard deviation of the reward based on the mean reward and the ensemble-level reward of each candidate subset; The selection training module is also used to determine the candidate subset with ensemble-level reward greater than the average reward as the first branch subset, and the candidate subset with ensemble-level reward less than or equal to the average reward as the second branch subset; The selection training module is also used to determine the policy gain of the first branch subset based on the set-level reward, the mean reward, and the standard deviation of the reward, and to determine the policy optimization loss based on the policy gain and the predicted conditional probability corresponding to the first branch subset. The selection training module is also used to determine the candidate subset with the largest set-level reward as the reference policy from the M candidate subsets, and to perform self-distillation training on the initial policy model based on the reference policy and the second branch subset to determine the distillation loss. The selection training module is also used to merge the policy optimization loss and the distillation loss to obtain the joint loss of the initial policy model, and to update the policy of the initial policy model based on the joint loss to obtain the selected policy model.

[0017] The data processing device also includes: The query acquisition module is used to acquire second query information, determine the initial document set corresponding to the second query information based on the task type of the second query information, and acquire the second evaluation criteria of the second query information. The document retrieval module is used to encode the second query information through the retrieval encoder to obtain the second query encoding vector, and to encode the initial documents in the initial document set through the retrieval encoder to obtain the document encoding vector of the initial documents. The document recall module is also used to recall a set of candidate documents from the initial document set based on the vector similarity between the second query encoding vector and the document encoding vector. The document ranking module is used to evaluate the utility of candidate documents in the candidate document set through the ranking stage model, obtain the ranking evaluation value of the candidate documents, and determine K recommended documents in the candidate document set based on the ranking evaluation value, where K is a positive integer; The document selection module is used to filter a subset of documents from K recommended documents using a selection strategy model, evaluate the utility of the document subset according to the second evaluation criterion, and obtain the evaluation result of the document subset. The second response generation module is used to integrate information from a subset of documents through a target language model and output the second response text of the second query information if the evaluation result indicates that the preset conditions are met.

[0018] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method provided in one aspect of this application.

[0019] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0020] According to one aspect of this application, a computer program product is provided, which may include a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and executes the computer program, causing the computer device to perform the method provided in the above aspect.

[0021] In this embodiment, for the first query information, at least two first response texts can be generated according to at least two independent data processing paths. Candidate entries are extracted from the at least two first response texts, and these candidate entries are anchored to the actual recall result candidate set to generate a first evaluation criterion for the first query information. This first evaluation criterion can be used to generate a set-level reward for the selection training phase of the first query information. A reference model can be used to annotate the first sample documents in the first training corpus, obtaining reference utility labels for the first sample documents. These reference utility labels are used as supervision signals for the initial annotation model during imitation learning training to obtain a candidate annotation model. The candidate annotation model is then trained using reinforcement learning according to the first evaluation criterion to obtain a label annotation model. This label annotation model is used to generate document utility labels for the recall and ranking training phases of the first query information. In other words, a unique evaluation criterion can be generated for each query information. This evaluation criterion allows the three training phases of recall, ranking, and selection to be aligned and optimized towards the same goal of "answer quality," thereby improving the response quality of the query information. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a schematic diagram of a network architecture provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 ; Figure 3 This is a schematic diagram of the generation process of an evaluation criterion provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the structure of an evaluation standard provided in an embodiment of this application; Figure 5 This is a schematic diagram of distillation training of an initial labeled model provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 ; Figure 7 This is a schematic diagram of a dual-branch training method for selecting a training stage, provided in an embodiment of this application. Figure 8 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 3 ; Figure 9 This is a schematic diagram of an online service process provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] The basic concepts involved in the embodiments of this application are described below.

[0026] Large Language Models (LLMs) are large-scale language models that use massive amounts of unlabeled natural language text as training data and are pre-trained through a self-supervised learning paradigm. LLMs possess zero-shot or few-shot learning capabilities and can efficiently perform multi-dimensional downstream tasks such as natural language understanding (e.g., text classification, information extraction), natural language generation (e.g., text summarization, machine translation), and logical reasoning (e.g., mathematical calculation, code generation).

[0027] Retrieval-augmented generation (RAG) is a technique that endows generative artificial intelligence models (such as large language models) with information retrieval capabilities. It allows the model to respond to the query information input by the object based on a set of retrieved documents, thereby enhancing the information in the model's static training data.

[0028] Intelligent body-based search: Instead of browsing a ranked list of results, a large language model synthesizes a single answer based on reasoning across a bounded set of documents. The quality of the answer depends on the bounded set of documents delivered to the large language model, rather than on the correctness of the document relevance judgments in the ranked list.

[0029] The first evaluation criterion (Rubric) refers to an evaluation method that breaks down an overall quality judgment into several discrete, verifiable items. This first evaluation criterion can be applied to the training and supervision signals throughout the entire search process.

[0030] Reinforcement learning (RL) is a machine learning method in which an agent adjusts its policy based on reward signals by interacting with the environment in order to maximize long-term cumulative rewards.

[0031] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. The network architecture may include a terminal cluster and a server 10d. The terminal cluster may include one or more terminal devices. No limitations are placed on the number of terminal devices included in the terminal cluster or the number of servers included in the network architecture. Figure 1 As shown, the terminal cluster may specifically include terminal device 10a, terminal device 10b, and terminal device 10c, etc. Each terminal device in the terminal cluster can connect to server 10d via the network so that each terminal device can interact with server 10d through the network connection.

[0032] in, Figure 1 The terminal devices in the terminal cluster shown may include: smartphones, tablets, laptops, PDAs, desktop computers, wearable devices (such as smartwatches, smart bracelets, etc.), smart voice interaction devices, smart home appliances (such as smart TVs, etc.), in-vehicle devices, aircraft, and other electronic devices. This application does not limit the type of terminal device.

[0033] Figure 1The server 10d shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not limit the type of server 10d.

[0034] In a search scenario, after an object inputs query information, the search system can recall candidate documents from a large-scale corpus. During the ranking phase, the recalled candidate documents are rearranged, and during the selection phase, a final subset of documents (boundary document set) is selected. Then, a large language model generates a cited answer based on this final document subset. Here, the search system refers to a system that helps an object find the content it needs from a large amount of information. This application's embodiment does not change the product's interaction form, but rather improves the quality of the query answer by changing the selection strategy of the final document subset.

[0035] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 It is understood that this data processing method can be executed by a computer device, which can be a terminal device (such as...). Figure 1 The terminal device 10a shown can be a server (such as...). Figure 1 The server shown is 10d, and this application does not limit this. Figure 2 As shown, the data processing method may include the following steps S101 to S104: Step S101: Obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths; there is no data interaction between the at least two data processing paths.

[0036] Here, the first query information refers to the historical keywords or expressions submitted by the object to the search system. The data processing path refers to the entire set of internal calculations and information processing processes that the large language model used by the search system undergoes after receiving the first query information, without relying on external search results or pre-fixed answers, but solely on its own parameterized knowledge to understand, reason, and generate the first query information, and finally independently output the first response text.

[0037] The first response text refers to a single answer generated for the first query information according to a single data processing path. Each data processing path can output a single answer. There is no data interaction between different data processing paths; that is, each first response text for the first query information is an independently generated, free-form answer.

[0038] In one or more embodiments, at least two data processing paths correspond one-to-one with at least two pre-trained language models, and each data processing path uses an independent pre-trained language model. The at least two pre-trained language models refer to large language models that differ in architecture, training data, task specialization, parameter size, data modality, etc.

[0039] The following description uses the generation process of a first response text as an example. For the i-th pre-trained language model among at least two pre-trained language models, the computer device can input the first query information into the i-th pre-trained language model, where i is a positive integer and the value of i is less than or equal to the number of pre-trained language models.

[0040] The first query information is converted into a query embedding vector through the embedding layer of the i-th pre-trained language model; the query embedding vector is semantically encoded through the semantic encoding layer of the i-th pre-trained language model to obtain the semantic encoding vector of the first query information; the semantic encoding vector is processed by text prediction to generate the first response text of the i-th pre-trained language model for the first query information.

[0041] The embedding layer is a network layer that transforms the first query information, described in natural language, into a continuous dense vector. The query embedding vector is the digital representation of the first query information in the vector space. The semantic encoding layer is the basic stacked unit of the i-th pre-trained language model, used to gradually abstract the representation of each token in the first query information from local syntactic information to global semantic information. Semantic encoding refers to the process of gradually abstracting the representation of each token from local syntactic information to global semantic information.

[0042] Semantic encoding vectors refer to the comprehensive representation of each token in the first query information at its current position and in the current context. They can integrate the semantics of each token itself, the relationship between the current token and other tokens, the contextual information of the first query information, and the language knowledge accumulated by the i-th pre-trained language model during pre-training.

[0043] For example, the semantic encoding layer of the i-th pre-trained language model can be a Transformer (a deep learning model based on an attention mechanism) layer or a DiT (Diffusion Transformer) layer; this application does not limit this. Taking the semantic encoding layer as a Transformer layer as an example, this Transformer layer can include multi-head self-attention, feed-forward network (FFN), residual connection, layer normalization, etc.

[0044] Multi-head self-attention is used to aggregate information from other tokens for each token in the first query. A feedforward neural network is used to perform a non-linear transformation on the output of multi-head self-attention. Residual connections are an architecture that allows network layer inputs to bypass one or more transformation layers and be added to the output of those layers, ensuring that low-level features are not completely lost during multiple non-linear transformations. Layer normalization is a technique that normalizes the mean and variance of the first query information across its feature dimensions.

[0045] Text prediction processing refers to mapping semantic encoding vectors to probability distributions, selecting the next token based on these distributions, and ultimately generating the first response text. This text prediction process includes: mapping the semantic encoding vectors to a vocabulary dimension using a linear layer to obtain the logits (unnormalized scores) of each vocabulary token; then passing this logits through a softmax layer (normalization layer) to obtain the probability distribution of the next token. The vocabulary dimension represents the size of the vocabulary, such as the total number of tokens contained in the vocabulary. The next token is selected based on the probability distribution, and the newly generated token is added to the text generation sequence (a sequence composed of tokens predicted by the i-th pre-trained language model). These steps are repeated to finally output the first response text for the first query.

[0046] Step S102: Obtain candidate entries from at least two first response texts, obtain a candidate set of recall results for the first query information, and generate a first evaluation criterion for the first query information based on the candidate entries and the candidate set of recall results; the first evaluation criterion is used to generate a set-level reward for the first query information to select the training stage.

[0047] Here, candidate entries refer to the scoring dimensions or scoring points generated for the first query information. The recall result candidate set refers to the header candidate documents actually recalled by the search system for the first query information. The first evaluation criterion, also known as the scoring criterion, can serve as a dedicated "utility contract" for the first query information.

[0048] Specifically, the computer device can extract candidate items from each first response text that can serve as scoring dimensions or scoring points for the first query information. The candidate items for each first response text are deduplicated to obtain evaluation items for the first query information, which are then used to construct an evaluation dimension architecture. Here, an evaluation item refers to the result of semantic alignment and deduplication of the candidate items for each first response text. The evaluation dimension architecture refers to a high-level dimensional skeleton organized from discrete evaluation items, such as "performance," "quality," and "service."

[0049] In one or more embodiments, a prompt word can drive a large language model (such as any one of the at least two pre-trained language models described above) to extract candidate items from each first response text. The prompt word can declare the role played by the large language model (e.g., an "information demand extraction assistant"), and list information such as task requirements and output format. The task requirements inform the large language model of the true objective of the current task and constrain the large language model to output results that meet the task requirements. The output format constrains the form of the large language model's output, such as JSON (JavaScript Object Notation; JavaScript is a web page programming language).

[0050] Optionally, each first response text can be semantically segmented to obtain a set of fragments for each first response text; the semantic similarity between the first query information and each fragment in the fragment set can be calculated, and fragments with a semantic similarity greater than a first preset threshold (which can be flexibly set according to specific needs, such as 0.85) can be used as candidate entries for the first query information. The semantic similarity can be used to represent the degree of similarity between the first query information and a single fragment. This semantic similarity can be calculated using methods such as Cosine Similarity, Euclidean Distance, Manhattan Distance / L1, and Jaccard Similarity; this application does not limit the specific methods used.

[0051] In one or more embodiments, the semantic deduplication process may include: summarizing the candidate entries extracted from each first response text to obtain a candidate entry set. The same vector embedding model can be used to map each candidate entry in the candidate entry set to a vector representation; based on the vector representation of each candidate entry, the vector similarity between any two candidate entries is calculated. When the vector similarity between two candidate entries is greater than a second preset threshold (which can be flexibly set according to specific needs, such as 0.9), it can be determined that these two candidate entries are redundant, and one of the candidate entries can be deleted.

[0052] Obtain the recall result candidate set of the first query information from the production log, perform semantic matching between the evaluation dimension architecture and the recall result candidate set, determine the matching content of the evaluation item in the recall result candidate set, and convert the matching content into verifiable sub-items of the evaluation item; if the verifiable sub-item passes the fact verification, then merge the evaluation item and the verifiable sub-item into the first evaluation criterion of the first query information.

[0053] Production logs refer to the records automatically generated by the search system when it officially provides services to the public, including every query, system response, internal processing step, and exception. Production logs can include object-side operations and system-side processing, and are a true "snapshot" of the search system's operating status.

[0054] The candidate set of recall results refers to the few documents that are ranked first, scored the highest, and are most likely to be relevant from the initial set of documents selected from a large number of documents during the recall phase of the first query information, such as the top 5 documents or the top 10 documents.

[0055] The evaluation items in the evaluation dimension architecture are semantically matched with each document in the recall result candidate set to identify the text fragments (matching content) in the document that match the evaluation items. The matching content can then be instantiated into specific falsifiable sub-items (verifiable sub-items). For example, the evaluation item "performance" is instantiated as "gives the performance improvement multiple relative to the previous generation", rather than "contains performance information".

[0056] It should be noted that each verifiable sub-item in the first evaluation criterion needs to undergo fact verification to prevent errors in the documents themselves within the candidate recall set from being included in the first evaluation criterion, thereby improving its effectiveness. Fact verification refers to the process of confirming the authenticity, accuracy, or credibility of the factual content in each verifiable sub-item by searching for evidence, comparing sources, and making inferences.

[0057] In one or more embodiments, verifiable sub-entries can be input into one or more large language models (such as at least two pre-trained language models mentioned above), and the large language models are required to determine the verifiability of the verifiable sub-entry. If more than 80% (this is just an example and can be flexibly set according to actual needs) of the large language models output answers such as "true," "accurate," or "credible," then the verifiable sub-entry can be determined to have passed fact verification. If less than 80% are reached, the verifiable sub-entry is determined to have failed fact verification and can be discarded.

[0058] Optionally, the first part of the text of a verifiable sub-entry can be submitted as a question to the large language model. If the answer output by the large language model matches the verifiable sub-entry, the verifiable sub-entry has passed fact verification; if the answer output by the large language model does not match the verifiable sub-entry, the verifiable sub-entry has failed fact verification and can be discarded. For example, if the verifiable sub-entry is "The champion of the 20xx xx competition is Xiao A", then "Who is the champion of the 20xx xx competition?" can be input as a question into the large language model. If the answer output by the large language model is "Xiao A", it means that the fact verification has passed; if the output answer is not "Xiao A", it means that the fact verification has failed.

[0059] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the generation process of an evaluation criterion provided in an embodiment of this application. For example... Figure 3 As shown, the process of generating evaluation criteria can include two stages: a coarse-grained stage and a fine-grained stage. The coarse-grained stage can be used to ensure the comprehensiveness of the evaluation criteria, while the fine-grained stage can be used to ensure the distinctiveness of the evaluation criteria.

[0060] For ease of understanding, the following description uses the generation process of the first evaluation criterion as an example. In the coarse-grained stage, for any query information, such as the first query information, at least two pre-trained language models can be used to independently generate a free-form first response text for each query information. When the at least two pre-trained language models include pre-trained language model 1, pre-trained language model 2, and pre-trained language model 3, the pre-trained language model can generate first response text a for the first query information, pre-trained language model 2 can generate first response text b for the first query information, and pre-trained language model 3 can generate first response text c for the first query information.

[0061] Candidate items are extracted from each first response text. These candidate items are then semantically aligned and deduplicated to obtain evaluation items. The discrete evaluation items are then organized into a high-level dimensional skeleton. It is understandable that since the generation process of each first response text has already unfolded the information aspect, extracting candidate items from the first response text, rather than directly from the first query information, not only enables cross-model aggregation but also covers a range of items that a single model cannot exhaustively encompass, thus improving the comprehensiveness of the first evaluation criteria.

[0062] In the fine-grained stage, the high-level dimensional skeleton is anchored to the header candidate documents (i.e., the candidate set of recall results) actually retrieved by the search system for the first query information. Evaluation items in the high-level dimensional skeleton are instantiated into specific verifiable sub-items. Each verifiable sub-item undergoes fact verification, and those that pass the fact verification are written into the first evaluation criterion. This avoids including document errors in the first evaluation criterion. By instantiating evaluation items into verifiable sub-items within the candidate set of recall results, the discriminative power of the first evaluation criterion is improved, making it easier to distinguish between different documents.

[0063] Understandably, due to the vastly different nature of the evidence required for different queries, a static relevance system cannot cover the unique requirements of different queries for the nature of evidence. A static relevance system refers to a predefined and fixed scoring dimension or relevance framework. This static relevance system may be applicable to some queries but not to others. The proposed solution can generate a unique evaluation standard for each query.

[0064] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the structure of an evaluation standard provided in an embodiment of this application. Figure 3 The coarse-grained and fine-grained stages shown can generate a first evaluation criterion specific to the first query information. This first evaluation criterion can include positive and negative entries. Positive entries refer to the information aspects that the document set needs to provide collectively. Each positive entry carries an importance weight, which is used to represent the proportion of the current positive entry in the total score. Negative entries refer to defects that need to be avoided. Each negative entry carries a deduction margin.

[0065] The importance weight of positive items and the penalty for negative items can both be referred to as the item weights of the evaluation items in the first evaluation criterion. When the documents in a document set collectively provide the information required for a positive item, it means that the document set satisfies that positive item. When any document in a document set violates a negative item, it means that the document set has triggered that negative item. Thus, the evaluation criteria themselves are set-level.

[0066] For example, suppose the first query is "mobile office review and price of the latest version of the xx brand xx series tablet". Then the first evaluation criterion for the first query can be expressed as follows: "1. The document collection mentions the specific model of the latest version of the tablet of brand xx series (importance weight: 6 points)."

[0067] 1.1 The document mentions that the latest version of the xx brand xx series tablet is equipped with the xx5 chip (the fifth generation xx chip).

[0068] 2. The document collection mentions the performance of the latest version of the xx brand xx series tablet in mobile office scenarios, including the xx chip, up to 12GB (gigabytes) of memory, AI (Artificial Intelligence) processing speed and professional computing power improvement, and smooth operation of tasks such as 4K (3840×2160 ultra-high-definition resolution) video editing and 3D rendering (importance weight: 9 points).

[0069] 2.1 The document mentions that it can smoothly run professional tasks such as 4K video editing and 3D (three-dimensional) rendering.

[0070] 2.2 The document mentions that AI performance is improved by up to 3.5 times (compared to the fourth-generation xx chip).

[0071] 2.3 The document mentions that the 3D ray tracing rendering performance is 6.7 times faster than the xx1 chip (the first generation xx chip).

[0072] 2.4 The document mentions that the video transcoding performance is 6 times faster than that of the xx1 chip.

[0073] 3. The document collection mentions the mobility and battery life experience of the latest version of the xx brand xx series tablet, including its thin and light body, support for 60W (watt) fast charging (charges to 50% in 30 minutes), screen enhancement anti-oil and fingerprint capabilities, and N1 wireless network chip (importance weight: 8 points).

[0074] 3.1 The document mentions support for Wi-Fi 7 and 5G cellular networks.

[0075] 3.2 The document mentions a built-in 10300mAh battery, providing 10 hours of local video playback and approximately 7.5 hours of online meeting time.

[0076] 3.3 The document mentions support for 27W wired fast charging, but no reverse fast charging.

[0077] 3.4 The document mentions that it is equipped with an N1 wireless network chip.

[0078] 4. The document collection mentions the official price and market price of the latest version of the xx brand xx series tablet (importance weight: 8 points).

[0079] ..." Evaluation items 1, 2, 3, and 4 in the first evaluation standard mentioned above are all positive items. “1.1” is a verifiable sub-item of evaluation item 1, “2.1, 2.2, 2.3, and 2.4” are verifiable sub-items of evaluation item 2, and “3.1, 3.2, 3.3, and 3.4” are verifiable sub-items of evaluation item 3.

[0080] Step S103: Obtain the first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criterion and the reference utility label to obtain the candidate annotation model.

[0081] Specifically, in the recall and ranking stages of a search scenario, each document needs to be scored independently. However, the evaluation criteria are set-level. To address the granularity mismatch between the evaluation criteria and the recall and ranking stages, an annotation tool can be trained to map "query information, document, and evaluation criteria" into four utility levels. These four utility levels are ordered from highest to lowest: complete evidence, sufficient evidence, supporting context, and useless or harmful. These levels can also be referred to as priorities; higher levels indicate greater priority, and lower levels indicate less priority.

[0082] Complete evidence indicates coverage of all high-weight positive entries without triggering any negative entries. Sufficient evidence indicates coverage of most high-weight positive entries, but omitting some minor positive entries; for example, coverage of 60% or more (excluding 100%) of high-weight positive entries. Auxiliary context indicates coverage of only low-weight positive entries or marginal background positive entries. Useless or harmful indicates triggering any negative entry or not covering any positive entries.

[0083] Here, "high-weight item" refers to an item weight greater than the high-weight threshold; "low-weight item" refers to an item weight less than or equal to the high-weight threshold and greater than the marginal threshold. "Secondary positive items" refer to positive items ranking in the bottom 30% (which can be any value less than 50%) among the high-weight positive items; optionally, secondary positive items can also be low-weight positive items. "Marginal background positive items" refer to positive items with an item weight less than or equal to the marginal threshold. The high-weight threshold is greater than the marginal threshold; both thresholds can be preset, and their specific values ​​can be flexibly set according to actual needs.

[0084] Understandably, mapping "query information, documents, and evaluation criteria" to four levels of utility means that for simple queries, a single document can achieve the highest level (complete evidence); for complex queries, even a partial match at the second-highest level (sufficient evidence) can reduce the number of documents required to jointly meet the evaluation criteria.

[0085] The process of determining the above four levels of utility requires checking each evaluation item (including verifiable sub-items) in the document and evaluation criteria, weighing their relative importance, and simultaneously generating a utility level and justification. Lightweight models (also known as small models) lack the above reasoning ability, while calling large models pairwise (documents and evaluation items one by one) on the entire corpus may be computationally and time-consuming.

[0086] In this embodiment of the application, distillation training is used to resolve the above contradictions. The distillation training includes: the strong teacher model annotates a portion of the samples offline according to the pre-declared document annotation rules, and then trains a lightweight student model to approximate the judgment of the strong teacher model. The lightweight student model obtained from the training amplifies the annotation to the entire corpus.

[0087] Lightweight models refer to machine learning models with fewer parameters and lower computational complexity, capable of running efficiently in resource-constrained environments. They are characterized by small parameter size, low computational cost, and low deployment threshold. Large models, on the other hand, refer to deep learning models with a large number of parameters, trained on large-scale data, and exhibiting strong generalization and emergence capabilities. They are characterized by large parameter size, extensive training data, high computational cost, and strong emergence capabilities.

[0088] A strong teacher model is a large model that can provide high-quality outputs, intermediate representations, or supervisory signals to guide the learning of smaller models. A lightweight student model is a lightweight model that can mimic the strong teacher model to achieve judgment capabilities close to those of the strong teacher model, and is used for online practical services. Document annotation rules refer to predefined, uniform annotation rules (such as scoring dimensions, scoring standards, output formats, etc.); optionally, these document annotation rules may include a primary evaluation criterion.

[0089] In one or more embodiments, the computer device may acquire a second training corpus set, and determine a first training corpus set from the second training corpus set according to a preset ratio parameter. Document annotation rules are acquired, and a first sample document in the first training corpus set is annotated according to the document annotation rules using a reference model to obtain a reference utility label for the first sample document.

[0090] The second training corpus set refers to the entire corpus in the search system, or the entire corpus in the search system related to the task type of the first query information. The preset ratio parameter is a pre-set value used to represent the ratio of the number of documents in the first training corpus set to the number of documents in the second training corpus set. This preset ratio parameter can be 5%, 8%, 10%, etc., and this application does not limit it.

[0091] The reference model refers to the strong teacher model in distillation training. The reference utility label is the utility level assigned by the reference model to the first sample document in the first training corpus according to the document annotation rules. The reference utility label for each first sample document can be any of the four utility levels mentioned above. The first sample document refers to any document in the first training corpus.

[0092] The first query information, the first evaluation criterion, and the first sample document from the first training corpus are input into the initial annotation model. This initial annotation model evaluates the utility of the first sample document based on the first query information and the first evaluation criterion, obtaining the first predicted utility category of the first sample document. Based on the first predicted utility category and the reference utility label, the imitation learning loss of the initial annotation model is determined. Based on this imitation learning loss, the network parameters of the initial annotation model are corrected to obtain the candidate annotation model.

[0093] The initial annotation model is a lightweight student model used in distillation training. Utility evaluation refers to the process by which the initial annotation model assesses the utility level of the first sample document based on the first evaluation criterion and the first query information. The first predicted utility category refers to the utility level assigned to the first sample document by the initial annotation model; the first predicted utility category can be one of the four utility levels mentioned above. In other words, the initial annotation model takes the triple "first query information, first sample document, first evaluation criterion" as input data, and maps each triple "first query information, first sample document, first evaluation criterion" to one of the four utility levels mentioned above, i.e., the first predicted utility category.

[0094] Imitation learning loss refers to the loss of the initial labeled model during imitation learning training. Imitation learning training refers to a training method where the initial labeled model imitates the semantic understanding ability or structured output of the reference model, rather than directly learning the real labels. The reference utility labels output by the reference model are used as the supervision signal for the initial labeled model during imitation learning training; this type of imitation learning training can also be called supervised fine-tuning. The candidate labeled model refers to the lightweight student model obtained after the initial labeled model has undergone imitation learning training.

[0095] In one or more embodiments, the imitation learning loss can be calculated using the following formula (1): (1) in, This indicates the learning loss due to imitation; This represents the first training corpus set. This indicates the number of documents in the first training corpus. This indicates the first query information and the first sample document pair; c represents the utility level index, and the four utility level indexes are ranked from high to low as c=1, 2, 3, 4. The one-hot indicator representing the reference utility label; when the reference utility label of the first sample document is grade c. Set the value to 1, and set the other utility levels to 0. This represents the probability that the initial annotation model predicts the first sample document to belong to the c-th utility level.

[0096] Step S104: The candidate annotation model is trained by reinforcement learning according to the first evaluation criterion to obtain the label annotation model; the label annotation model is used to generate document utility labels for the recall training stage and the ranking training stage for the first query information.

[0097] Specifically, the computer device can acquire a second sample document from a second training corpus. Using a candidate labeling model, based on the first query information and a first evaluation criterion, it evaluates the utility of the second sample document and outputs a second predicted utility category. Based on the second predicted utility category and the first evaluation criterion, it determines the policy reward for the second predicted utility category. The network parameters of the candidate labeling model are then adjusted according to the policy reward to obtain the labeling model.

[0098] Here, the second sample document refers to any document in the second training corpus that is not included in the first sample document in the first training corpus. The second predicted utility category refers to the utility level assigned to the second sample document by the candidate annotation model. The second predicted utility category can be one of the four utility levels mentioned above. It is understood that the candidate annotation model also uses the triple "first query information, second sample document, first evaluation criterion" as input data. Through the candidate annotation model, each triple "first query information, second sample document, first evaluation criterion" can be mapped to one of the four utility levels mentioned above, i.e., the second predicted utility category.

[0099] Policy reward refers to the scalar signal returned by the environment to the candidate labeling model after it takes an action at a certain time (utility evaluation), which represents the "goodness" or "badness" of this action at the current position. The labeling model refers to the lightweight student model obtained after the candidate labeling model has been trained through reinforcement learning.

[0100] In one or more embodiments, it is assumed that the number of second sample documents is N, where N is an integer greater than 1. The strategy reward is calculated by: determining the utility accuracy corresponding to the second predicted utility category of each second sample document according to a first evaluation criterion; and determining the structural matching degree corresponding to the second predicted utility category of each second sample document according to document annotation rules.

[0101] The true ranking relationship between every two second sample documents is determined based on the first evaluation criterion. The predicted ranking relationship between every two second sample documents is determined based on the second predicted utility category of every two second sample documents. The pairwise ranking accuracy is determined based on the true ranking relationship and the predicted ranking relationship. The utility accuracy, structure matching accuracy and pairwise ranking accuracy are combined to determine the policy reward for the second predicted utility category of each second sample document.

[0102] The strategy rewards can include utility accuracy, pairwise ranking accuracy, and structural matching. Utility accuracy indicates whether the second predicted utility category output by the candidate annotation model for the second sample document is consistent with the utility level determined by the reference model for the same second sample document according to the first evaluation criterion. For example, a utility accuracy of 1 indicates that the second predicted utility category output by the candidate annotation model is consistent with the utility level determined by the reference model; a utility accuracy of 0 indicates that the second predicted utility category output by the candidate annotation model is inconsistent with the utility level determined by the reference model.

[0103] Pairwise ranking accuracy is used to indicate whether the ranking between two second sample documents is correct. For example, if the candidate annotation model outputs the second predicted utility category as complete evidence for second sample document 1 and the second predicted utility category as auxiliary context for second sample document 2, the correct ranking between second sample document 1 and second sample document 2 can be determined as: second sample document 1, second sample document 2 (descending order), which is consistent with the true ranking relationship determined by the first evaluation criterion. In this case, the pairwise ranking accuracy can be 1.

[0104] The structure matching score indicates whether the output format of the second predicted utility category conforms to the output format in the document annotation rules. For example, a structure matching score of 1 indicates that the second predicted utility category output by the candidate annotation model for the second sample document is consistent with the output format in the document annotation rules; a structure matching score of 0 indicates that the second predicted utility category output by the candidate annotation model for the second sample document is inconsistent with the output format in the document annotation rules.

[0105] In one or more embodiments, the policy reward of the j-th second sample document in the second training corpus can be calculated using the following formula (2): (2) in, Let represent the j-th second sample document out of N second sample documents. The j-th second sample document is any document in the second training corpus set, and j is a positive integer less than or equal to N. Let represent the policy reward for the j-th second sample document. Let represent the utility accuracy of the j-th second sample document, where the second predicted utility category of the j-th second sample document matches the true value of the utility ranking. Take 1 otherwise take 0. The utility level truth value refers to the utility level that the reference model determines for the j-th second sample document according to the first evaluation criterion. It can also be called the reference utility label of the second sample document. This represents the structural matching degree of the j-th second sample document. It is defined as the output format of the second predicted utility category of the j-th second sample document conforming to the output format in the document annotation rules. Take 1 if the value is not specified, otherwise take 0. P represents the pairwise sorting accuracy, which is determined by formula (3); , , The pre-defined merging weights are used to represent the importance of the corresponding reward in the strategy reward. For example, they can be 0.5, 0.2, and 0.3 respectively. This application does not limit this.

[0106] In one or more embodiments, pairwise sorting accuracy can be calculated using the following formula (3): (3) Where N represents the number of second sample documents. This represents the truth value of the utility ranking of the a-th second sample document. This indicates that the candidate annotation model is the second predicted utility category output by the a-th second sample document; similarly, This represents the truth value of the utility ranking of the b-th second sample document. This indicates that the candidate annotation model outputs the second predicted utility category for the b-th second sample document. The a-th and b-th second sample documents are any two distinct documents in the second training corpus (including N second sample documents).

[0107] 1. [·] indicates an indicator function; the expression within the square brackets... The summation is set to 1 if true, and 0 otherwise. The summation iterates through all document pairs in the N second sample documents, with P ranging from [0,1]. Optionally, the calculation can be performed only on document pairs containing the j-th second sample document, and the result is denoted as... Used to replace P in the above formula (2).

[0108] Please see Figure 5 , Figure 5 This is a schematic diagram of distillation training of an initial labeled model provided in an embodiment of this application. For example... Figure 5 As shown, after generating the first evaluation criteria for the first query information, the strong teacher model (reference model) can be used to annotate the first sample document in the first training corpus offline according to the document annotation rules to obtain the reference utility label of the first sample document. Then, the lightweight student model (initial annotation model) can be approximated to the judgment of the strong teacher model through distillation training.

[0109] The training process of the lightweight student model can include two steps. The first step is supervised fine-tuning (also known as imitation learning training), in which the reference utility label of the first sample document is used as a supervision signal to fine-tune the lightweight student model. The second step is reinforcement learning training, which directly optimizes the alignment between the lightweight student model and the first evaluation criterion. This alignment can be reflected by utility accuracy, pairwise ranking accuracy, and structural matching.

[0110] Supervised fine-tuning can provide a necessary cold start for lightweight student models, preventing subsequent reinforcement learning training from blindly exploring the unstructured output space. Supervised fine-tuning only optimizes surface imitation, and its upper limit of accuracy is the same as that of a strong teacher model. Through reinforcement learning training, it can directly optimize the alignment with the first evaluation criterion, and its policy rewards cover three failure modes: utility accuracy, pairwise ranking accuracy, and structure matching.

[0111] The lightweight student model trained through the above two steps can be called a labeling model. Expanding the labeling model to the entire corpus, document utility labels are generated for each document. These labels include complete evidence, sufficient evidence, supporting context, useless, or harmful. Each document corresponds to one document utility label, i.e., one document corresponds to one utility level. The document utility labels from the entire corpus can serve as supervisory signals for the first query information during the recall and ranking training phases.

[0112] In this embodiment, multiple pre-trained language models are used to independently generate first response texts for the same query information (first query information). Candidate entries are extracted from each first response text, and these candidate entries are anchored to the actual recalled result candidate set to generate a first evaluation criterion for the first query information. In other words, a dedicated evaluation criterion can be generated for each query information. This evaluation criterion allows the recall training stage, ranking training stage, and selection training stage to be aligned and optimized towards the same goal of "answer quality," thereby improving the response quality of the query information.

[0113] Understandably, after training the label annotation model, the following will combine... Figure 6 and Figure 7The recall training phase, ranking training phase, and selection training phase involved in the search system are described.

[0114] Please see Figure 6 , Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 It is understood that this data processing method can be executed by a computer device, which can be a terminal device (such as...). Figure 1 The terminal device 10a shown can be a server (such as...). Figure 1 The server shown is 10d, and this application does not limit this. Figure 6 As shown, the data processing method may include the following steps S201 to S214: Step S201: Obtain the recall training set of the first query information from the production log, and divide the recall training documents in the recall training set into the first positive sample and the first negative sample.

[0115] The recall training set refers to the co-occurring documents obtained from the production logs for the first query. These co-occurring documents are those that are highly semantically or content-related to the core keywords or entities of the first query and are retrieved or mentioned together. The recall training document is any document in the recall training set, which can be used as training data in the recall training phase to train the encoder. The encoder in the recall phase refers to a machine learning model that maps documents to representations in a semantic vector space.

[0116] Specifically, the recall training phase can be divided into two stages: the first stage is intra-domain contrastive pre-training, and the second stage is utility alignment fine-tuning. Intra-domain contrastive pre-training is a pre-training method that applies contrastive learning to a specific data range to learn relevant representations of the data. Utility alignment fine-tuning is a training method that aligns the representation space of the encoder in the recall phase to utility based on a first evaluation criterion.

[0117] The intra-domain contrastive pre-training includes: the computer device can obtain co-occurring documents of the first query information from the production log, and use these co-occurring documents as the recall training set for the first query information during the recall training phase. One recall training document is randomly selected from the recall training set as the first positive sample, and the remaining documents are used as the first negative samples. The first negative sample is a group-level difficult negative example and can include at least one recall training document. In other words, the first negative sample can be at least one recall training document in the recall training set other than the first positive sample.

[0118] The first positive sample is a pseudo-positive example, which can be any of the recalled training documents in the recall training set. A group-level difficult negative example refers to a recalled training document that constitutes the first negative sample and has topical overlap with the first query information; it is more difficult and has greater training value than negative documents from other query information. The first negative sample can be any recalled training document in the recall training set other than the first positive sample.

[0119] Step S202: The initial encoder is trained by comparison using the first positive sample and the first negative sample to obtain the candidate encoder.

[0120] Specifically, in the intra-domain contrastive pre-training of the recall training phase, the computer device can obtain the initial encoder of the recall training phase. The initial encoder can refer to the encoder that needs to be trained or is in the process of training. The initial encoder can be BERT (Bidirectional Encoder Representation from Transformers) and its variants, or sparse autoencoders, etc. This application does not limit the network structure of the initial encoder.

[0121] The first positive sample is encoded using an initial encoder to obtain its first sample encoding vector. The first negative sample is then encoded using the same initial encoder to obtain its second sample encoding vector. Finally, the first query information is encoded using the same initial encoder to obtain its first query encoding vector. In other words, the same encoder model (the initial encoder) can be used to obtain vector representations of the first positive sample, the first negative sample, and the first query information.

[0122] Encoding refers to the process of converting the original input (such as the first positive sample, the first query information, the first negative sample, etc.) into a dense vector of fixed dimensions through an encoder model (such as the initial encoder). The first sample encoded vector is the dense vector obtained by the initial encoder after encoding the first positive sample; the second sample encoded vector is the dense vector obtained by the initial encoder after encoding the first negative sample; and the first query encoded vector is the dense vector obtained by the initial encoder after encoding the first query information.

[0123] Obtain the first similarity between the first query encoding vector and the first sample encoding vector, and obtain the second similarity between the first query encoding vector and the second sample encoding vector. Based on the first and second similarities, determine the contrastive loss of the initial encoder, and adjust the network parameters of the initial encoder according to the contrastive loss to obtain the candidate encoder.

[0124] Contrastive loss is a loss function that constructs positive and negative samples to constrain the initial encoder to bring positive samples closer together and push negative samples further away in the representation space. The first similarity is used to characterize the similarity between the first query information and the first positive sample, while the second similarity is used to characterize the similarity between the first query information and each recalled training document that constitutes the first negative sample.

[0125] It is understandable that the same similarity calculation method can be used to calculate the first similarity and the second similarity. The similarity calculation method may include, but is not limited to: cosine similarity, Euclidean distance, Manhattan distance (L1), Jaccard similarity, etc.

[0126] For example, the first similarity can be used as the numerator, and the sum of the values ​​of all second similarities can be used as the denominator to form the contrast loss. Alternatively, the logarithm of the first similarity can be used as the numerator, and the logarithm of the sum of the values ​​of all second similarities can be used as the denominator to form the contrast loss.

[0127] In one or more embodiments, the contrast loss can be calculated using the following formula (4): (4) in, This represents the contrast loss. q represents the first query encoding vector of the first query information; This represents the first sample encoding vector of the first positive sample; This represents the set of recalled training documents included in the first negative sample. This represents the second sample encoding vector of the recall training samples included in the first negative sample. Indicates the first similarity. The second similarity is represented by the cosine similarity (as shown in formula (16)). The temperature parameter is used to adjust the smoothness of the similarity distribution on an exponential scale. exp(·) represents an exponential function with the natural constant as its base. By minimizing the contrastive loss, the initial encoder can bring the first positive sample closer to the first query information and push the first negative sample further away from the first query information within the vector representation space.

[0128] By minimizing the contrastive loss, the network parameters of the initial encoder are iteratively updated. When the intra-domain contrastive pre-training of the initial encoder reaches the training stopping condition, the initial encoder at the point of stopping is determined as a candidate encoder. The training stopping condition can refer to reaching convergence or reaching a pre-set maximum number of training iterations; this application does not limit this. A candidate encoder refers to the encoder model obtained after the initial encoder has undergone intra-domain contrastive pre-training.

[0129] Step S203: Output the document utility labels corresponding to the recalled training documents in the recall training set through the label annotation model, and determine the second positive sample and the second negative sample in the recall training set based on the document utility labels.

[0130] Specifically, co-occurring documents in the production logs only reflect that "the document was retrieved or mentioned together with the first query information," and cannot distinguish between recalled training documents that fully meet the first evaluation criteria and recalled training documents that only cover a minor aspect. Therefore, utility alignment fine-tuning of the candidate encoder can continue.

[0131] The utility alignment fine-tuning includes: using the previously trained labeling model, labeling the recalled training documents in the recall training set, and outputting document utility labels for each recalled training document. Based on the document utility labels of each recalled training document, determining the second positive sample and the second negative sample in the recall training set. The document utility label corresponding to the second positive sample has a higher priority than the document utility label corresponding to the second negative sample.

[0132] The priority of document utility labels can be based on utility level. For example, recalled training documents labeled as complete evidence and sufficient evidence can be used as the second positive samples, while recalled training documents labeled as auxiliary context, useless, or harmful can be used as the second negative samples.

[0133] Step S204: Utility alignment training is performed on the candidate encoder using the second positive sample and the second negative sample to obtain the recall encoder.

[0134] Specifically, in the utility alignment fine-tuning during the recall training phase, after constructing the second positive and second negative samples based on the document utility labels, the first query information, the second positive sample, and the second negative sample are converted into dense vectors of fixed dimensions by the candidate encoder. Based on the dense vectors corresponding to the first query information, the second positive sample, and the second negative sample, the third similarity between the first query information and the second positive sample, and the fourth similarity between the first query information and the second negative sample are calculated.

[0135] Based on the third and fourth similarity scores, the training loss of candidate encoders in utility alignment fine-tuning is calculated. By minimizing the training loss, the network parameters of the candidate encoders are iteratively updated. Candidate encoders that reach the training stopping condition are identified as recall encoders. Utility alignment fine-tuning aligns the representation space of candidate encoders to utility based on the first evaluation criterion.

[0136] It is understandable that the training loss of the candidate encoder in utility alignment fine-tuning can be directly reused in the form of formula (4). For example, the first positive sample in formula (4) can be replaced with the second positive sample, the first negative sample can be replaced with the second negative sample, the first similarity can be replaced with the third similarity, and the second similarity can be replaced with the fourth similarity to obtain the training loss of the candidate encoder in utility alignment fine-tuning. These will not be elaborated here.

[0137] Step S205: Obtain the sorting training set of the first query information from the production log, and encode the sorting training documents in the sorting training set using the shared encoder in the initial sorting model to obtain the third sample encoding vector of the sorting training documents.

[0138] The ranking training set refers to the set of documents selected from the co-occurring documents in the production logs for the first query. These documents, known as ranking training documents, serve as training data for the ranking training phase, used to train the ranking model. The ranking model is a machine learning model that comprehensively scores the documents selected in the recall phase and re-ranks them according to their scores.

[0139] Specifically, the computer device can acquire the initial ranking model during the ranking training phase. This initial ranking model refers to the ranking model that needs to be trained or is in the process of training. This initial ranking model may include a shared encoder and at least two prediction components (also called prediction heads). The shared encoder is used to transform the ranking training documents in the ranking training set into dense vectors in a vector space. Each prediction component is used to score the ranking training documents from different dimensions. At least two prediction components share an encoder model, i.e., the shared encoder mentioned above.

[0140] The shared encoder in the initial ranking model encodes each ranking training document in the ranking training set, resulting in a third-sample encoding vector for each document. The third-sample encoding vector is a dense vector obtained after the shared encoder encodes the ranking training document.

[0141] Step S206: Map the third sample encoding vector to at least two prediction evaluation values ​​using at least two prediction components in the initial ranking model; the at least two prediction components include a main prediction component and at least one auxiliary prediction component, and each prediction component corresponds to a prediction evaluation value.

[0142] Specifically, the initial ranking model comprises at least two prediction components, including a primary prediction component and at least one auxiliary prediction component. Each prediction component maps a third sample encoding vector to a prediction evaluation value. One prediction component corresponds to one prediction evaluation value, which represents the ranking score of the training document in the corresponding dimension.

[0143] At least two prediction components can have the same network structure (different network parameters) or different network structures. A prediction component can consist of one or more linear layers, activation layers, softmax, etc. This application does not limit this.

[0144] In one or more embodiments, assuming at least two prediction components include a utility prediction component, an authority prediction component, a timeliness prediction component, and a quality prediction component, the utility prediction component can be used as the primary prediction component, while the authority, timeliness, and quality prediction components can all be used as auxiliary prediction components. That is, the above four prediction components can predict the ranking of training documents based on the third sample encoding vector from four dimensions: utility, authority, timeliness, and quality, thus obtaining the prediction evaluation value of each prediction component.

[0145] For example, the predicted evaluation value output by the utility prediction component represents the ranking score of the training documents on the utility dimension. The predicted evaluation value output by the authority prediction component represents the ranking score of the training documents on the authority dimension. The predicted evaluation value output by the timeliness prediction component represents the ranking score of the training documents on the timeliness dimension. The predicted evaluation value output by the quality prediction component represents the ranking score of the training documents on the quality dimension.

[0146] Step S207: Determine the multi-task loss of the initial ranking model based on the prediction evaluation value output by the main prediction component and the document utility labels of the ranked training documents, as well as the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component.

[0147] The multi-task loss refers to the comprehensive loss constructed by combining the prediction evaluation values ​​output by at least two prediction components and the supervision signals of at least two prediction components. The supervision signal of the main prediction component is the document utility label of the ranked training documents, and the label information of the auxiliary prediction component is used as the supervision signal, which is the existing production signal. As with the general processing method, it will not be described in detail here.

[0148] Specifically, the computer device can determine the utility loss of the main prediction component based on the prediction evaluation value output by the main prediction component and the document utility labels of the sorted training documents; and determine the auxiliary loss of each auxiliary prediction component based on the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component.

[0149] Obtain the control parameters corresponding to the auxiliary loss of each auxiliary prediction component. Based on the control parameters corresponding to the auxiliary loss of each auxiliary prediction component, perform a weighted summation of the auxiliary loss of each auxiliary prediction component to obtain the regularization term. Combine the utility loss and the regularization term to form the multi-task loss of the initial ranking model.

[0150] The utility loss characterizes the difference between the ranking score (predicted evaluation value) actually predicted by the main prediction component and the score indicated by the document utility label. The auxiliary loss characterizes the difference between the ranking score actually predicted by the auxiliary prediction component and the label information.

[0151] The control parameter is used to control the importance of the auxiliary loss of the auxiliary prediction component in the multi-task loss, and can also be called the weight; the control parameter can be a value less than 1, such as 0.1, 0.2, etc. The auxiliary losses of different auxiliary prediction components can correspond to the same control parameter or different control parameters, and this application does not limit this.

[0152] In one or more embodiments, the multitasking loss can be calculated using the following formulas (5) and (6): (5) (6) in, Indicates multi-tasking loss. This represents the utility loss of the primary prediction component. Q represents the number of auxiliary prediction components. This represents the auxiliary loss of the m-th auxiliary prediction component. This represents the control parameter corresponding to the auxiliary loss of the m-th auxiliary prediction component, where m is a positive integer less than or equal to Q. The one-hot indicator representing the document utility label of the sorted training documents, when the document utility label is at level c in the four-level utility ranking. Set the value to 1, and set the value to 0 for all other utility levels; This represents the probability of the normalized predicted evaluation value output by the main prediction component corresponding to the c-th utility level. Optionally, the utility loss can also take the form of the mean squared error between the predicted evaluation value and the score indicated by the document utility label, etc., and this application does not limit this.

[0153] For example, suppose the primary forecasting component is a utility forecasting component, and the auxiliary forecasting components are an authority forecasting component, a timeliness forecasting component, and a quality forecasting component (in which case Q is 3). The utility loss of the utility forecasting component is denoted as... The auxiliary loss for the authority prediction component is L1, with a corresponding control parameter of a1. The auxiliary loss for the timeliness prediction component is L2, with a corresponding control parameter of a2. The auxiliary loss for the quality prediction component is L3, with a corresponding control parameter of a3. The regularization term, which is the weighted sum of these three auxiliary losses, can be expressed as: Multitasking loss can be expressed as utility loss. With regularization terms The sum is shown in formula (5).

[0154] Step S208: Based on the multi-task loss, the network parameters of the initial ranking model are corrected to obtain the ranking stage model; the ranking stage model includes the trained shared encoder and the master prediction component.

[0155] Specifically, after calculating the multi-task loss, the network parameters of the initial ranking model can be iteratively updated by minimizing this loss until the training stopping condition is met. The shared encoder and master predictor component at the point of training stopping constitute the ranking stage model. The ranking stage model refers to the ranking model after training is complete.

[0156] Understandably, the auxiliary prediction component only acts as a multi-task regularization term during the ranking training phase, prompting the shared encoder to learn a representation that balances authority, timeliness, and quality. The auxiliary loss of the auxiliary prediction component is weighted overall (by setting a control parameter with a value less than 1) to ensure that the prediction evaluation value of the utility prediction component (the main prediction component) is always the main training signal. During inference, only the utility prediction component outputs a single ranking score; the auxiliary prediction component does not participate in inference.

[0157] In the later stages of the ranking training phase, simple sample pairs (easily distinguishable positive and negative samples) dominate the gradient, which may lead to insufficient ability of the initial ranking model to distinguish difficult samples. In this embodiment, the utility loss shown in formula (6) above can be replaced or supplemented with a focused pairwise loss in the later stages of the ranking training phase. The focused pairwise loss means applying focused reweighting to the pairwise loss, increasing the weight of difficult sample pairs (difficult-to-distinguish positive and negative samples), and decreasing the contribution of simple sample pairs. Focused reweighting means that during the ranking training process, the loss weights of the sample pairs can be dynamically adjusted according to their "difficulty," making the initial ranking model more "focused" on difficult sample pairs.

[0158] In one or more embodiments, the focused pairwise loss can be calculated using the following formulas (7) and (8): (7) (8) in, This indicates focused pairwise loss; Indicates the expectation regarding the information queried in the first query; This represents the set of document pairs whose order can be determined by document utility tags under the first query information. This indicates that the document utility label of the sorted training document u is higher than that of the sorted training document v. This represents the predicted evaluation value of the sorted training document u. This represents the predicted evaluation value of the sorted training document v; σ(·) represents the probability that the initial ranking model determines that the training document u is ranked before the training document v; σ(·) represents the logistic function. Indicates the focus parameter, When the value is 0, formula (7) degenerates into ordinary pairwise loss; The larger the value, the stronger the focus on difficult sample pairs (document pairs with small differences in predicted evaluation values ​​or misranked documents). This represents the scaling parameter. To focus on weights, simple sample pairs ( The loss contribution of pairs close to 1) is automatically reduced, thereby increasing the weight of hard sample pairs.

[0159] Step S209: Obtain the selected training set of the first query information. Through the initial policy model, sample M candidate subsets in the selected training set. Determine the hit items of each candidate subset in the first evaluation criterion. Determine the set-level reward of each candidate subset according to the item weight of the hit items.

[0160] It is understandable that the selection training phase is different from the recall training phase and the ranking training phase of document-by-document independent scoring. The selection training phase needs to optimize the standard coverage of the entire subset of documents selected from the ranking results under the constraint of diversity, which cannot be expressed by point-by-point scoring. The embodiments of this application can adopt a reinforcement learning training selection strategy.

[0161] Specifically, the computer device can obtain a selection training set of the first query information. The selection training set can refer to the ranking results based on the first query information, and the documents in this selection training set can be called selection training documents. An initial policy model for the selection training phase is obtained, and M candidate subsets are sampled from the selection training set using the initial policy model, where M is a positive integer.

[0162] Semantically match the documents in each candidate subset with each evaluation item included in the first evaluation criterion, and determine the evaluation items that each candidate subset matches in the first evaluation criterion based on the matching results. The evaluation items that each candidate subset matches can be called hit items. Hit items can refer to positive items satisfied by a single candidate subset, as well as negative items triggered by that candidate subset.

[0163] Each evaluation item in the first evaluation criterion can carry an item weight, such as the importance weight (represented by a positive number) for positive items and the deduction magnitude (represented by a negative number) for negative items. The item weights carried by the hit items in each candidate subset are summed to obtain the total score for each candidate subset. The total score is then normalized to obtain the set-level reward for each candidate subset. This set-level reward is a scalar reward and does not require any word-level signals. For example, the set-level reward can be a value between 0 and 1, or a value within another uniform range; this application does not limit this.

[0164] In one or more embodiments, the set-level reward of the t-th candidate subset can be calculated using the following formula (9): (9) in, Represents the t-th candidate subset The set-level reward, where t is a positive integer less than or equal to M; Represents candidate subsets The set of positive entries that are hit, in the candidate subset A hit is considered to occur when at least one document satisfies the positive entry (or a verifiable sub-entry of the positive entry). Represents candidate subsets The set of negative entries that are triggered. A violation of this negative entry in any document is considered a trigger. This represents the importance weight of the x-th positive entry; This indicates the penalty for the y-th negative item; represents the normalization constant, such as the sum of the importance weights of all positive items in the first assessment criterion; clip(·) represents the interval cutoff function, used to normalize the total score to the interval [0,1].

[0165] Understandably, the set-level reward can be output by the evaluator, which checks each candidate subset item by item based on the first evaluation criterion and outputs the set-level reward (scalar) and item-by-item hit details for the candidate subset; wherein, a hit is considered when at least one document in the candidate subset satisfies a positive item, and a trigger is considered when any document in the candidate subset violates a negative item.

[0166] In this embodiment, when the same item is repeatedly satisfied by multiple documents in the candidate subset, it is not scored repeatedly (saturation coverage). This is the essential feature that distinguishes set-level utility from "sum of scores per document". It can solve the granularity mismatch between set-level evaluation criteria and score per document. Through formula (9), the scoring logic of the evaluator in the training stage and the online stage can be unified, realizing the characteristic of "the same evaluation criteria runs through the entire link of recall, sorting and selection".

[0167] Step S210: Determine the mean reward of M candidate subsets based on the set-level reward of each candidate subset, and determine the standard deviation of the reward based on the mean reward and the set-level reward of each candidate subset.

[0168] Specifically, the computer equipment can accumulate the ensemble-level rewards for each candidate subset to obtain the cumulative reward value for M candidate subsets. Dividing the cumulative reward value by M yields the mean reward for the M candidate subsets. The deviation between the ensemble-level reward and the mean reward for each candidate subset is calculated (e.g., ensemble-level reward minus the mean reward). The squared deviation for each candidate subset is also calculated (e.g., the squared deviation between the ensemble-level reward and the mean reward). The sum of the squared deviations for each candidate subset is then divided by M, and the square root is taken to obtain the standard deviation of the rewards.

[0169] Step S211: The candidate subset with set-level reward greater than the average reward is determined as the first branch subset, and the candidate subset with set-level reward less than or equal to the average reward is determined as the second branch subset.

[0170] Specifically, the computer equipment can group the M candidate subsets according to the average reward, resulting in a first branch subset and a second branch subset. The first branch subset refers to the candidate subsets within the M candidate subsets whose set-level reward is greater than the average reward. The second branch subset refers to the candidate subsets within the M candidate subsets whose set-level reward is less than or equal to the average reward.

[0171] Step S212: Determine the policy gain of the first branch subset based on the set-level reward, the mean reward, and the standard deviation of the reward. Determine the policy optimization loss based on the policy gain and the predicted conditional probability corresponding to the first branch subset.

[0172] Specifically, the first subset can enter the policy optimization branch, reinforcing the success patterns of the initial policy model at the lexical level. Policy gain represents the policy reward of the initial policy model's current policy at the current position within a single candidate subset; this policy gain can be called the advantage function of reinforcement learning. Alternatively, policy gain represents the degree of "advantage" of the initial policy model's current policy at the current position within a single candidate subset relative to the average performance of M candidate subsets.

[0173] In one or more embodiments, the policy gain can be as shown in formula (10): (10) in, Let represent the policy gain (advantage function) of the t-th candidate subset, where t is a positive integer less than or equal to M, and the t-th candidate subset belongs to the first branch subset. This represents the set-level reward of the t-th candidate subset (calculated by formula (9)). ), This represents the average reward. Indicates the standard deviation of the reward. This is a preset, extremely small positive number used to avoid the denominator being zero.

[0174] After calculating the policy gain for each candidate subset, the policy optimization loss of the initial model policy in the policy optimization branch can be calculated based on the policy gain and the predicted conditional probability corresponding to the first branch subset. Here, the predicted conditional probability refers to the conditional probability value predicted by the initial policy model for the candidate subset based on the first query information and the selected training set.

[0175] Policy optimization loss refers to the loss function used in reinforcement learning to update the network parameters of the initial policy network so that the policy is optimized in the direction of "obtaining higher cumulative rewards". This application does not limit the form of policy optimization loss.

[0176] In one or more embodiments, the policy optimization loss can be calculated using the following formulas (11) and (12): (11) (12) in, Indicates the probability ratio; (·) represents the predicted conditional probability output by the initial policy model (current policy model) with network parameters θ; (·) represents the predicted conditional probability output by the historical strategy model in the previous iteration. Represents the network parameters of the historical strategy model; This represents the k-th selection action when generating the t-th candidate subset, where k is a positive integer; Let |G+| represent the set of the first branch subsets, and |G+| represent the number of the first branch subsets. This represents the number of selection actions required to generate the t-th candidate subset; min(·) represents the minimum function; clip(·) represents the interval cutoff function. The preset cutting width can be set, for example, 0.2. Let represent the policy gain of the t-th candidate subset as shown in formula (10); This represents the loss from strategy optimization.

[0177] In one or more embodiments, the initial policy model can be a sequence generative model that outputs selection actions one by one in an autoregressive manner for the first query information and the selection training set. Each selection action is used to indicate whether to select a candidate document in the selection training set into the document subset, or to indicate the next candidate document to be selected into the document subset. M candidate subsets can be obtained by repeatedly sampling from the policy distribution of the initial policy model.

[0178] Step S213: In the M candidate subsets, the candidate subset with the largest set-level reward is determined as the reference policy. Based on the reference policy and the second branch subset, the initial policy model is trained by self-distillation to determine the distillation loss.

[0179] Specifically, the second branch subset can enter the self-distillation branch. The candidate subset with the largest set-level reward among the M candidate subsets is taken as the teacher policy (also known as the reference policy). Based on the reference policy and the second branch subset, the initial policy model is trained by self-distillation, and the distillation loss of the initial policy model is calculated.

[0180] Self-distillation training refers to an initial policy model imitating itself. Performance is improved by having one version of the initial policy model (the subset of candidates with the largest ensemble reward) guide another version (the subset of candidates with ensemble rewards less than the average reward), without relying on an external teacher model. Distillation loss is a loss function that measures the difference in output or representation between different versions of the initial policy model during self-distillation training.

[0181] In one or more embodiments, the distillation loss can be calculated using the following formulas (13) and (14): (13) (14) in, Describes the set consisting of the subsets of the second branch. Indicates the number of subsets in the second branch; The distillation correction intensity corresponding to the t-th candidate subset is determined by formula (14); This represents the k-th choice action of the reference strategy, which is the subset of M candidate subsets that has the maximum set-level reward. This indicates the number of action choices included in the reference strategy. This represents the predicted conditional probability of the k-th choice action of the current policy model outputting the reference policy; This indicates distillation loss.

[0182] This represents the maximum set-level reward among the M candidate subsets. Let represent the minimum set-level reward among the M candidate subsets. Represents the set-level reward for the t-th candidate subset; is the smallest positive constant in formula (10). It is proportional to the difference between the set-level reward of the t-th candidate subset and the maximum set-level reward: the worse the quality of the t-th candidate subset, the stronger the distillation correction; when the t-th candidate subset is the reference strategy... Take 0.

[0183] Step S214: Combine the policy optimization loss and distillation loss to obtain the joint loss of the initial policy model. Update the policy of the initial policy model based on the joint loss to obtain the selected policy model.

[0184] Specifically, the computer device can add the policy optimization loss of the policy optimization branch to the distillation loss of the self-distillation branch to obtain the joint loss of the initial policy model.

[0185] In one or more embodiments, the joint loss can be calculated using the following formula (15): (15) in, Indicates joint loss; This represents the strategy optimization loss shown in formula (12); This represents the distillation loss shown in formula (13); The balance factor between policy optimization loss and distillation loss can be a fixed value or it can decay with training epochs (e.g., linearly decaying from 1 to 0.2) so that the policy optimization branch becomes dominant after the advantage estimation becomes more reliable.

[0186] By minimizing the joint loss, the initial policy model is updated to obtain the selected policy model. Policy update can essentially be understood as iteratively updating the network parameters of the initial policy model.

[0187] Please see Figure 7 , Figure 7 This is a schematic diagram of a dual-branch training method for selecting a training stage, provided in an embodiment of this application. For example... Figure 7 As shown, during the selection training phase, M candidate subsets can be sampled from the ranking results of the first query information (selection training set) through the initial strategy model, such as candidate subset 1, candidate subset 2, candidate subset 3, ..., candidate subset M.

[0188] A judge, based on evaluation criteria (such as the first evaluation criterion of the first query information), can calculate the set-level reward for each of the M candidate subsets. The calculation of the set-level reward includes: accumulating the item weights of the positive items commonly satisfied by each candidate subset according to the item weights of each evaluation item in the first evaluation criterion, subtracting the deduction magnitude of the negative items triggered by each candidate subset, to obtain the total score for each candidate subset. The total score is then normalized to a unified interval, such as [0,1], to obtain the set-level reward for each candidate subset. While outputting the set-level reward for each candidate subset, the judge can also output item-by-item details for subsequent auditing.

[0189] It is understandable that each candidate subset corresponds to a set-level reward, without any word-level signals. This application's embodiment employs a two-branch training approach, combined with group relative advantage. For example, a judge scores each of the M candidate subsets of the first query information, outputting the set-level reward for each subset. The mean and standard deviation of the rewards for the M candidate subsets are calculated. Candidate subsets whose set-level rewards are greater than the mean reward (such as the subset in the first branch) enter the policy optimization branch, reinforcing the success mode of the initial policy model at the word-level.

[0190] For candidate subsets whose set-level rewards are less than or equal to the mean reward (second branch subsets), directly applying a negative advantage (negative policy gain) would indiscriminately suppress all selection actions in the second branch subset, even though it may contain correct actions. Therefore, candidate subsets whose set-level rewards are less than or equal to the mean reward can enter the self-distillation branch. For example, taking the candidate subset with the largest set-level reward from among M candidate subsets as the lexical-level teacher policy and distilling it into the initial policy model, its correction strength is proportional to the reward difference.

[0191] In other words, for a second branch subset entering the self-distillation branch, the larger the gap between its set-level reward and the maximum set-level reward, the worse the quality of the second branch subset and the greater the intensity of correction required; the smaller the gap between the set-level reward of the second branch subset and the maximum set-level reward value, the better the quality of the second branch subset and the less intensity of correction required.

[0192] It is understandable that if only policy optimization is used in the selection training phase, reward collapse will occur due to overfitting the evaluator. The self-distillation branch is crucial for the stability of the selection training phase. In this embodiment, for candidate subsets (within-group) whose ensemble-level reward is greater than the mean reward within the M candidate subsets, the policy gradient can be used in the policy optimization branch to increase their probability.

[0193] For candidate subsets whose ensemble-level rewards are less than or equal to the mean reward within a group, the self-distillation branch does not directly use negative gradients for suppression. Instead, it uses the candidate subset with the largest ensemble-level reward within the group as the lexical-level teacher policy, and performs distillation correction based on the weighted difference between its own ensemble-level reward and the largest ensemble-level reward. These two branches (policy optimization branch and self-distillation branch) form a complementary credit allocation, densifying the sparse ensemble-level rewards into lexical-level signals, effectively preventing reward collapse.

[0194] Understandably, after training the labeling model, recall encoder, ranking stage model, and selection strategy model, the following describes the application process of the above models in the search system.

[0195] Please see Figure 8 , Figure 8 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 3 It is understood that this data processing method can be executed by a computer device, which can be a terminal device (such as...). Figure 1 The terminal device 10a shown can be a server (such as...). Figure 1 The server shown is 10d, and this application does not limit this. Figure 8 As shown, the data processing method may include the following steps S301 to S306: Step S301: Obtain the second query information; determine the initial document set corresponding to the second query information based on the task type of the second query information; and obtain the second evaluation criteria of the second query information.

[0196] Specifically, when an object inputs second query information into the search system, the computer device can obtain the second query information received by the search system and acquire the second evaluation criteria for that second query information. The second query information refers to the text expression submitted by the object to the search system, and the second evaluation criteria refers to the specific evaluation criteria generated for the second query information.

[0197] The second evaluation criterion can be generated offline or online; this application does not limit this. When the evaluation criterion for the second query information is stored in the cache database of the search system, the evaluation criterion stored in the cache database can be directly used as the second evaluation criterion.

[0198] When the evaluation criteria for the second query information are not stored in the cache database of the search system, the system can search the cache database to see if candidate query information with the same or similar search intent as the second query information is stored. The determination of whether the search intent of the candidate query information is the same or similar to the search intent of the second query information can be made through vector similarity or by directly outputting the search intents of both through a large language model; this application does not limit the specific method used.

[0199] If the aforementioned candidate query information exists in the cache database, a second evaluation criterion for the second query information can be generated based on the evaluation criteria of the candidate query information. This is different from using the previous method. Figure 3 The complete generation process shown can improve the generation efficiency of the second evaluation criterion. If the above candidate query information does not exist in the cache database, then... Figure 3 The complete generation process shown generates the second evaluation criteria for the second query information.

[0200] The data retrieval scope of the second query can be determined based on the task type. Documents within this scope can be used to construct an initial document set. Determining the data retrieval scope by task type can narrow the search range and improve search efficiency.

[0201] The task type can refer to the search intent of the second query, such as navigational, informational, transactional, or comparative. The initial documents in a navigational initial document set can come from a specific website. The initial documents in an informational initial document set can come from general web page indexes, vertical knowledge bases, etc. The initial documents in a transactional initial document set can come from e-commerce indexes, local service libraries, etc. The initial documents in a comparative initial document set can come from evaluations, comparison articles, parameter databases, etc.

[0202] Step S302: The second query information is encoded by the recall encoder to obtain the second query encoding vector, and the initial documents in the initial document set are encoded by the recall encoder to obtain the document encoding vector of the initial documents.

[0203] Specifically, during the recall phase, the second query information and each initial document in the initial document set can be encoded using the same recall encoder to obtain their respective dense vectors, such as the second query encoding vector of the second query information and the document encoding vector of the initial document.

[0204] Step S303: Based on the vector similarity between the second query encoding vector and the document encoding vector, recall the candidate document set from the initial document set.

[0205] The calculation method for vector similarity is the same as that for the first similarity, and will not be repeated here. A candidate document set can be retrieved from the initial document set by using the vector similarity between the second query encoding vector and the document encoding vectors of each initial document. The number of documents in the candidate document set is less than the number of documents in the initial document set.

[0206] In one or more embodiments, similarity (including first similarity, second similarity, third similarity, fourth similarity, and vector similarity, etc.) can be calculated using cosine similarity, as shown in formula (16): (16) in, Representing vectors with vector Similarity between them and These are two vectors whose similarity is to be calculated, such as the first query encoding vector and the first sample encoding vector, or the second query encoding vector and the document encoding vector, etc. · represents the vector inner product; ||·|| represents the Euclidean norm of the vectors. Similarity can also be calculated using Euclidean distance, Manhattan distance, etc., and this application does not limit this method.

[0207] Step S304: The candidate documents in the candidate document set are evaluated for utility using the ranking stage model to obtain the ranking evaluation value of the candidate documents. Based on the ranking evaluation value, K recommended documents are determined in the candidate document set.

[0208] Specifically, during the ranking phase, the shared encoder and utility prediction component (main prediction component) in the ranking phase model can be used to evaluate the utility of candidate documents in the candidate document set (score them on the utility dimension) to obtain the utility ranking score of each candidate document. The utility ranking score at this time can be called the ranking evaluation value.

[0209] The candidate documents in the candidate document set are sorted in descending order of their ranking evaluation values ​​to obtain a descending document list. The top K candidate documents are then taken from the descending document list as the ranking result of the second query information. These K candidate documents can also be called K recommended documents, where K is a positive integer, such as 20 or 15, etc. This application does not limit the value of K.

[0210] Step S305: By selecting a strategy model, a subset of documents is selected from the K recommended documents. The utility of the document subset is evaluated according to the second evaluation criterion to obtain the evaluation result of the document subset.

[0211] Specifically, the selection strategy model trained above can be used to select a subset of documents from K recommended documents (ranking results); the number of documents in the subset is less than or equal to K. A judge conditioned on a second evaluation criterion scores the subset of documents to obtain an evaluation result, which can be a scalar reward for the subset.

[0212] Step S306: If the evaluation result indicates that the document subset meets the preset conditions, then the document subset is integrated through the target language model to output the second response text of the second query information.

[0213] In this embodiment, the preset condition can be that the scalar reward of the document subset output in the selection phase reaches a pre-set value (which can be denoted as T0). If the evaluation result (scalar reward) of the document subset output by the selection strategy model satisfies the preset condition, the document content contained in the document subset can be integrated by the target language model to output the second response text of the second query information. The target language model refers to a specific large language model used to generate the second response text, and the second response text refers to the answer generated by the target language model for the second query information.

[0214] If the evaluation result (scalar reward) of the document subset does not meet the preset conditions, it can fall back to the traditional diversity selection method. If the traditional diversity selection method is adopted, a new candidate subset is selected from the K recommended documents (ranking results), and the newly selected candidate subset is input into the target language model. The target language model outputs the second response text of the second query information.

[0215] Please see Figure 9 , Figure 9 This is a schematic diagram of an online service process provided in an embodiment of this application. For example... Figure 9 As shown, the online link of the search system can sequentially execute steps such as obtaining evaluation criteria (e.g., the second evaluation criterion), vector recall, utility ranking, subset selection, and large model generation.

[0216] If obtaining the evaluation criteria fails, such as due to a missing evaluation criterion, a fallback generation process can be used to generate an answer based on the current query information (such as the second query information). Optionally, this fallback generation process can be a link generation process sorted by relevance.

[0217] The vector recall process may include: in the recall phase, the current query information is encoded using a recall encoder to obtain a vector representation of the current query information, and vector retrieval is performed based on this vector representation to recall a set of candidate documents from the initial document set. This set of candidate documents can be referred to as the recall result of the recall phase.

[0218] The utility ranking process may include: using the utility prediction component in the ranking stage model to score the candidate documents in the candidate document set, obtaining the ranking evaluation value of each candidate document, and retaining the top K candidate documents as the ranking result according to the ranking evaluation value from largest to smallest, i.e. the above-mentioned K recommended documents.

[0219] The subset selection process includes: in the selection phase, the final document subset is output using the selection strategy model, and the evaluator is called to quickly score the document subset (for example, it can be a scalar reward). If the scalar reward does not reach the preset value T0, it means that the document subset does not meet the standard (does not meet the preset conditions), and it can degenerate into traditional diversity selection and reselect the document subset.

[0220] The large model generation process includes: delivering a subset of documents output from the selection phase to the target language model; integrating the information from the document content contained in the document subset through the target language model; and outputting the answer to the current query.

[0221] like Figure 9 As shown, heavyweight inference processes such as evaluation criterion generation, teacher annotation and distillation (distillation training of lightweight student models), recall and ranking training (recall training phase and ranking training phase), and selection strategy training (selection training phase) can all be performed offline, which can reduce costs.

[0222] It is understood that the solutions in this application can be applied to scenarios such as multi-faceted comparison queries, long-tail and entity-based queries, and queries sensitive to authority and timeliness. For example, in a multi-faceted comparison query scenario, assuming the current query information is "performance and price of a new tablet computer in a certain year", what the object truly needs is complete evidence composed of multiple aspects such as chip benchmark scores, battery life, and official pricing.

[0223] Traditional relevance-based search methods might select multiple documents that only superficially describe the chip name. While each document may be relatively relevant, they often contain redundant information, lack crucial data, and ultimately fail to provide answers regarding pricing or battery life. The solution proposed in this application generates a dedicated second evaluation criterion for the current query information. Positive entries can include "reporting chip benchmark scores and providing a comparison benchmark," "providing official pricing and applicable regions," and "providing actual battery life test duration," each carrying an importance weight. Negative entries can include "confusing different generations of models" and "citing prices from unofficial channels," each with a deduction penalty.

[0224] The labeling model can classify documents that only mention the chip name as low-level (e.g., auxiliary context) and documents that simultaneously include benchmark scores, battery life, and official pricing as high-level (e.g., complete evidence). A selection strategy model is used to select several documents with complementary coverage using ensemble-level rewards. The system outputs the answer to the current query, providing both performance and price information, with each data point traceable.

[0225] In long-tail and entity queries, the semantic matching ability of general open-source vector models is insufficient for long-tail queries (where the proportion of relevant documents in the candidate pool is extremely low) and entity queries containing names of people, organizations, and places. This can easily lead to the loss of a unique, crucial document during the recall phase, making it impossible to recover the lost document regardless of subsequent sorting or selection. The recall encoder in this application's embodiment, after in-domain comparative pre-training and utility alignment fine-tuning, can improve recall rates on both entity and long-tail class benchmarks. Even when faced with niche questions, it can find the webpage that truly answers the question, rather than returning a bunch of seemingly relevant but generalized content.

[0226] In query scenarios sensitive to authority and timeliness, authority, timeliness, and medical queries account for a significant portion of production traffic. The risk lies in referencing expired or non-authoritative sources. This application's embodiment explicitly introduces authority, timeliness, and quality as regularization terms during the ranking training phase using at least one auxiliary prediction component. Optionally, positive entries such as "must cite official or authoritative sources" and "must be recent information" can be directly written into the evaluation criteria, along with negative entries such as "cites expired data that has been corrected." This can improve the utility evaluation value of the final selected document subset, while maintaining stability in authority, timeliness, and quality, indicating that quality improvement does not come at the expense of authority and timeliness.

[0227] In one or more embodiments, the evaluation criteria stored in the cache database can be dynamically refreshed and reused. For example, for rapidly evolving topics, the evaluation criteria are periodically refreshed with fresh webpage evidence. If it is detected that the numerical or temporal facts involved in the evaluation entry have expired, regeneration is triggered, and a standard version number is introduced to make downstream labels and models traceable. Optionally, semantically similar query information can be clustered to share a standard skeleton, and only the fine-grained stage can be re-executed. Standard caching can be established for high-frequency query information, and lightweight fallback generation can be used for long-tail queries, which can reduce the generation cost of evaluation criteria.

[0228] In one or more embodiments, the evaluation items and verifiable sub-items in the evaluation criteria can be extended to cross-text, image and structured data, such as "must contain real product photos" and "must contain structured parameter tables", making the solutions of this application applicable to the utility evaluation of multimodal queries and multimodal evidence sets, thereby improving the applicability of the solutions of this application.

[0229] In one or more embodiments, set-level rewards can be used to perform reinforcement learning on the initial ranking model; or the output format of the labeling model can be adjusted, replacing the output of the labeling model from a single utility level to an item-level coverage vector, providing a supervision signal for coverage-aware greedy or subset selection during the ranking training phase, so that set-level signals (such as set-level rewards, set-level utility) can cover further upstream.

[0230] In one or more embodiments, human evaluation results from the search system are collected, and the evaluator is back-calibrated using these results. The scalar reward for the document subset is then regenerated using the calibrated evaluator. This scalar reward can be fed back into the labeling model to form a closed loop. Simultaneously, multiple evaluators can be used to score the document subset, reducing the risk of a single evaluator overfitting to the policy.

[0231] In this embodiment, by fine-tuning the utility alignment during the recall training phase, sorting the multi-task utility ranking during the ranking training phase, and conducting bi-branch training during the selection training phase, the quality of the query information answers can be improved after the entire link is online.

[0232] It is understood that in the specific implementation of this application, the query information of users in the search system may be involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from relevant institutions or departments or the users themselves is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions.

[0233] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 10 As shown, the data processing device 1 may include: a first response generation module 101, an evaluation criterion generation module 102, an imitation learning module 103, and a reinforcement learning module 104; The first response generation module 101 is used to obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths; there is no data interaction between the at least two data processing paths. The evaluation criterion generation module 102 is used to obtain candidate entries from at least two first response texts, obtain a recall result candidate set of the first query information, and generate a first evaluation criterion for the first query information based on the candidate entries and the recall result candidate set; the first evaluation criterion is used to generate a set-level reward for the first query information to select the training stage. The imitation learning module 103 is used to obtain the first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criterion and the reference utility label to obtain the candidate annotation model. The reinforcement learning module 104 is used to train the candidate annotation model using reinforcement learning based on the first evaluation criterion to obtain the label annotation model; the label annotation model is used to generate document utility labels for the first query information in the recall training phase and the ranking training phase.

[0234] In one or more embodiments, at least two data processing paths correspond one-to-one with at least two pre-trained language models, and each data processing path uses an independent pre-trained language model; the at least two pre-trained language models include the i-th pre-trained language model, where i is a positive integer; The first response generation module 101 generates at least two first response texts for the first query information according to at least two data processing paths, for use in performing the following steps: The first query information is input into the i-th pre-trained language model, and the first query information is converted into a query embedding vector through the embedding layer of the i-th pre-trained language model. The semantic encoding layer of the i-th pre-trained language model is used to semantically encode the query embedding vector to obtain the semantic encoding vector of the first query information. Perform text prediction processing on the semantic encoding vector to generate the first response text of the i-th pre-trained language model for the first query information.

[0235] In one or more embodiments, the evaluation criterion generation module 102 obtains a candidate set of recall results for the first query information, and generates a first evaluation criterion for the first query information based on the candidate entries and the candidate set of recall results, for performing the following steps: For each candidate item in the first response text, deduplication is performed to obtain the evaluation items of the first query information, and the evaluation items are constructed into an evaluation dimension architecture; Obtain the recall result candidate set of the first query information from the production log, perform semantic matching between the evaluation dimension architecture and the recall result candidate set, determine the matching content of the evaluation item in the recall result candidate set, and convert the matching content into verifiable sub-items of the evaluation item. If the verifiable sub-entry passes the fact check, the evaluation entry and the verifiable sub-entry will be combined into the first evaluation criterion of the first query information.

[0236] In one or more embodiments, the imitation learning module 103 acquires a first training corpus set and outputs reference utility labels for the first sample documents in the first training corpus set through a reference model, for performing the following steps: Obtain a second training corpus set, and determine the first training corpus set from the second training corpus set according to a preset ratio parameter; Obtain document annotation rules, and through the reference model, annotate the first sample document in the first training corpus according to the document annotation rules to obtain the reference utility label of the first sample document.

[0237] In one or more embodiments, the imitation learning module 103 performs imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criterion, and the reference utility label to obtain a candidate annotation model, which is used to perform the following steps: Using the initial annotation model, based on the first query information and the first evaluation criteria, the utility of the first sample document in the first training corpus is evaluated to obtain the first predicted utility category of the first sample document; Based on the first predicted utility category and the reference utility label, the imitation learning loss of the initial annotation model is determined. The network parameters of the initial annotation model are then corrected based on the imitation learning loss to obtain the candidate annotation model.

[0238] In one or more embodiments, the reinforcement learning module 104 performs reinforcement learning training on the candidate labeling model according to a first evaluation criterion to obtain a labeling model, which is used to perform the following steps: Obtain the second sample document from the second training corpus set, evaluate the utility of the second sample document using the candidate annotation model based on the first query information and the first evaluation criteria, and output the second predicted utility category of the second sample document; Based on the second predicted utility category and the first evaluation criterion, the policy reward for the second predicted utility category is determined. The network parameters of the candidate labeling model are then modified according to the policy reward to obtain the labeling model.

[0239] In one or more embodiments, the number of second sample documents is N, where N is an integer greater than 1; Reinforcement learning module 104 determines the policy reward for the second predicted utility category based on the second predicted utility category and the first evaluation criterion, for use in performing the following steps: Based on the first evaluation criterion, determine the utility accuracy corresponding to the second predicted utility category for each second sample document; Based on the document annotation rules, determine the structural matching degree corresponding to the second predicted utility category of each second sample document; The true ranking relationship between every two second sample documents in the N second sample documents is determined according to the first evaluation criterion, and the predicted ranking relationship between every two second sample documents is determined according to the second predicted utility category of every two second sample documents. Based on the actual ranking relationship and the predicted ranking relationship, the pairwise ranking accuracy is determined. The utility accuracy, structure matching accuracy and pairwise ranking accuracy are combined to determine the policy reward for the second predicted utility category of each second sample document.

[0240] In one or more embodiments, the data processing device 1 further includes: a recall training module 105; The recall training module 105 is used to obtain the recall training set of the first query information from the production log, and divide the recall training documents in the recall training set into first positive samples and first negative samples; the first positive sample is any recall training document in the recall training set. The recall training module 105 is also used to perform comparative training on the initial encoder using the first positive sample and the first negative sample to obtain the candidate encoder; The recall training module 105 is also used to output the document utility labels corresponding to the recalled training documents in the recall training set through the label annotation model, and to determine the second positive sample and the second negative sample in the recall training set according to the document utility labels; the priority of the document utility label corresponding to the second positive sample is higher than the priority of the document utility label corresponding to the second negative sample. The recall training module 105 is also used to perform utility alignment training on the candidate encoder using the second positive sample and the second negative sample to obtain the recall encoder.

[0241] In one or more embodiments, the recall training module 105 performs comparative training on the initial encoder using the first positive sample and the first negative sample to obtain a candidate encoder, which is used to perform the following steps: The first positive sample is encoded by the initial encoder to obtain the first sample encoding vector of the first positive sample; The first negative sample is encoded by the initial encoder to obtain the second sample encoding vector of the first negative sample; The first query encoding vector of the first query information is output by the initial encoder, the first similarity between the first query encoding vector and the first sample encoding vector is obtained, and the second similarity between the first query encoding vector and the second sample encoding vector is obtained. Based on the first and second similarities, the contrast loss of the initial encoder is determined, and the network parameters of the initial encoder are corrected according to the contrast loss to obtain the candidate encoder.

[0242] In one or more embodiments, the data processing device 1 further includes: a sorting training module 106; The sorting training module 106 is used to obtain the sorting training set of the first query information in the production log, and to encode the sorting training documents in the sorting training set through the shared encoder in the initial sorting model to obtain the third sample encoding vector of the sorting training documents. The ranking training module 106 is also used to map the third sample encoding vector to at least two prediction evaluation values ​​through at least two prediction components in the initial ranking model; the at least two prediction components include a main prediction component and at least one auxiliary prediction component, and each prediction component corresponds to a prediction evaluation value; The ranking training module 106 is also used to determine the multi-task loss of the initial ranking model based on the prediction evaluation value output by the main prediction component and the document utility label of the ranking training document, as well as the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component. The ranking training module 106 is also used to correct the network parameters of the initial ranking model based on the multi-task loss to obtain the ranking stage model; the ranking stage model includes the trained shared encoder and the master prediction component.

[0243] In one or more embodiments, the ranking training module 106 determines the multi-task loss of the initial ranking model based on the prediction evaluation value output by the main prediction component and the document utility labels of the ranking training documents, as well as the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component, for performing the following steps: The utility loss of the main prediction component is determined based on the prediction evaluation value output by the main prediction component and the document utility labels of the ranked training documents. Based on the predicted evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component, determine the auxiliary loss of each auxiliary prediction component; Obtain the control parameters corresponding to the auxiliary loss of each auxiliary prediction component. Based on the control parameters corresponding to the auxiliary loss of each auxiliary prediction component, perform a weighted summation of the auxiliary losses of each auxiliary prediction component to obtain the regularization term. The utility loss and regularization term are combined into the multi-task loss of the initial ranking model.

[0244] In one or more embodiments, the evaluation entries in the first evaluation criterion carry entry weights; The data processing device 1 also includes: a training selection module 107; The selection training module 107 is used to obtain the selection training set of the first query information. Through the initial policy model, M candidate subsets are sampled from the selection training set. The hit items of each candidate subset are determined in the first evaluation criterion. The set-level reward of each candidate subset is determined according to the item weight of the hit items; M is a positive integer. The selection training module 107 is also used to determine the mean reward of M candidate subsets based on the ensemble-level reward of each candidate subset, and to determine the standard deviation of the reward based on the mean reward and the ensemble-level reward of each candidate subset; The selection training module 107 is also used to determine the candidate subset with set-level reward greater than the average reward as the first branch subset, and the candidate subset with set-level reward less than or equal to the average reward as the second branch subset; The selection training module 107 is also used to determine the policy gain of the first branch subset based on the set-level reward and the mean reward, as well as the standard deviation of the reward, and to determine the policy optimization loss based on the policy gain and the predicted conditional probability corresponding to the first branch subset. The selection training module 107 is also used to determine the candidate subset with the largest set-level reward as the reference policy from the M candidate subsets, and to perform self-distillation training on the initial policy model based on the reference policy and the second branch subset to determine the distillation loss. The selection training module 107 is also used to merge the policy optimization loss and the distillation loss to obtain the joint loss of the initial policy model, and to update the policy of the initial policy model according to the joint loss to obtain the selected policy model.

[0245] In one or more embodiments, the data processing device 1 further includes: a query acquisition module 108, a document retrieval module 109, a document sorting module 110, a document selection module 111, and a second response generation module 112. The query acquisition module 108 is used to acquire second query information, determine the initial document set corresponding to the second query information based on the task type of the second query information, and acquire the second evaluation criteria of the second query information. The document recall module 109 is used to encode the second query information through the recall encoder to obtain the second query encoding vector, and to encode the initial documents in the initial document set through the recall encoder to obtain the document encoding vector of the initial documents. The document recall module 109 is also used to recall a set of candidate documents from the initial document set based on the vector similarity between the second query encoding vector and the document encoding vector. The document ranking module 110 is used to evaluate the utility of candidate documents in the candidate document set through the ranking stage model, obtain the ranking evaluation value of the candidate documents, and determine K recommended documents in the candidate document set based on the ranking evaluation value, where K is a positive integer; The document selection module 111 is used to select a subset of documents from K recommended documents through a selection strategy model, evaluate the utility of the document subset according to the second evaluation criterion, and obtain the evaluation result of the document subset. The second response generation module 112 is used to integrate information from a subset of documents through a target language model and output the second response text of the second query information if the evaluation result indicates that the preset conditions are met.

[0246] According to one embodiment of this application, the foregoing Figure 2 The data processing method shown can be related to the steps involved by Figure 10 The data processing apparatus 1 shown is executed by each module. For example, Figure 2 The step S101 shown can be performed by Figure 10 The first response generation module 101 shown is used to execute this. Figure 2 The step S102 shown can be performed by Figure 10 The evaluation criteria generation module 102 shown is used to execute the evaluation criteria. Figure 2 The step S103 shown can be performed by Figure 10 The imitation learning module 103 shown is used to execute this. Figure 2 The step S104 shown can be performed by Figure 10 The reinforcement learning module 104 shown is used to perform such actions.

[0247] According to one embodiment of this application, Figure 10 The modules in the data processing device 1 shown can be individually or entirely combined into one or more modules, or some of the modules can be further divided into at least two functionally smaller units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by at least two units, or the function of at least two modules can be implemented by one module. In other embodiments of this application, the data processing device 1 may also include other modules or units. In practical applications, these functions can also be implemented with the assistance of other modules, and can be implemented collaboratively by at least two modules.

[0248] In this embodiment, multiple pre-trained language models are used to independently generate first response texts for the same query information (first query information). Candidate entries are extracted from each first response text, and these candidate entries are anchored to the actual recalled result candidate set to generate a first evaluation criterion for the first query information. In other words, a dedicated evaluation criterion can be generated for each query information. This evaluation criterion allows the recall training stage, ranking training stage, and selection training stage to be aligned and optimized towards the same goal of "answer quality," thereby improving the response quality of the query information.

[0249] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 11As shown, the computer device 1000 can be a terminal device, for example, the one described above. Figure 1 The terminal device 10a in the corresponding embodiment; or it can be a server, for example, Figure 1 The server 10d shown is illustrated below. For ease of understanding, the computer device is described below as a terminal device; the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located remotely from the aforementioned processor 1001. Figure 11 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0250] The network interface 1004 in the computer device 1000 can also provide network communication functions, and the optional user interface 1003 can also include a display screen and a keyboard. Figure 11 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve: Obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths; there is no data interaction between at least two data processing paths. Candidate entries are obtained from at least two first response texts, and a candidate set of recall results for the first query information is obtained. Based on the candidate entries and the candidate set of recall results, a first evaluation criterion for the first query information is generated. The first evaluation criterion is used to generate a set-level reward for the first query information to select the training phase. Obtain the first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criteria and the reference utility label to obtain the candidate annotation model; The candidate annotation model is trained using reinforcement learning based on the first evaluation criterion to obtain the label annotation model; the label annotation model is used to generate document utility labels for the recall training phase and the ranking training phase for the first query information.

[0251] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 2 , Figure 6 as well as Figure 8 The description of the data processing method in any of the embodiments can also be performed as described above. Figure 10 The description of the data processing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0252] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned data processing device 1. The computer program includes computer instructions, and when the processor executes the computer instructions, it can execute the aforementioned... Figure 2 , Figure 6 as well as Figure 8 The description of the data processing method in any of the embodiments is redundant and will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated here either. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed and executed on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network. Multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0253] Furthermore, it should be noted that this application also provides a computer program product, which may include a computer program that can be stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the computer device to perform the aforementioned... Figure 2 , Figure 6 as well as Figure 8The description of the data processing method in any of the embodiments is redundant and will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated here either. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0254] It is understood that in the description of the embodiments of this application, "when," "if," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not limited to a time, nor do they require the device to have a judgment action when it is implemented, nor do they mean that there are other limitations.

[0255] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different media content, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0256] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0257] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data display device to create a machine, such that the instructions executable by the processor of the computer or other programmable data display device generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data display device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable display device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0258] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0259] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths; There is no data interaction between the at least two data processing paths; Candidate entries are obtained from the at least two first response texts, and a recall result candidate set of the first query information is obtained. Based on the candidate entries and the recall result candidate set, a first evaluation criterion for the first query information is generated. The first evaluation criterion is used to generate a set-level reward for the first query information to select a training phase. Obtain a first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criterion and the reference utility label to obtain a candidate annotation model; The candidate annotation model is trained using reinforcement learning based on the first evaluation criterion to obtain a tag annotation model; the tag annotation model is used to generate document utility tags for the first query information in the recall training phase and the ranking training phase.

2. The method according to claim 1, characterized in that, The at least two data processing paths correspond one-to-one with at least two pre-trained language models, and each data processing path uses an independent pre-trained language model; the at least two pre-trained language models include the i-th pre-trained language model, where i is a positive integer; The process of generating at least two first response texts for the first query information according to at least two data processing paths includes: The first query information is input into the i-th pre-trained language model, and the first query information is converted into a query embedding vector through the embedding layer of the i-th pre-trained language model. The semantic encoding layer of the i-th pre-trained language model is used to semantically encode the query embedding vector to obtain the semantic encoding vector of the first query information. The semantic encoding vector is subjected to text prediction processing to generate the first response text of the i-th pre-trained language model for the first query information.

3. The method according to claim 1, characterized in that, The step of obtaining a candidate set of recall results for the first query information, and generating a first evaluation criterion for the first query information based on the candidate entries and the candidate set of recall results, includes: For each candidate item of the first response text, deduplication is performed to obtain the evaluation items of the first query information, and the evaluation items are constructed into an evaluation dimension architecture; Obtain the recall result candidate set of the first query information from the production log, perform semantic matching between the evaluation dimension architecture and the recall result candidate set, determine the matching content of the evaluation item in the recall result candidate set, and convert the matching content into a verifiable sub-item of the evaluation item. If the verifiable sub-entry passes the fact verification, then the evaluation entry and the verifiable sub-entry are merged into the first evaluation criterion of the first query information.

4. The method according to claim 1, characterized in that, The step of obtaining the first training corpus set and outputting the reference utility label of the first sample document in the first training corpus set through the reference model includes: Obtain a second training corpus set, and determine the first training corpus set from the second training corpus set according to a preset ratio parameter; Obtain document annotation rules, and through a reference model, annotate the first sample document in the first training corpus according to the document annotation rules to obtain the reference utility label of the first sample document.

5. The method according to claim 1, characterized in that, The step of training the initial annotation model through imitation learning based on the first training corpus set, the first query information, the first evaluation criterion, and the reference utility label to obtain a candidate annotation model includes: Using the initial annotation model, the utility of the first sample document in the first training corpus is evaluated based on the first query information and the first evaluation criteria to obtain the first predicted utility category of the first sample document; Based on the first predicted utility category and the reference utility label, the imitation learning loss of the initial labeling model is determined, and the network parameters of the initial labeling model are corrected according to the imitation learning loss to obtain a candidate labeling model.

6. The method according to claim 1, characterized in that, The step of training the candidate labeling model using reinforcement learning based on the first evaluation criterion to obtain a labeling model includes: Obtain the second sample document from the second training corpus set, and evaluate the utility of the second sample document using the candidate annotation model based on the first query information and the first evaluation criterion, and output the second predicted utility category of the second sample document; Based on the second predicted utility category and the first evaluation criterion, the policy reward for the second predicted utility category is determined, and the network parameters of the candidate labeling model are corrected according to the policy reward to obtain the labeling model.

7. The method according to claim 6, characterized in that, The second sample document contains N documents, where N is an integer greater than 1. The step of determining the strategy reward for the second predicted utility category based on the second predicted utility category and the first evaluation criterion includes: Based on the first evaluation criterion, determine the utility accuracy corresponding to the second predicted utility category for each second sample document; Based on the document annotation rules, determine the structural matching degree corresponding to the second predicted utility category of each second sample document; The true ranking relationship between every two second sample documents in the N second sample documents is determined according to the first evaluation criterion, and the predicted ranking relationship between every two second sample documents is determined according to the second predicted utility category of every two second sample documents. Based on the actual ranking relationship and the predicted ranking relationship, the pairwise ranking accuracy is determined. The utility accuracy, the structural matching degree, and the pairwise ranking accuracy are combined to determine the policy reward for the second predicted utility category of each second sample document.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the recall training set of the first query information from the production log, and divide the recall training documents in the recall training set into a first positive sample and a first negative sample; the first positive sample is any recall training document in the recall training set; By comparing and training the initial encoder using the first positive sample and the first negative sample, a candidate encoder is obtained; The labeling model outputs the document utility labels corresponding to the recalled training documents in the recalled training set. Based on the document utility labels, a second positive sample and a second negative sample are determined in the recalled training set. The priority of the document utility label corresponding to the second positive sample is higher than the priority of the document utility label corresponding to the second negative sample. The candidate encoder is trained using the second positive sample and the second negative sample to obtain the recall encoder.

9. The method according to claim 8, characterized in that, The step of comparing and training the initial encoder using the first positive sample and the first negative sample to obtain a candidate encoder includes: The first positive sample is encoded by the initial encoder to obtain the first sample encoding vector of the first positive sample; The first negative sample is encoded by the initial encoder to obtain the second sample encoding vector of the first negative sample; The first query encoding vector of the first query information is output by the initial encoder, the first similarity between the first query encoding vector and the first sample encoding vector is obtained, and the second similarity between the first query encoding vector and the second sample encoding vector is obtained. Based on the first similarity and the second similarity, the contrast loss of the initial encoder is determined, and the network parameters of the initial encoder are corrected according to the contrast loss to obtain a candidate encoder.

10. The method according to claim 1, characterized in that, The method further includes: Obtain the sorting training set of the first query information from the production log, and encode the sorting training documents in the sorting training set through the shared encoder in the initial sorting model to obtain the third sample encoding vector of the sorting training documents. The third sample encoding vector is mapped to at least two predicted evaluation values ​​through at least two prediction components in the initial ranking model; the at least two prediction components include a main prediction component and at least one auxiliary prediction component, and each prediction component corresponds to a predicted evaluation value; The multi-task loss of the initial ranking model is determined based on the prediction evaluation value output by the main prediction component and the document utility label of the ranking training document, as well as the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component. Based on the multi-task loss, the network parameters of the initial ranking model are corrected to obtain the ranking stage model; the ranking stage model includes a trained shared encoder and a master prediction component.

11. The method according to claim 10, characterized in that, The step of determining the multi-task loss of the initial ranking model based on the prediction evaluation value output by the main prediction component and the document utility label of the ranked training document, and the prediction evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component, includes: The utility loss of the main prediction component is determined based on the prediction evaluation value output by the main prediction component and the document utility label of the ranked training document. Based on the predicted evaluation value output by each auxiliary prediction component and the label information of each auxiliary prediction component, the auxiliary loss of each auxiliary prediction component is determined; Obtain the control parameters corresponding to the auxiliary loss of each auxiliary prediction component, and perform a weighted summation of the auxiliary losses of each auxiliary prediction component based on the control parameters corresponding to the auxiliary loss of each auxiliary prediction component to obtain the regularization term; The utility loss and the regularization term are combined into the multi-task loss of the initial ranking model.

12. The method according to claim 1, characterized in that, The evaluation items in the first evaluation criterion carry item weights; The method further includes: Obtain the selected training set of the first query information, sample M candidate subsets in the selected training set through the initial policy model, determine the hit items of each candidate subset in the first evaluation criterion, and determine the set-level reward of each candidate subset according to the item weight of the hit items; M is a positive integer; Based on the set-level reward of each candidate subset, determine the mean reward of the M candidate subsets, and based on the mean reward and the set-level reward of each candidate subset, determine the standard deviation of the reward; The candidate subset whose set-level reward is greater than the average reward is determined as the first branch subset, and the candidate subset whose set-level reward is less than or equal to the average reward is determined as the second branch subset; Based on the set-level reward of the first branch subset, the mean reward, and the standard deviation of the reward, determine the policy gain of the first branch subset; based on the policy gain and the predicted conditional probability corresponding to the first branch subset, determine the policy optimization loss. Among the M candidate subsets, the candidate subset with the largest set-level reward is determined as the reference policy. Based on the reference policy and the second branch subset, the initial policy model is trained by self-distillation to determine the distillation loss. The policy optimization loss and the distillation loss are combined to obtain the joint loss of the initial policy model. The initial policy model is then updated based on the joint loss to obtain the selected policy model.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Obtain the second query information; determine the initial document set corresponding to the second query information based on the task type of the second query information; and obtain the second evaluation criteria for the second query information. The second query information is encoded by the recall encoder to obtain the second query encoding vector. The initial documents in the initial document set are encoded by the recall encoder to obtain the document encoding vector of the initial documents. Based on the vector similarity between the second query encoding vector and the document encoding vector, a candidate document set is recalled from the initial document set; The candidate documents in the candidate document set are evaluated for utility using a ranking stage model to obtain a ranking evaluation value for the candidate documents. Based on the ranking evaluation value, K recommended documents are determined in the candidate document set, where K is a positive integer. By selecting a strategy model, a subset of documents is selected from the K recommended documents, and the utility of the document subset is evaluated according to the second evaluation criterion to obtain the evaluation result of the document subset; If the evaluation result indicates that the preset conditions are met, the document subset is integrated using the target language model, and the second response text of the second query information is output.

14. A data processing apparatus, characterized in that, include: The first response generation module is used to obtain the first query information and generate at least two first response texts for the first query information according to at least two data processing paths. There is no data interaction between the at least two data processing paths; An evaluation criterion generation module is used to obtain candidate entries from the at least two first response texts, obtain a recall result candidate set of the first query information, and generate a first evaluation criterion for the first query information based on the candidate entries and the recall result candidate set; the first evaluation criterion is used to generate a set-level reward for the first query information to select a training phase. The imitation learning module is used to acquire a first training corpus set, output the reference utility label of the first sample document in the first training corpus set through the reference model, and perform imitation learning training on the initial annotation model based on the first training corpus set, the first query information, the first evaluation criterion and the reference utility label to obtain a candidate annotation model. The reinforcement learning module is used to train the candidate annotation model using reinforcement learning according to the first evaluation criterion to obtain a label annotation model; the label annotation model is used to generate document utility labels for the first query information in the recall training phase and the ranking training phase.

15. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 13.

17. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 13.