Evaluation method for target processing node of language generation model, and answer generation method

Abstract

The embodiments of the present description relate to the technical field of artificial intelligence, and particularly relate to an evaluation method for a target processing node of a language generation model, and an answer generation method. The evaluation method for a target processing node of a language generation model comprises: parsing a text document in a document data set, generating a plurality of pieces of query data, and generating a first keyword and a second keyword group which correspond to each of the plurality of pieces of query data, wherein each piece of query data comprises a reference question and a reference answer corresponding to the reference question; using a target processing node of the language generation model to generate a predicted answer corresponding to each reference question in each piece of query data; and on the basis of the first keyword and the second keyword group which correspond to each piece of query data, the reference question and the reference answer in each piece of query data, and the predicted answer corresponding to each reference question, performing evaluation on the target processing node, so as to obtain an evaluation result of the target processing node.

Description

Evaluation methods for target processing nodes and answer generation methods in language generation models

[0001] This disclosure claims priority to Chinese Patent Application No. 202411441950.8, filed with the China Patent Office on October 15, 2024, entitled “A Method for Evaluating Target Processing Nodes and Generating Answers in a Language Generation Model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a method for evaluating target processing nodes and generating answers in a language generation model. Background Technology

[0003] Retrieval-Augmented Generation (RAG) is an artificial intelligence approach that combines information retrieval techniques with generative models. Its main goal is to enhance the capabilities of generative models by incorporating information from external knowledge bases, enabling them to provide more accurate, timely, and context-relevant output. Typically, the application of RAG includes stages for knowledge retrieval, ranking, and generation, which constitute the entire chain of the RAG model.

[0004] However, the current evaluation of retrieval generation models is poor, which in turn affects the quality of the answers generated by the retrieval generation models. Therefore, there is an urgent need for an effective technical solution to solve the above problems. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a method for evaluating target processing nodes of a language generation model. One or more embodiments of this specification also relate to a target processing node evaluation apparatus for a language generation model, an answer generation method, an answer generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a method for evaluating target processing nodes in a language generation model is provided, comprising:

[0007] The text documents in the document dataset are parsed to generate multiple query data, and a first keyword and a second keyword group corresponding to each query data are generated. Each query data includes a reference question and a reference answer corresponding to the reference question.

[0008] The target processing node of the language generation model is used to generate the predicted answer for each reference question in each query data.

[0009] The target processing node is evaluated based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation result of the target processing node.

[0010] According to a second aspect of the embodiments of this specification, a target processing node evaluation apparatus for a language generation model is provided, comprising:

[0011] The first generation module is configured to parse text documents in the document dataset, generate multiple query data, and generate a first keyword and a second keyword group corresponding to each query data in the multiple query data, wherein each query data includes a reference question and a reference answer corresponding to the reference question;

[0012] The second generation module is configured to use the target processing node of the language generation model to generate the predicted answer corresponding to each reference question in each query data.

[0013] The evaluation module is configured to evaluate the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference question and the reference answer in each query data, and the predicted answer corresponding to each reference question, and obtain the evaluation result of the target processing node.

[0014] According to a third aspect of the embodiments of this specification, an answer generation method is provided, comprising:

[0015] Define the target problem;

[0016] The target answer corresponding to the target question is generated using the target processing node of the language generation model;

[0017] The language generation model is a language generation model obtained by adjusting the target processing node evaluation method of the language generation model provided in the first aspect of the embodiments of this specification.

[0018] According to a fourth aspect of the embodiments of this specification, an answer generation apparatus is provided, comprising:

[0019] The module is configured to define the target problem.

[0020] The generation module is configured to use the target processing node of the language generation model to generate the target answer corresponding to the target question.

[0021] The language generation model is a language generation model obtained by adjusting the target processing node evaluation method of the language generation model provided in the first aspect of the embodiments of this specification.

[0022] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0023] Memory and processor;

[0024] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0025] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0026] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0027] This specification provides an embodiment of a method for evaluating a target processing node of a language generation model, comprising: parsing text documents in a document dataset, generating multiple query data, and generating a first keyword and a second keyword group corresponding to each query data, wherein each query data includes a reference question and a reference answer corresponding to the reference question; using the target processing node of the language generation model to generate a predicted answer corresponding to each reference question in each query data; and evaluating the target processing node based on the first keyword and the second keyword group corresponding to each query data, the reference question and the reference answer in each query data, and the predicted answer corresponding to each reference question, to obtain an evaluation result of the target processing node.

[0028] The above method constructs a document dataset and generates multiple query data based on the text documents in the dataset. It then generates first and second keyword groups for each query data item. Utilizing the target processing node of the language generation model, it generates predicted answers for each reference question in each query data item. Based on the first and second keyword groups for each query data item, the reference questions and answers, and the predicted answers for each reference question, the target processing node is evaluated to obtain its evaluation results. This process improves the evaluation effect of the language generation model by using the first and second keyword groups, enabling the identification of problematic nodes within the model. This facilitates subsequent optimization of the language processing model based on the evaluation results, ensuring the processing capability of the language generation model and ultimately guaranteeing the quality of the answers generated. Attached Figure Description

[0029] Figure 1 is a schematic diagram of an application scenario of a target processing node evaluation method for a language generation model provided in an embodiment of this specification;

[0030] Figure 2 is a flowchart of a target processing node evaluation method for a language generation model provided in one embodiment of this specification;

[0031] Figure 3 is a flowchart of the data preparation process for a target processing node evaluation method for a language generation model provided in one embodiment of this specification.

[0032] Figure 4 is a flowchart of the evaluation process of a target processing node evaluation method for a language generation model provided in one embodiment of this specification;

[0033] Figure 5 is a schematic diagram of the structure of a target processing node evaluation device for a language generation model provided in one embodiment of this specification;

[0034] Figure 6 is a flowchart of an answer generation method provided in one embodiment of this specification;

[0035] Figure 7 is a schematic diagram of an answer generation device provided in one embodiment of this specification;

[0036] Figure 8 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0037] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0038] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0040] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0041] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0042] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0043] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0044] BLEU: Bilingual Evaluation Understudy, is a metric used to evaluate the quality of generated answers. It measures the quality of answer generation by comparing the degree of n-gram (continuous word sequences) overlap between the machine-generated answer and the reference answer. A higher BLEU score indicates a higher similarity between the machine-generated answer and the reference answer.

[0045] Rouge-L: Used to evaluate the quality of answer generation. It measures the accuracy of the summary by calculating the overlap of n-grams between the system-generated answer and the reference answer. Rouge-L pays particular attention to the longest common subsequence (LCS), a method for measuring the similarity between two sequences.

[0046] n-gram: A sequence of n consecutive units (such as words, phonemes, characters, etc.) used to represent text.

[0047] LCS stands for Longest Common Subsequence, which refers to the longest common subsequence among two or more sequences. In text similarity calculations, LCS can be used to measure the similarity between two text segments.

[0048] LlamaIndex is a data framework for enhancing large language model (LLM) applications. It enhances the contextual understanding capabilities of LLMs through indexing and retrieval techniques, enabling them to better utilize private or domain-specific data.

[0049] JSON: JavaScript Object Notation, is a lightweight data interchange format.

[0050] PDF: Portable Document Format, a file format.

[0051] DOC: A file format.

[0052] PPT: A file format.

[0053] XLSX: A file format.

[0054] This specification provides a method for evaluating the target processing node of a language generation model. It also relates to a device for evaluating the target processing node of a language generation model, an answer generation method, an answer generation device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0055] Referring to Figure 1, Figure 1 shows a schematic diagram of an application scenario of a target processing node evaluation method for a language generation model according to an embodiment of this specification. The target processing node evaluation method for the language generation model specifically includes the following steps.

[0056] The text documents in the document dataset are parsed to generate multiple query data, and a first keyword and a second keyword group corresponding to each query data are generated. Each query data includes a reference question and a reference answer corresponding to the reference question.

[0057] The target processing node of the language generation model is used to generate the predicted answer for each reference question in each query data.

[0058] The target processing node is evaluated based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation result of the target processing node.

[0059] Figure 1 includes a model evaluation platform 102, which can be used to execute the target processing node evaluation method of the language generation model, evaluate and optimize the target processing node of the language generation model. Specifically, text documents in the document dataset can be parsed to obtain multiple query data, and first keywords and second keyword groups corresponding to each query data can be generated. The target processing node of the language generation model can be used to generate predicted answers corresponding to each reference question in each query data. Based on the first keywords and second keyword groups corresponding to each query data, the reference questions and answers in each query data, and the predicted answers corresponding to each reference question, the target processing node is evaluated to obtain the evaluation results, thereby achieving the evaluation of the language generation model.

[0060] The model evaluation platform 102 can be understood as a server providing various services, including physical servers and cloud servers. For example, it can be a server providing communication services to multiple clients, a server supporting the models used on clients for background training, or a server processing data sent by clients. It should be noted that the model evaluation platform 102 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The model evaluation platform 102 can also be a server in a distributed system, or a server integrated with blockchain. Furthermore, the model evaluation platform 102 can be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0061] Referring to Figure 2, Figure 2 shows a flowchart of a target processing node evaluation method for a language generation model according to an embodiment of this specification, which specifically includes the following steps.

[0062] Step 202: Parse the text documents in the document dataset, generate multiple query data, and generate the first keyword and second keyword group corresponding to each query data in the multiple query data, wherein each query data includes a reference question and a reference answer corresponding to the reference question.

[0063] In this context, a document dataset can be understood as a collection of data containing multiple text documents. Query data can be understood as reference questions and answers for the text documents in the document dataset. Reference questions are questions posed based on the text documents, and reference answers are the answers to these reference questions based on the text documents. For example, if a text document describes the current state and development level of intelligent vehicles, then the reference questions would be about intelligent vehicles, and the reference answers would be the responses to these questions based on the content of the text document. The first keyword can be understood as a coarse-grained keyword, which can be a word that can simply summarize the theme of the text document, possessing representativeness and relevance. For example, for a document describing the current state and development level of intelligent vehicles, the first keyword for the query data corresponding to that text document could be "intelligent vehicle." The first keyword can be one or two words. The second keyword group can be understood as a fine-grained keyword group, which can be information points extracted from the text documents corresponding to the query data. There can be multiple second keyword groups; for example, a given query data might have two second keyword groups, each containing one or more fine-grained keywords.

[0064] Specifically, it can parse text documents in a document dataset, generate multiple query data sets containing reference questions and answers, and generate coarse-grained keywords and fine-grained keyword groups for each query data set.

[0065] In practical applications, when creating document datasets, text documents in various formats can be collected from online platforms. These formats include, but are not limited to, PDF, DOC, PPT, and XLSX formats. The content of these documents can include, but is not limited to, financial, technical, medical, commercial, and internet content. They can also include industry reports, manuals, and statistical data. The text documents can be uploaded within a preset time period, such as within the last year or several years, ensuring a rich and timely information source for subsequent evaluation of the language generation model.

[0066] For example, text documents in a document dataset can be parsed to generate query data A, query data B, and query data C. Query data A can be a reference question and answer for text document A, query data B can be a reference question and answer for text document B, and query data C can be a reference question and answer for text document C. Coarse-grained keywords and multiple fine-grained keyword groups are generated for query data A, query data B, and query data C. For ease of understanding, this embodiment uses the generation of three query data sets A, B, and C as an example. In practical applications, parsing text documents in a document dataset can generate a large number of query data sets, such as 100, 1000, or 5000. This embodiment does not limit the number of query data sets generated.

[0067] Furthermore, the text documents in the parsed document dataset generate multiple query data, including:

[0068] The text documents in the document dataset are sliced ​​to obtain multiple document fragments;

[0069] The multiple document fragments are sequentially identified as document fragments to be processed, and multiple query data corresponding to the document fragments to be processed are generated, wherein the query types of the multiple query data are different.

[0070] The query types can include, but are not limited to, factual queries, analytical queries, comparative queries, tutorial queries, application queries, and evaluation queries. Factual queries can be understood as queries seeking specific and explicit facts or data, such as the year a building was built. Analytical queries can be understood as queries seeking analysis, explanation, or summaries of specific concepts, terms, or phenomena, such as what the advantages of renewable energy are. Comparative queries can be understood as queries seeking comparisons of information from different dimensions, such as what the differences are between region A and region B. Tutorial queries can be understood as queries seeking steps to perform a task or process, such as what the steps are to install a certain software. Sequentially identifying multiple document fragments as document fragments to be processed and generating multiple query data corresponding to each document fragment can be understood as performing the operation of generating multiple query data for each document fragment.

[0071] Based on this, multiple text documents included in the document dataset can be sliced ​​to obtain multiple document fragments, and these multiple document fragments can be sequentially identified as document fragments to be processed, generating query data of different query types corresponding to the document fragments to be processed.

[0072] In another embodiment of this specification, query data of different query types corresponding to text documents can also be generated directly.

[0073] In practical applications, a question generator can be used to construct reference questions for various query types using each document fragment as given content, and then generate corresponding reference answers for those reference questions. Specifically, the question generator can input question construction requirements and various query types, allowing it to construct reference questions based on these requirements. The requirements may include: 1. The questions should conform to human question-asking style and be independent of each other; 2. The questions should be clear, specific, and detailed, avoiding vague pronouns or broad references; 3. The questions should be answerable from the given content, and the relevant content should be searchable through the questions; 4. If a query type cannot generate a question that meets the above requirements, the corresponding query type will be empty, and the output will be "Unable to generate". The output format of the question generator can be a JSON dictionary. For example, if the document fragment input to the question generator is a company introduction, then the reference questions for fact-based queries could be "In what year was the company founded?", for analysis-based queries could be "What is the company's market coverage?", and for comparison and tutorial-based queries, the output could be "Unable to generate", along with the corresponding reference answers for each reference question.

[0074] Furthermore, the generated query data can be evaluated based on query criteria. These criteria may include: semantic clarity and accuracy, conformity to the definition of the corresponding query type, and the ability to be inferred from document fragments. The generated query data can be filtered based on these criteria, retaining only those that meet them. In practical applications, evaluation can be performed using a model or manually; this specification does not limit the specific methods used in the embodiments.

[0075] For example, taking text document A in the document dataset as an example, text document A can be sliced ​​to obtain document fragments A1, A2, and A3. This generates query data for four query types corresponding to document fragment A1, document fragment A2, and document fragment A3. In other words, for text document A, 12 query data are generated. Similarly, similar processing is performed on each text document in the document dataset to obtain multiple query data for subsequent evaluation.

[0076] In addition, text documents can include semi-structured text documents and structured text documents. Semi-structured text documents can be parsed and then sliced, or structured text documents can be directly sliced ​​to obtain multiple document fragments corresponding to each text document.

[0077] In summary, slicing text documents can reduce computational resource consumption and make the generated query data more targeted, ensuring the richness and detail of the query data, which facilitates subsequent evaluation of the language generation model based on multiple query data.

[0078] In practical applications, each query can be labeled with multi-granularity keywords, including coarse-grained and fine-grained keywords. The specific implementation method is as follows:

[0079] The generation of the first keyword and second keyword group corresponding to each of the multiple query data includes:

[0080] Identify the document fragment to be processed corresponding to each of the multiple query data;

[0081] Information is extracted from the document fragments to be processed to determine the first keyword corresponding to each query data;

[0082] Based on the reference questions in each query data, multiple information points corresponding to the reference questions are determined from the document fragments to be processed, and based on the multiple information points, the second keyword group corresponding to each query data is determined.

[0083] The multiple information points corresponding to the reference question can be understood as the key points for answering the reference question.

[0084] Specifically, for the multiple query data generated above, for each query data, the corresponding document fragment to be processed can be determined, information can be extracted from the document fragment to determine the coarse-grained keywords corresponding to the query data. Based on the reference question included in the query data, multiple information points corresponding to the reference question are obtained from the document fragment to be processed, and these multiple information points are used as the fine-grained keyword group corresponding to the query data.

[0085] In practical applications, coarse-grained keywords are representative and relevant words extracted from query data and document fragments to be processed. They typically consist of one or two words and concisely summarize the theme of the document fragment. Fine-grained keywords are defined as a list, with each list corresponding to an information point extracted from the document fragment. A large model can be used to generate coarse-grained and fine-grained keyword groups. The generated multi-granular keywords must meet the following keyword criteria: 1. Both coarse-grained and fine-grained keywords originate from a text fragment within the given document fragment to be processed, not necessarily a fragment of the reference question, and cannot contain text content not present in the document fragment; 2. Coarse-grained keywords are generally entities, representing one or more entities that can represent paragraphs and the main body of the question; 3. Fine-grained keywords are divided according to the reference question, with each sub-list corresponding to an information point. For the same document fragment to be processed, the grouping of fine-grained keywords (i.e., the second keyword group) will differ depending on the reference question. The first and second keyword groups generated by the large model can be filtered through manual evaluation or model evaluation, retaining those that meet the above keyword criteria.

[0086] In summary, by generating multi-granular keywords, the performance evaluation of the subsequent retrieval stage reduces the reliance on document slicing and enables the evaluation of the context retrieved in the subsequent retrieval stage.

[0087] Step 204: Utilize the target processing node of the language generation model to generate predicted answers corresponding to each reference question in the query data.

[0088] The language generation model can be understood as a retrieval-enhanced generative model (RAG), or other large models using the target processing node evaluation method of the language generation model applicable to the embodiments of this specification. The predicted answer can be understood as the predicted answer output by the language generation model to the reference question given as input to the sentence.

[0089] In practical applications, the language generation model includes multiple target processing nodes;

[0090] The target processing node utilizing the language generation model generates predicted answers for each reference question in each query data, including:

[0091] By utilizing multiple target processing nodes of the language generation model, predicted answers are generated for each reference question in each query data.

[0092] The step of evaluating the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation result of the target processing node, includes:

[0093] The evaluation of the multiple target processing nodes is performed based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation results of the multiple target processing nodes.

[0094] Specifically, the language generation model may include multiple target processing nodes, including but not limited to document slicing nodes, document retrieval nodes, document sorting nodes, and answer generation nodes. In one embodiment of this specification, the language generation model may include document retrieval nodes and answer generation nodes; in another embodiment, the language generation model may include document sorting nodes, document retrieval nodes, and answer generation nodes; in yet another embodiment, the language generation model may further include document slicing nodes, document retrieval nodes, document sorting nodes, and answer generation nodes, but this embodiment does not limit the scope of the embodiments.

[0095] Therefore, the target processing node evaluation method for the language generation model provided in the embodiments of this specification can evaluate any one or all target processing nodes in the language generation model, so as to realize the customized evaluation and full-link evaluation of the language generation model.

[0096] In practice, the target processing nodes include document retrieval nodes, document sorting nodes, and answer generation nodes;

[0097] The target processing node utilizing the language generation model generates predicted answers for each reference question in each query data, including:

[0098] Each reference question in the query data is sequentially identified as a reference question to be processed.

[0099] Using the document retrieval node of the language generation model, retrieve multiple reference document fragments corresponding to the reference question to be processed;

[0100] Using the document sorting node of the language generation model, the multiple reference document fragments are sorted to obtain a reference sorting result;

[0101] Using the answer generation node of the language generation model, a combined document fragment is determined from the multiple reference document fragments according to the reference sorting result, and a predicted answer corresponding to the reference question to be processed is generated according to the combined document fragment, thereby obtaining the predicted answer corresponding to each reference question in each query data.

[0102] In this process, each reference question in each query data is sequentially identified as a reference question to be processed. This can be understood as performing the above processing on each reference question in each query data to obtain the predicted answer corresponding to each reference question.

[0103] Specifically, taking the processing of a reference question using a language generation model as an example, the reference question can be input into the language generation model. The document retrieval node of the language generation model retrieves multiple reference document fragments corresponding to the reference question. The document sorting node of the language generation model sorts these multiple reference document fragments to obtain a reference sorting result. The answer generation node of the language generation model, based on this reference sorting result, determines a combined document fragment from the multiple reference document fragments and generates a predicted answer corresponding to the reference question based on this combined document fragment. Similarly, similar processing is performed on reference questions in each query dataset to obtain predicted answers for reference questions across multiple query datasets.

[0104] In practice, when sorting multiple reference document fragments to obtain a reference sorting result, the multiple reference document fragments can be sorted according to the text similarity between each reference document fragment and the reference question, and the reference sorting result can be obtained by sorting from high to low text similarity.

[0105] For example, a document retrieval node can be used to retrieve 100 reference document fragments corresponding to the reference question to be processed. A document sorting node can be used to sort the 100 reference document fragments according to the text similarity from high to low to obtain a reference sorting result. Based on the reference sorting result, a combined document fragment can be determined from the 100 reference document fragments. Based on the combined document fragment, a predicted answer corresponding to the reference question to be processed can be generated.

[0106] In practical applications, the reference question for input language generation models is: By 2025, what level will the technological innovation and industrial ecosystem of intelligent vehicles in a certain region reach?

[0107] The reference document fragments recalled by the document retrieval node include the following:

[0108] Slice 1: In the first half of 2020, the number of motor vehicles reached 360 million, of which 270 million were cars.

[0109] It is expected that by 2025, a certain region will have basically established a system for technological innovation, industrial ecosystem, infrastructure, regulatory standards, product supervision, and cybersecurity for standard intelligent vehicles.

[0110] Segment 2: By then, PA (Partial Automated Driving) and CA (Conditional Automated Driving) level intelligent vehicles will account for more than 50% of total vehicle sales in a certain region. The installation rate of C-V2X (Cellular Vehicle-to-Everything) terminals in new vehicles will reach 50%.

[0111] Segment 3: Intelligent vehicles will initially achieve commercial application in specific scenarios and priority areas, and will continue to expand their operational scope. In a narrow sense, auto finance can be defined as the financial services provided to both buyers and sellers during the car sales process; in a broader sense, auto finance is the combination of the automotive industry and the financial industry…

[0112] The multi-granularity keywords corresponding to this reference question are as follows:

[0113] Coarse-grained keyword: intelligent vehicle.

[0114] Fine-grained keywords:

[0115] [It is projected that by 2025, intelligent vehicles, encompassing technological innovation, industrial ecosystem, infrastructure, regulatory standards, product supervision, cybersecurity systems, and regional standards, will have been largely established.]

[0116] [Sales of PA (partial automation) and CA (conditional automation) level intelligent vehicles accounted for more than 50% of total car sales that year];

[0117] [C-V2X (Cellular Vehicle-to-Everything) technology will be installed in 50% of new vehicles];

[0118] Intelligent vehicles will first achieve commercial applications, and their operational range will be continuously expanded in specific scenarios and limited areas.

[0119] The suggested answer to this question is: By 2025, the technological innovation and industrial ecosystem of intelligent vehicles in a certain region will have basically taken shape…

[0120] In summary, the reference question is processed using the document retrieval node, document ranking node, and answer generation node of the language generation model. The document retrieval node retrieves the reference document fragments corresponding to the reference question, the document ranking node ranks the reference document fragments, and the answer generation node generates the predicted answer to the reference question. This facilitates the subsequent evaluation of the document retrieval node, document ranking node, and answer generation node.

[0121] Further, determining the combined document fragment from the plurality of reference document fragments based on the reference sorting result includes:

[0122] Based on the reference sorting result, a plurality of intermediate document fragments are determined from the plurality of reference document fragments, wherein the number of intermediate document fragments is less than the number of reference document fragments;

[0123] Based on the first keyword, the plurality of intermediate document fragments are filtered to obtain a plurality of target document fragments, wherein the number of target document fragments is less than the number of intermediate document fragments;

[0124] The multiple target document fragments are combined to obtain a combined document fragment.

[0125] Specifically, based on the ranking results, several top-ranking intermediate document fragments can be identified from multiple reference document fragments (for example, the top n reference document fragments can be used as intermediate document fragments). Then, based on coarse-grained keywords, these intermediate document fragments are filtered, removing those that do not contain the coarse-grained keywords and retaining those that do as target document fragments. These target document fragments are then combined to obtain a complete combined document fragment. For example, if the coarse-grained keyword is "intelligent car," then based on this keyword, intermediate document fragments that do not contain the phrase "intelligent car" can be filtered, and those containing the phrase "intelligent car" can be retained as target document fragments.

[0126] Following the previous example, the 100 reference document fragments are sorted from high to low text similarity to obtain a reference ranking result. The top 50 reference document fragments can be selected as intermediate document fragments. These 50 intermediate document fragments are then filtered using coarse-grained keywords to obtain 20 intermediate document fragments containing the coarse-grained keywords as target document fragments. These 20 target document fragments are then combined to obtain a combined document fragment.

[0127] In summary, by using coarse-grained keywords as a relaxed constraint, intermediate document fragments that do not contain these keywords are filtered out. This ensures that only document fragments relevant to the context of the reference question are retained, thus initially identifying potentially high-quality document fragments and reducing the interference of irrelevant content on the subsequent evaluation process.

[0128] In addition, the target processing node also includes document slice nodes;

[0129] The document retrieval node utilizing the language generation model retrieves multiple reference document fragments corresponding to the reference question to be processed, including:

[0130] Using the document retrieval node of the language generation model, retrieve multiple reference text documents corresponding to the reference question to be processed;

[0131] Using the document slice nodes of the language generation model, the multiple reference text documents are sliced ​​according to a preset slice size to obtain multiple reference document fragments.

[0132] The preset slice size can be understood as the pre-set size for slicing a reference text document into reference document fragments. For example, a reference text document can be sliced ​​into reference document fragments of size 256, or it can be sliced ​​into reference document fragments of size 512.

[0133] Specifically, a slice size set can be set, which includes multiple slice sizes, such as 128, 256, and 512. Then, the document retrieval node can retrieve multiple reference text documents corresponding to the reference question to be processed from the network platform or knowledge base. Using the document slice node, the multiple slice sizes included in the slice size set are used as preset slice sizes in sequence. The multiple reference text documents are sliced ​​according to the preset slice sizes to obtain multiple reference document fragments of the preset slice sizes.

[0134] In summary, by slicing multiple reference text documents according to a preset slice size, it is convenient to evaluate the performance of the language generation model at different slice sizes when evaluating the language generation model. This enables the evaluation of the slice performance of the document slice nodes of the language generation model, and allows us to understand at which preset slice size the language generation model performs best.

[0135] Step 206: Evaluate the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, and obtain the evaluation result of the target processing node.

[0136] In specific implementation, the evaluation of the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation result of the target processing node, includes:

[0137] Based on the second keyword group corresponding to each query data and the combined document fragment corresponding to each reference question, the document retrieval node is evaluated to obtain the retrieval evaluation result of the document retrieval node;

[0138] Based on the first keyword and second keyword group corresponding to each query data, the reference sorting result, and the multiple reference document fragments, the document sorting node is evaluated to obtain the sorting evaluation result of the document sorting node;

[0139] Based on the reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data, the answer generation node is evaluated to obtain the generation evaluation result of the answer generation node.

[0140] Specifically, when evaluating the target processing nodes of a language generation model, the document retrieval node, document ranking node, and answer generation node can be evaluated separately. In practice, when evaluating the document retrieval node, the evaluation can be based on the second keyword groups corresponding to each query data point, the combined document fragments corresponding to each reference question, and so on, to obtain the retrieval evaluation result. When evaluating the document ranking node, the evaluation can be based on the first and second keyword groups corresponding to each query data point, the reference ranking results corresponding to each query data point, and multiple reference document fragments, to obtain the ranking evaluation result. When evaluating the answer generation node, the evaluation can be based on the reference questions and answers in each query data point, the predicted answers corresponding to the reference questions in each query data point, and the combined document fragments corresponding to the reference questions in each query data point, to obtain the generation evaluation result.

[0141] In summary, by evaluating the document retrieval node, document sorting node, and answer generation node separately, we can quickly identify the nodes that need optimization and perform targeted optimizations. This enables a full-link evaluation of the language generation model, improves the optimization efficiency of the language generation model, and further ensures the processing performance of the language generation model.

[0142] Further, the step of evaluating the document retrieval node based on the second keyword group corresponding to each query data and the combined document fragment corresponding to each reference question, and obtaining the retrieval evaluation result of the document retrieval node, includes:

[0143] Match the second keyword group corresponding to each query data with the combined document fragment corresponding to each reference question to obtain the keyword matching result corresponding to each query data;

[0144] Based on the keyword matching results corresponding to each query data, the retrieval evaluation index of the document retrieval node is calculated, and the retrieval evaluation index is determined as the retrieval evaluation result of the document retrieval node.

[0145] Keyword matching results can be understood as whether the combined document fragment includes all the second keyword groups corresponding to the query data, and the number of such second keyword groups. For example, if query data 1 corresponds to 5 second keyword groups, the keyword matching result for query data 1 could be that the combined document fragment corresponding to the reference question in query data 1 includes all 5 second keyword groups, or it could be that the combined document fragment corresponding to the reference question in query data 1 includes 3 second keyword groups. Retrieval evaluation metrics can include recall and precision for document retrieval nodes. Recall measures how many fine-grained keyword groups are correctly retrieved from all labeled fine-grained keyword groups in the entire dataset, reflecting the coverage capability of the document retrieval node. Precision evaluates the proportion of completely accurate retrieval results across all fine-grained keyword groups. When all fine-grained keyword groups corresponding to a query data are correctly retrieved, the result is considered completely accurate.

[0146] Specifically, the second keyword group corresponding to each query data and the combined document fragments corresponding to the reference questions in each query data can be matched to obtain the keyword matching results corresponding to each query data. Based on the keyword matching results corresponding to each query data, the recall rate and precision of the document retrieval node can be calculated, and the recall rate and precision can be determined as the retrieval evaluation results of the document retrieval node.

[0147] For example, when calculating the recall rate of a document retrieval node, let's take 100 second keyword groups corresponding to multiple query data as an example. Match the combined document fragments corresponding to the reference questions in each query data with the second keyword groups corresponding to each query data. For multiple query data, if 50 second keyword groups are correctly recalled, then the recall rate is 50 / 100, or 50%.

[0148] Furthermore, for ease of understanding, let's take three query data sets as an example. Query data 1 corresponds to 5 second keyword groups, query data 2 corresponds to 5 second keyword groups, and query data 3 corresponds to 5 second keyword groups. Therefore, the total number of second keyword groups corresponding to these three query data sets is 15. Based on the combined document fragment 1 corresponding to the reference question in query data 1 and the 5 second keyword groups corresponding to query data 1, keyword matching result 1 is that combined document fragment 1 includes all 5 second keyword groups. That is, for query data 1, all 5 second keyword groups were correctly retrieved. Similarly, based on the combined document fragment 2 corresponding to the reference question in query data 2 and the 5 second keyword groups corresponding to query data 2, keyword matching result 2 is that combined document fragment 2 includes 3 second keyword groups. That is, for query data... For query 2, 3 second keyword groups were correctly recalled, but not all of them were recalled. Similarly, based on the combined document fragment 3 corresponding to the reference question in query data 3 and the 5 second keyword groups corresponding to query data 3, keyword matching was performed. The keyword matching result 3 obtained was that the combined document fragment 3 included 2 second keyword groups. That is to say, for query data 3, 2 second keyword groups were correctly recalled, but not all of them were recalled. Therefore, for these 3 query data, out of all 15 second keyword groups, 5 + 3 + 2 = 10 second keyword groups were correctly recalled, so the recall rate is 10 / 15 = 2 / 3.

[0149] Using the previous example, when calculating the accuracy of document retrieval nodes, taking 100 query data as an example, if the second keyword group corresponding to 20 of the query data is correctly recalled, it means that the results of these 20 query data are completely correct, then the accuracy is 20 / 100, which equals 20%.

[0150] Furthermore, for ease of understanding, let's take three query data sets as an example. Query data 1 corresponds to 5 second keyword groups, query data 2 corresponds to 5 second keyword groups, and query data 3 corresponds to 5 second keyword groups. Therefore, the total number of second keyword groups corresponding to these three query data sets is 15. Based on the combined document fragment 1 corresponding to the reference question in query data 1 and the 5 second keyword groups corresponding to query data 1, keyword matching result 1 shows that combined document fragment 1 includes all 5 second keyword groups. That is, for query data 1, all 5 second keyword groups were correctly retrieved. Similarly, based on the combined document fragment 2 corresponding to the reference question in query data 2 and the 5 second keyword groups corresponding to query data 2, keyword matching result 2 shows that combined document fragment 2 includes 3 second keyword groups. In other words, for query data 2, 3 second keyword groups were correctly retrieved, but not all of them were retrieved. Similarly, based on the combined document fragment 3 corresponding to the reference question in query data 3 and the 5 second keyword groups corresponding to query data 3, keyword matching was performed. The obtained keyword matching result 3 is that the combined document fragment 3 includes 2 second keyword groups. That is to say, for query data 3, 2 second keyword groups were correctly retrieved, but not all of them were retrieved. Therefore, for these 3 query data, only when all the second keyword groups corresponding to query data 1 are correctly retrieved will the accuracy rate be 1 / 3.

[0151] In summary, by calculating retrieval evaluation metrics for document retrieval nodes based on keyword matching results, the retrieval phase of the language generation model can be evaluated. A comprehensive evaluation of recall and precision provides a more complete understanding of the retrieval phase's performance at different granularities, thus offering a reliable basis for further optimization. This multi-level and multi-dimensional evaluation improves the accuracy and reliability of the assessment and provides a clearer direction for improving and adjusting the language generation model.

[0152] Further, the step of evaluating the document ranking node based on the first keyword and second keyword group corresponding to each query data, the reference ranking result, and the multiple reference document fragments to obtain the ranking evaluation result of the document ranking node includes:

[0153] Based on the reference sorting result, a plurality of evaluation document fragments are determined from the plurality of reference document fragments, wherein the number of evaluation document fragments is less than the number of reference document fragments;

[0154] Based on the first keyword, the plurality of evaluation document fragments are filtered to obtain the plurality of filtered evaluation document fragments;

[0155] The filtered multiple evaluation document fragments are combined to obtain a combined evaluation document fragment;

[0156] The document ranking node is evaluated based on the second keyword group corresponding to each query data and the combined evaluation document fragment corresponding to each reference question, and the ranking evaluation result of the document ranking node is obtained.

[0157] The evaluation document fragments can be understood as the document fragments selected for evaluating the document ranking node. It's understood that the number of evaluation document fragments is less than the number of intermediate document fragments mentioned earlier. It's understood that since the document ranking node needs to sort multiple reference document fragments retrieved by the document retrieval node and select the top-ranked reference document fragment, in order to evaluate the document ranking node and reduce dependence on the document retrieval node, the top k reference document fragments can be selected as evaluation document fragments, where k is less than n. For example, k is 10 and n is 100. That is, when evaluating the document retrieval node, the top 100 reference document fragments can be selected as intermediate document fragments, and when evaluating the document ranking node, the top 10 reference document fragments can be selected as evaluation document fragments.

[0158] Specifically, based on the reference ranking results, the top k document fragments from multiple reference document fragments can be selected as evaluation document fragments. Then, based on coarse-grained keywords, these evaluation document fragments are filtered, removing those that do not contain the coarse-grained keywords and retaining only those containing them. These filtered evaluation document fragments are then combined to obtain a combined evaluation document fragment. Similarly, combined evaluation document fragments corresponding to the reference questions in each query can be obtained. Following the same process, the document ranking nodes can be evaluated based on the second keyword group corresponding to each query and the combined evaluation document fragments corresponding to the reference questions in each query, thus obtaining the ranking evaluation results for the document ranking nodes.

[0159] Understandably, when evaluating document ranking nodes, similar to the evaluation of document retrieval nodes, the recall and precision of document ranking nodes are calculated, and the recall and precision are used as the ranking evaluation results. The specific calculation process is similar to that described above, and will not be repeated here.

[0160] In summary, by constructing combined evaluation document fragments and calculating the ranking evaluation index of document ranking nodes based on keyword matching results, the ranking stage of the language generation model can be evaluated.

[0161] Further, the step of evaluating the answer generation node based on the reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data, to obtain the generation evaluation result of the answer generation node, includes:

[0162] The reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data are input into the evaluator;

[0163] Using the evaluator, the answer generation node is evaluated based on multiple answer generation metrics to obtain the generation evaluation result of the answer generation node.

[0164] The answer generation metrics may include, but are not limited to, BLEU, Rouge-L, fidelity, relevance, and correctness.

[0165] In practical applications, BLEU measures similarity by calculating an exact n-gram match between the generated response (i.e., the predicted answer) and the reference answer. Rouge-L, on the other hand, measures similarity between the generated response and the reference answer using the Longest Common Subsequence (LCS), focusing on order and coverage. Fidelity, relevance, and correctness can be calculated using LlamaIndex's built-in evaluator, which automatically evaluates based on context learning using a large model. Fidelity assesses whether the generated response is faithful to the retrieved context. Relevance assesses the relevance of the retrieved context and the generated response to the query. Correctness assesses the system's correctness. This evaluator outputs a score between 1 and 5 based on the query, generated response, and reference answer, providing the rationale for the score. The score represents the average correctness score across all samples. The pass rate is defined as the proportion of query data with a score greater than or equal to 4.

[0166] In summary, by using multiple answer generation metrics, a comprehensive evaluation of the answer generation nodes can be achieved.

[0167] In addition, document slice nodes in the language generation model can be evaluated, as follows:

[0168] Based on the retrieval evaluation results, the sorting evaluation results, and the generation evaluation results, the slice evaluation result of the document slice node under the preset slice size is determined.

[0169] Specifically, the retrieval evaluation results, ranking evaluation results, and generated evaluation results can be used as the slice evaluation results of the document slice node under the preset slice size. Alternatively, the retrieval evaluation results, ranking evaluation results, and generated evaluation results can be weighted and summed, or simply summed, to obtain the slice evaluation results of the document slice node under the preset slice size.

[0170] For example, the preset slice size can be set to 128 to evaluate the performance of the document retrieval node, document sorting node, and answer generation node when the preset slice size is 128. Alternatively, the preset slice size can be set to 256 to evaluate the performance of the document retrieval node, document sorting node, and answer generation node when the preset slice size is 256.

[0171] In practical applications, the evaluation of the retrieval stage involves first using a vector model to retrieve relevant slices from the document database based on semantic similarity to the query, and then scoring them using a keyword strategy. The evaluation of the ranking stage involves first using a vector model to rank the retrieved slices, and then selecting the top K slices and scoring them using a keyword strategy. The evaluation of the slice stage involves dividing the reference text documents into different slice sizes, and then evaluating the retrieval, ranking, and generation performance separately.

[0172] In summary, the performance of document slice nodes is evaluated by assessing their performance under different slice sizes, thus achieving the goal of evaluating document slice nodes.

[0173] In practical applications, after obtaining the evaluation result of the target processing node, the process further includes:

[0174] Based on the evaluation results, the node parameters of the target processing node are adjusted to obtain an updated language generation model.

[0175] Specifically, the node parameters of the document retrieval node can be adjusted based on the retrieval evaluation results, and / or the node parameters of the document sorting node can be adjusted based on the sorting evaluation results, and / or the node parameters of the answer generation node can be adjusted based on the generation evaluation results, and / or the node parameters of the document slicing node can be adjusted based on the slicing evaluation results, to obtain an updated language generation model.

[0176] In summary, the language generation model can be optimized by adjusting the node parameters of the target processing node.

[0177] In summary, the above method constructs a document dataset and generates multiple query data based on the text documents in the dataset. It then generates first and second keyword groups for each query data item. Utilizing the target processing node of the language generation model, it generates predicted answers for each reference question in each query data item. Based on the first and second keyword groups for each query data item, the reference questions and answers, and the predicted answers for each reference question, the target processing node is evaluated, yielding an evaluation result. This evaluation of the target processing node of the language generation model is achieved. The first and second keyword groups enhance the evaluation effect of the language generation model, enabling the identification of problematic nodes within the model. This facilitates subsequent optimization of the language processing model based on the evaluation results, ensuring the processing capability of the language generation model and ultimately guaranteeing the quality of the answers generated.

[0178] The following description, in conjunction with Figures 3 and 4, uses the application of the target processing node evaluation method for language generation models provided in this specification in the performance evaluation of language generation models as an example to further illustrate the target processing node evaluation method for language generation models. Figure 3 shows a flowchart of the data preparation process for a target processing node evaluation method for language generation models according to an embodiment of this specification, specifically including the following steps.

[0179] Step 302: Collect text documents of various formats and industries from multiple open source websites, where the text documents have a relatively recent creation date.

[0180] Step 304: Parse and slice the semi-structured text document, or slice the structured text document directly to obtain multiple document fragments corresponding to each text document.

[0181] Step 306: Using the large model, generate four types of query data for each document fragment.

[0182] Step 308: Manually evaluate the acceptability of each query data to determine the final usable query data.

[0183] Step 310: Construct a query dataset based on all the query data above.

[0184] Step 312: Using the large model, generate a coarse-grained keyword and at least one fine-grained keyword group for each query.

[0185] Figure 4 shows a flowchart of the evaluation process of a target processing node evaluation method for a language generation model provided in one embodiment of this specification, which specifically includes the following steps.

[0186] Step 402: Take any query data in the query dataset as the target query data, input the reference question in the target query data into the language generation model, and in the language generation model, recall multiple corresponding reference text documents for the reference question.

[0187] Step 404: Slice the multiple reference text documents according to the slice size set to obtain multiple reference document fragments corresponding to each slice size. The slice size set includes multiple slice sizes.

[0188] Step 406: Select one of the slice sizes as the preset slice size, and determine the multiple reference document fragments corresponding to the preset slice size.

[0189] Step 408: Sort the multiple reference document fragments according to their text similarity and select the top n reference document fragments.

[0190] Step 410: Based on the coarse-grained keywords of the target query data, filter the first n reference document fragments, remove the reference document fragments that do not contain the coarse-grained keywords of the target query data, and obtain the remaining intermediate document fragments.

[0191] Step 412: Combine the remaining intermediate document fragments to obtain a combined document fragment.

[0192] Step 414: Match the combined document fragments with the fine-grained keyword groups of the target query data to obtain keyword matching results.

[0193] Step 416: Obtain the keyword matching results of all query data in the query dataset, and calculate the recall and precision of the document retrieval node of the language generation model based on the keyword matching results of all query data.

[0194] Step 418: Combining with step 408, select the top K reference document fragments and execute steps 410-416 to calculate the recall and precision of the document ranking node of the language generation model.

[0195] Where k is less than n, for example, n is 100 and k is 10.

[0196] Step 420: Based on the combined document fragments from Step 412, obtain the predicted answer to the reference question.

[0197] Step 422: Input the reference question, the combined document fragment from Step 412, and the predicted answer into the evaluator to obtain the generation evaluation result of the answer generation node.

[0198] Step 424: Use the recall and precision of document retrieval nodes of each slice size, the recall and precision of document ranking nodes, and the generation evaluation results of answer generation nodes as the slice evaluation results of document slice nodes.

[0199] In summary, the above method constructs a document dataset and generates multiple query data based on the text documents in the dataset. It then generates the first keyword and second keyword group for each query data. Utilizing the target processing node of the language generation model, it generates predicted answers for each reference question in each query data. Based on the first and second keyword groups for each query data, the reference questions and answers in each query data, and the predicted answers for each reference question, the target processing node is evaluated to obtain the evaluation results. This achieves a full-link evaluation of the language generation model, enabling the location of problematic nodes within the model. This facilitates subsequent optimization of the language processing model based on the evaluation results, ensuring the full-link processing capability of the language generation model and thus guaranteeing the quality of the answers generated by the model.

[0200] Corresponding to the above method embodiments, this specification also provides an embodiment of a target processing node evaluation device for a language generation model. Figure 5 shows a schematic diagram of the structure of a target processing node evaluation device for a language generation model provided in one embodiment of this specification. As shown in Figure 5, the device includes:

[0201] The first generation module 502 is configured to parse text documents in the document dataset, generate multiple query data, and generate a first keyword and a second keyword group corresponding to each query data in the multiple query data, wherein each query data includes a reference question and a reference answer corresponding to the reference question;

[0202] The second generation module 504 is configured to use the target processing node of the language generation model to generate the predicted answer corresponding to each reference question in each query data.

[0203] The evaluation module 506 is configured to evaluate the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference question and the reference answer in each query data, and the predicted answer corresponding to each reference question, and obtain the evaluation result of the target processing node.

[0204] In an optional embodiment, the first generation module 502 is further configured to:

[0205] The text documents in the document dataset are sliced ​​to obtain multiple document fragments;

[0206] The multiple document fragments are sequentially identified as document fragments to be processed, and multiple query data corresponding to the document fragments to be processed are generated, wherein the query types of the multiple query data are different.

[0207] In an optional embodiment, the first generation module 502 is further configured to:

[0208] Identify the document fragment to be processed corresponding to each of the multiple query data;

[0209] Information is extracted from the document fragments to be processed to determine the first keyword corresponding to each query data;

[0210] Based on the reference questions in each query data, multiple information points corresponding to the reference questions are determined from the document fragments to be processed, and based on the multiple information points, the second keyword group corresponding to each query data is determined.

[0211] In one optional embodiment, the language generation model includes multiple target processing nodes;

[0212] The second generation module 504 is further configured to include:

[0213] By utilizing multiple target processing nodes of the language generation model, predicted answers are generated for each reference question in each query data.

[0214] The evaluation module 506 is further configured as follows:

[0215] The evaluation of the multiple target processing nodes is performed based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation results of the multiple target processing nodes.

[0216] In one optional embodiment, the target processing node includes a document retrieval node, a document sorting node, and an answer generation node;

[0217] The second generation module 504 is further configured as follows:

[0218] Each reference question in the query data is sequentially identified as a reference question to be processed.

[0219] Using the document retrieval node of the language generation model, retrieve multiple reference document fragments corresponding to the reference question to be processed;

[0220] Using the document sorting node of the language generation model, the multiple reference document fragments are sorted to obtain a reference sorting result;

[0221] Using the answer generation node of the language generation model, a combined document fragment is determined from the multiple reference document fragments according to the reference sorting result, and a predicted answer corresponding to the reference question to be processed is generated according to the combined document fragment, thereby obtaining the predicted answer corresponding to each reference question in each query data.

[0222] In an optional embodiment, the second generation module 504 is further configured to:

[0223] Based on the reference sorting result, a plurality of intermediate document fragments are determined from the plurality of reference document fragments, wherein the number of intermediate document fragments is less than the number of reference document fragments;

[0224] Based on the first keyword, the plurality of intermediate document fragments are filtered to obtain a plurality of target document fragments, wherein the number of target document fragments is less than the number of intermediate document fragments;

[0225] The multiple target document fragments are combined to obtain a combined document fragment.

[0226] In an optional embodiment, the target processing node further includes a document slice node;

[0227] The second generation module 504 is further configured as follows:

[0228] Using the document retrieval node of the language generation model, retrieve multiple reference text documents corresponding to the reference question to be processed;

[0229] Using the document slice nodes of the language generation model, the multiple reference text documents are sliced ​​according to a preset slice size to obtain multiple reference document fragments.

[0230] In an optional embodiment, the evaluation module 506 is further configured to:

[0231] Based on the second keyword group corresponding to each query data and the combined document fragment corresponding to each reference question, the document retrieval node is evaluated to obtain the retrieval evaluation result of the document retrieval node;

[0232] Based on the first keyword and second keyword group corresponding to each query data, the reference sorting result, and the multiple reference document fragments, the document sorting node is evaluated to obtain the sorting evaluation result of the document sorting node;

[0233] Based on the reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data, the answer generation node is evaluated to obtain the generation evaluation result of the answer generation node.

[0234] In an optional embodiment, the target processing node further includes a document slice node;

[0235] The evaluation module 506 is further configured as follows:

[0236] Based on the retrieval evaluation results, the sorting evaluation results, and the generation evaluation results, the slice evaluation result of the document slice node under the preset slice size is determined.

[0237] In an optional embodiment, the evaluation module 506 is further configured to:

[0238] Match the second keyword group corresponding to each query data with the combined document fragment corresponding to each reference question to obtain the keyword matching result corresponding to each query data;

[0239] Based on the keyword matching results corresponding to each query data, the retrieval evaluation index of the document retrieval node is calculated, and the retrieval evaluation index is determined as the retrieval evaluation result of the document retrieval node.

[0240] In an optional embodiment, the evaluation module 506 is further configured to:

[0241] Based on the reference sorting result, a plurality of evaluation document fragments are determined from the plurality of reference document fragments, wherein the number of evaluation document fragments is less than the number of reference document fragments;

[0242] Based on the first keyword, the plurality of evaluation document fragments are filtered to obtain the plurality of filtered evaluation document fragments;

[0243] The filtered multiple evaluation document fragments are combined to obtain a combined evaluation document fragment;

[0244] The document ranking node is evaluated based on the second keyword group corresponding to each query data and the combined evaluation document fragment corresponding to each reference question, and the ranking evaluation result of the document ranking node is obtained.

[0245] In an optional embodiment, the evaluation module 506 is further configured to:

[0246] The reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data are input into the evaluator;

[0247] Using the evaluator, the answer generation node is evaluated based on multiple answer generation metrics to obtain the generation evaluation result of the answer generation node.

[0248] In an optional embodiment, the evaluation module 506 is further configured to:

[0249] Based on the evaluation results, the node parameters of the target processing node are adjusted to obtain an updated language generation model.

[0250] In summary, the aforementioned device constructs a document dataset and generates multiple query data based on the text documents within the dataset. It then generates first and second keyword groups for each query data item. Utilizing the target processing node of the language generation model, it generates predicted answers for each reference question within each query data item. Based on the first and second keyword groups for each query data item, the reference questions and answers within each query data item, and the predicted answers for each reference question, the target processing node is evaluated to obtain its evaluation results. This achieves end-to-end evaluation of the language generation model, enabling the identification of problematic nodes within the model. This facilitates subsequent optimization of the language processing model based on the evaluation results, thereby ensuring the end-to-end processing capability of the language generation model and ultimately guaranteeing the quality of the answers generated by the model.

[0251] The above is a schematic scheme of a target processing node evaluation device for a language generation model according to this embodiment. It should be noted that the technical solution of this target processing node evaluation device for a language generation model belongs to the same concept as the technical solution of the target processing node evaluation method for a language generation model described above. For details not described in detail in the technical solution of the target processing node evaluation device for a language generation model, please refer to the description of the technical solution of the target processing node evaluation method for a language generation model described above.

[0252] Referring to Figure 6, Figure 6 shows a flowchart of an answer generation method provided according to an embodiment of this specification, which specifically includes the following steps.

[0253] Step 602: Define the target problem;

[0254] Step 604: Utilize the target processing node of the language generation model to generate the target answer corresponding to the target question;

[0255] The language generation model is a language generation model obtained by adjusting the target processing node evaluation method of the language generation model provided in the first aspect of the embodiments of this specification.

[0256] Specifically, this answer generation method can be applied to the server side. Users can input a target question through the client, which then sends the target question to the server. The server receives the target question, uses the target processing node of the language generation model to perform retrieval and generation processing on the target question, obtains the target answer corresponding to the target question, and sends the target answer back to the client, which then displays it to the user through the client's display interface.

[0257] Alternatively, the answer generation method can also be directly applied to the client. The client can call the language generation model through an interface call, or the language generation model can be directly deployed on the client. This specification does not limit this embodiment.

[0258] In summary, the above method constructs a document dataset and generates multiple query data based on the text documents in the dataset. It then generates the first keyword and second keyword group for each query data. Utilizing the target processing node of the language generation model, it generates predicted answers for each reference question in each query data. Based on the first and second keyword groups for each query data, the reference questions and answers in each query data, and the predicted answers for each reference question, the target processing node is evaluated to obtain the evaluation results. This achieves a full-link evaluation of the language generation model, enabling the location of problematic nodes within the model. This facilitates subsequent optimization of the language processing model based on the evaluation results, ensuring the full-link processing capability of the language generation model and thus guaranteeing the quality of the answers generated by the model.

[0259] Corresponding to the above method embodiments, this specification also provides an answer generation device embodiment. Figure 7 shows a schematic diagram of the structure of an answer generation device provided in one embodiment of this specification. As shown in Figure 7, the device includes:

[0260] Module 702 is configured to determine the target problem;

[0261] The generation module 704 is configured as a target processing node using a language generation model to generate the target answer corresponding to the target question.

[0262] The language generation model is a language generation model obtained by adjusting the target processing node evaluation method of the language generation model provided in the first aspect of the embodiments of this specification.

[0263] In summary, the aforementioned device constructs a document dataset and generates multiple query data based on the text documents within the dataset. It then generates first and second keyword groups for each query data item. Utilizing the target processing node of the language generation model, it generates predicted answers for each reference question within each query data item. Based on the first and second keyword groups for each query data item, the reference questions and answers within each query data item, and the predicted answers for each reference question, the target processing node is evaluated to obtain its evaluation results. This achieves end-to-end evaluation of the language generation model, enabling the identification of problematic nodes within the model. This facilitates subsequent optimization of the language processing model based on the evaluation results, thereby ensuring the end-to-end processing capability of the language generation model and ultimately guaranteeing the quality of the answers generated by the model.

[0264] The above is a schematic scheme of an answer generation device according to this embodiment. It should be noted that the technical solution of this answer generation device and the technical solution of the answer generation method described above belong to the same concept. For details not described in detail in the technical solution of the answer generation device, please refer to the description of the technical solution of the answer generation method described above.

[0265] Figure 8 shows a structural block diagram of a computing device 800 according to one embodiment of this specification. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0266] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0267] In one embodiment of this disclosure, the aforementioned components of the computing device 800, as well as other components not shown in FIG. 8, may also be connected to each other, for example, via a bus. It should be understood that the computing device structural block diagram shown in FIG. 8 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.

[0268] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0269] The processor 820 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above method.

[0270] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0271] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0272] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0273] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0274] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of the above method.

[0275] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0276] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0277] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0278] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0279] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. Evaluation methods for target processing nodes in language generation models, including: The text documents in the document dataset are parsed to generate multiple query data, and a first keyword and a second keyword group corresponding to each query data are generated. Each query data includes a reference question and a reference answer corresponding to the reference question. The target processing node of the language generation model is used to generate the predicted answer for each reference question in each query data. The target processing node is evaluated based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation result of the target processing node.

2. The method according to claim 1, wherein parsing text documents in the document dataset to generate multiple query data includes: The text documents in the document dataset are sliced ​​to obtain multiple document fragments; The multiple document fragments are sequentially identified as document fragments to be processed, and multiple query data corresponding to the document fragments to be processed are generated, wherein the query types of the multiple query data are different.

3. The method according to claim 2, wherein generating the first keyword and the second keyword group corresponding to each of the plurality of query data includes: Identify the document fragment to be processed corresponding to each of the multiple query data; Information is extracted from the document fragments to be processed to determine the first keyword corresponding to each query data; Based on the reference questions in each query data, multiple information points corresponding to the reference questions are determined from the document fragments to be processed, and based on the multiple information points, the second keyword group corresponding to each query data is determined.

4. The method according to any one of claims 1-3, wherein the language generation model comprises a plurality of target processing nodes; The target processing node utilizing the language generation model generates predicted answers for each reference question in each query data, including: Using multiple target processing nodes of the language generation model, predictive answers are generated for each reference question in each query data. The step of evaluating the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation result of the target processing node, includes: The evaluation of the multiple target processing nodes is performed based on the first keyword and second keyword group corresponding to each query data, the reference questions and reference answers in each query data, and the predicted answers corresponding to each reference question, to obtain the evaluation results of the multiple target processing nodes.

5. The method according to any one of claims 1-4, wherein the target processing node includes a document retrieval node, a document sorting node, and an answer generation node; The target processing node utilizing the language generation model generates predicted answers for each reference question in each query data, including: Each reference question in the query data is sequentially identified as a reference question to be processed. Using the document retrieval node of the language generation model, retrieve multiple reference document fragments corresponding to the reference question to be processed; Using the document sorting node of the language generation model, the multiple reference document fragments are sorted to obtain a reference sorting result; Using the answer generation node of the language generation model, a combined document fragment is determined from the multiple reference document fragments according to the reference sorting result, and a predicted answer corresponding to the reference question to be processed is generated according to the combined document fragment, thereby obtaining the predicted answer corresponding to each reference question in each query data.

6. The method according to claim 5, wherein determining the combined document fragment from the plurality of reference document fragments based on the reference sorting result comprises: Based on the reference sorting result, a plurality of intermediate document fragments are determined from the plurality of reference document fragments, wherein the number of intermediate document fragments is less than the number of reference document fragments; Based on the first keyword, the plurality of intermediate document fragments are filtered to obtain a plurality of target document fragments, wherein the number of target document fragments is less than the number of intermediate document fragments; The multiple target document fragments are combined to obtain a combined document fragment.

7. The method according to claim 5 or 6, wherein the target processing node further comprises a document slice node; The document retrieval node utilizing the language generation model retrieves multiple reference document fragments corresponding to the reference question to be processed, including: Using the document retrieval node of the language generation model, retrieve multiple reference text documents corresponding to the reference question to be processed; Using the document slice nodes of the language generation model, the multiple reference text documents are sliced ​​according to a preset slice size to obtain multiple reference document fragments.

8. The method according to claim 7, wherein evaluating the target processing node based on the first keyword and second keyword group corresponding to each query data, the reference question and the reference answer in each query data, and the predicted answer corresponding to each reference question, to obtain the evaluation result of the target processing node, includes: Based on the second keyword group corresponding to each query data and the combined document fragment corresponding to each reference question, the document retrieval node is evaluated to obtain the retrieval evaluation result of the document retrieval node; Based on the first keyword and second keyword group corresponding to each query data, the reference sorting result, and the multiple reference document fragments, the document sorting node is evaluated to obtain the sorting evaluation result of the document sorting node; Based on the reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data, the answer generation node is evaluated to obtain the generation evaluation result of the answer generation node.

9. The method according to claim 8, wherein the target processing node further comprises a document slice node; The method further includes: Based on the retrieval evaluation results, the sorting evaluation results, and the generation evaluation results, the slice evaluation result of the document slice node under the preset slice size is determined.

10. The method according to claim 8 or 9, wherein evaluating the document retrieval node based on the second keyword group corresponding to each query data and the combined document fragment corresponding to each reference question, and obtaining the retrieval evaluation result of the document retrieval node, includes: Match the second keyword group corresponding to each query data with the combined document fragment corresponding to each reference question to obtain the keyword matching result corresponding to each query data; Based on the keyword matching results corresponding to each query data, the retrieval evaluation index of the document retrieval node is calculated, and the retrieval evaluation index is determined as the retrieval evaluation result of the document retrieval node.

11. The method according to any one of claims 8-10, wherein evaluating the document ranking node based on the first keyword and second keyword group corresponding to each query data, the reference ranking result, and the plurality of reference document fragments to obtain the ranking evaluation result of the document ranking node includes: Based on the reference sorting result, a plurality of evaluation document fragments are determined from the plurality of reference document fragments, wherein the number of evaluation document fragments is less than the number of reference document fragments; Based on the first keyword, the plurality of evaluation document fragments are filtered to obtain the plurality of filtered evaluation document fragments; The filtered multiple evaluation document fragments are combined to obtain a combined evaluation document fragment; The document ranking node is evaluated based on the second keyword group corresponding to each query data and the combined evaluation document fragment corresponding to each reference question, and the ranking evaluation result of the document ranking node is obtained.

12. The method according to any one of claims 8-11, wherein evaluating the answer generation node based on the reference questions and reference answers in the query data, the predicted answers corresponding to the reference questions, and the combined document fragments corresponding to the reference questions, to obtain the generation evaluation result of the answer generation node, includes: The reference questions and reference answers, the predicted answers corresponding to each reference question, and the combined document fragments corresponding to each reference question in the query data are input into the evaluator; Using the evaluator, the answer generation node is evaluated based on multiple answer generation metrics to obtain the generation evaluation result of the answer generation node.

13. The method according to any one of claims 1-4 and 6-12, wherein after obtaining the evaluation result of the target processing node, it further comprises: Based on the evaluation results, the node parameters of the target processing node are adjusted to obtain an updated language generation model.

14. A method for generating answers, comprising: Define the target problem; The target answer corresponding to the target question is generated using the target processing node of the language generation model; The language generation model is the language generation model obtained by adjusting the target processing node evaluation method according to any one of claims 1-13.

15. The method of any one of claims 1-14, wherein, The step of matching the second keyword group corresponding to each query data and the combined document fragment corresponding to each reference question to obtain the keyword matching result corresponding to each query data includes: A semantic similarity-based matching strategy is adopted to compare each fine-grained keyword in the second keyword group with the text fragment in the combined document fragment in a vectorized manner. When the semantic similarity is higher than a preset threshold, it is determined that the fine-grained keyword has been successfully recalled. The semantic similarity is calculated using a pre-trained language model, which includes at least one of BERT, RoBERTa, or Sentence-BERT.

16. The method of claim 15, wherein, After obtaining the retrieval evaluation result of the document retrieval node, the ranking evaluation result of the document ranking node, and the generation evaluation result of the answer generation node, the method further includes: A multi-dimensional comprehensive evaluation score is constructed, which is obtained by weighted fusion of at least four indicators among recall, accuracy, BLEU, Rouge-L, fidelity, relevance and correctness; The weights of each indicator are dynamically configured according to the application scenario, which includes financial Q&A, medical consultation, or technical document retrieval.

17. The method of any one of claims 1-16, wherein, After adjusting the node parameters of the target processing node based on the evaluation results to obtain the updated language generation model, the process further includes: The updated language generation model is used as the new version model, and the evaluation process described in claim 1 is returned to generate a new round of evaluation results; If the evaluation results of the new and old versions of the model are compared, and the evaluation indicators of at least two target processing nodes are improved by more than a preset ratio, then the new version of the model will be deployed as an online service model in the production environment.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

Images