Block evaluation method, device and equipment for retrieval enhancement generation

Abstract

The invention discloses a block evaluation method, device and equipment for retrieval enhancement generation, and relates to the technical field of natural language processing. The method comprises the following steps: matching a to-be-retrieved problem in the power grid field with an initial document fragment in a retrieval corpus of the power grid field by retrieving an enhanced large language model so as to determine a candidate document fragment; according to the to-be-retrieved question, determining at least one target coarse-grained keyword and at least one target fine-grained keyword from the initial coarse-grained keyword set and the initial fine-grained keyword set; screening the candidate document fragments according to the target coarseness keyword to obtain screened target candidate document fragments; and according to the target fine-grained keyword and the target candidate document fragment, determining a block evaluation result of the to-be-retrieved question. According to the scheme, the block evaluation result of the to-be-retrieved question is determined through the target coarse-grained keyword and the target fine-grained keyword, and the accuracy of block evaluation is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, in particular to the field of natural language processing technology, and specifically to a retrieval-augmented generation (RAG) chunking evaluation method, device and equipment. BACKGROUND

[0002] Retrieval-augmented generation (RAG) technology combines retrieval models and generation models, enabling generation models to generate content or question and answer responses with information from an external text library, effectively alleviating the hallucination phenomenon common in pure generation models. However, with the rapid development of RAG methods, how to accurately evaluate their performance has become one of the key issues in current research.

[0003] As the starting step of the RAG pipeline, the chunking strategy directly determines the integrity of document information preservation and the context relevance of subsequent retrieval. However, existing methods lack explicit evaluation of the chunking stage, and often use indirect changes in question and answer quality or retrieval effectiveness to infer chunking quality. Such implicit evaluation methods lack interpretability and are difficult to provide operational optimization basis. SUMMARY

[0004] The present application provides a retrieval-augmented generation chunking evaluation method, device and equipment to improve the accuracy of retrieval-augmented generation chunking evaluation.

[0005] According to an aspect of the present application, a retrieval-augmented generation chunking evaluation method is provided, which comprises:

[0006] By retrieval-augmented large language model, the power grid domain to be retrieved problem is matched with the initial document segment in the retrieval corpus of the power grid domain, and the candidate document segment is determined from the initial document segment according to the matching result; wherein the retrieval corpus is obtained by structuring the to-be-processed document in the power grid domain, and the retrieval corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set;

[0007] According to the to-be-retrieved problem, at least one target coarse-grained keyword and at least one target fine-grained keyword are determined from the initial coarse-grained keyword set and the initial fine-grained keyword set;

[0008] According to the target coarse-grained keyword, the candidate document segment is screened to obtain the target candidate document segment after screening;

[0009] According to the target fine-grained keyword and the target candidate document segment, the chunking evaluation result of the to-be-retrieved problem is determined.

[0010] According to another aspect of this application, a retrieval enhancement generation block evaluation apparatus is provided, the apparatus comprising:

[0011] The candidate document fragment determination module is used to match the search query in the power grid field with the initial document fragments in the power grid field retrieval corpus by retrieval enhancement large language model, and determine candidate document fragments from the initial document fragments based on the matching results; wherein, the retrieval corpus is obtained by structuring the documents to be processed in the power grid field, and the retrieval corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set;

[0012] The target keyword determination module is used to determine at least one target coarse-grained keyword and at least one target fine-grained keyword from the initial coarse-grained keyword set and the initial fine-grained keyword set according to the search question;

[0013] The target candidate document fragment determination module is used to filter the candidate document fragments based on the target coarse-grained keywords to obtain the filtered target candidate document fragments.

[0014] The block evaluation result determination module is used to determine the block evaluation result of the search question based on the target fine-grained keywords and the target candidate document fragments.

[0015] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0016] One or more processors;

[0017] Memory, used to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the retrieval enhancement generation block evaluation methods provided in the embodiments of this application.

[0019] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the retrieval enhancement generation block evaluation methods provided in the embodiments of this application.

[0020] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the retrieval enhancement generation chunking evaluation methods provided in the embodiments of this application.

[0021] This application uses a retrieval-enhanced large language model to match the retrieval question in the power grid domain with initial document fragments in a power grid domain retrieval corpus, thereby determining candidate document fragments. Based on the retrieval question, at least one target coarse-grained keyword and at least one target fine-grained keyword are determined from an initial coarse-grained keyword set and an initial fine-grained keyword set. The candidate document fragments are then filtered based on the target coarse-grained keyword to obtain filtered target candidate document fragments. Finally, the segmentation evaluation result of the retrieval question is determined based on the target fine-grained keyword and the target candidate document fragments. This approach improves the accuracy of segmentation evaluation by determining the segmentation evaluation result of the retrieval question through target coarse-grained and target fine-grained keywords. Attached Figure Description

[0022] Figure 1 This is a flowchart of a retrieval enhancement generation block evaluation method provided in Embodiment 1 of this application;

[0023] Figure 2 This is a flowchart of a retrieval enhancement generation block evaluation method provided in Embodiment 2 of this application;

[0024] Figure 3 This is a schematic diagram of a segmented evaluation device for retrieval enhancement generation according to Embodiment 3 of this application;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the retrieval enhancement generation block evaluation method of Embodiment 4 of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of documents and related data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] Example 1

[0030] Figure 1 This is a flowchart of a segmented evaluation method for retrieval enhancement generation according to Embodiment 1 of this application. This embodiment is applicable to situations where explicit evaluation of the segmentation stage in a retrieval enhancement generation system is performed. It can be executed by a segmented evaluation device for retrieval enhancement generation, which can be implemented in hardware and / or software and can be configured in a computer device, such as a server. Figure 1 As shown, the method includes:

[0031] S110. By retrieving and enhancing the large language model, the search question in the power grid field is matched with the initial document fragments in the power grid field search corpus, and candidate document fragments are determined from the initial document fragments based on the matching results; wherein, the search corpus is obtained by structuring the documents to be processed in the power grid field, and the search corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set.

[0032] The documents to be processed in the power grid field can cover data throughout the entire lifecycle of the power system, ensuring the comprehensiveness and professionalism of the retrieval and generation. These can include power grid technical documents, power grid technical standards and specifications, equipment and operation and maintenance documents, operation and fault data, and planning and research reports, in various formats (DOC, PDF, XLSX, etc.) to ensure data diversity. For example, power grid technical documents may include dispatching procedures and equipment parameters; power grid technical standards and specifications may include international / national standards (such as IEC 61850 substation automation, GB / T 14285 relay protection technical regulations), and enterprise regulations (such as the "Technical Guidelines for Distribution Automation" and the "Regulations on the Management of New Energy Grid Connection"); equipment and operation and maintenance documents may include equipment technical files (transformer factory test reports, including no-load loss, short-circuit impedance parameters, etc.), inspection logs, etc.; operation and fault data may include SCADA / EMS system data, fault analysis reports, etc.; and planning and research reports may include power grid planning schemes and academic papers.

[0033] Structured processing can include data cleaning and standardization, multimodal parsing, entity and relation extraction, and chunking and indexing optimization. For example, data cleaning and standardization can include redundancy handling (such as removing OCR noise from scanned documents, merging duplicate paragraphs, and removing duplicate headers from power grid reports) and format unification (converting non-standard units to national standards and standardizing timestamps); multimodal parsing can include text extraction and image processing; entity and relation extraction can include knowledge graph construction; and chunking and indexing optimization can include dynamic chunking strategies and vectorized indexes. Furthermore, the structured processing of documents in the power grid field yields a retrieval corpus, which is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set. The initial coarse-grained keyword set is a set of keywords that summarizes the core theme and is used for rapid classification and retrieval of document content. A coarse-grained keyword can quickly filter documents or information fragments (evidence) related to that theme from the retrieval corpus. The initial fine-grained keyword set is a set of keywords used to locate details of the evidence. A single piece of evidence can be associated with multiple fine-grained keywords, which allow for precise location of specific details within the evidence. For example, in the analysis of a power outage, "voltage fluctuation data" is a fine-grained keyword that can be used to quickly locate the specific values ​​and changes in voltage fluctuations.

[0034] Specifically, by using a retrieval-enhanced large language model, the retrieval question in the power grid domain is matched with initial document fragments in a retrieval corpus for the power grid domain, thereby determining candidate document fragments. The retrieval question in the power grid domain can be any question related to the power grid domain that needs to be retrieved. The initial document fragments can be multiple chunks obtained by structuring the documents to be processed in the power grid domain. For example, the retrieval question in the power grid domain can be input into the retrieval-enhanced large language model, and initial document fragments with a similarity greater than a preset threshold are determined as candidate document fragments.

[0035] In one optional implementation, standardized questions can be determined based on the initial document fragment and the standardized type question template; the standardized questions can be associated with the initial document fragment to obtain question association samples; and the question association samples can be labeled according to preset labeling rules to obtain the coarse-grained keyword set and the fine-grained keyword set.

[0036] The standardized question template is used to standardize the questions to be retrieved, thereby improving the quality of the questions and facilitating subsequent retrieval and answering. For example, based on the initial document fragments and the standardized question templates, a retrieval enhancement language model can be invoked to automatically generate standardized questions. The language model semantically restructures the document fragments to generate queries (e.g., converting technical terms into a user-friendly question-and-answer format), ensuring that the generated standardized questions are semantically clear, unambiguous, of consistent type, and logically derivable from the corresponding fragments, thus possessing traceability.

[0037] Specifically, standardized questions are associated with initial document fragments to obtain question-related samples. For example, human annotators can score the relevance of the "query-document fragment" relationship from 1 to 5 points, with a score ≥4 considered passing, thus forming question-related samples. Then, according to preset annotation rules, the question-related samples are annotated to obtain coarse-grained and fine-grained keyword sets. Preset annotation rules may include timeliness priority, security compliance, and terminology standardization. For example, according to the preset annotation rules, a retrieval-enhanced large language model can be invoked to annotate the question-related samples, obtaining coarse-grained and fine-grained keyword sets. The coarse-grained keyword set can be a set of keywords covering the highest-level topic of the document fragment, which can be used to quickly categorize the domain to which the question belongs. The fine-grained keyword set can be a set of keywords for locating evidence details, which can be used to accurately locate document fragments supporting the answer. Furthermore, manual verification can be performed; when the keyword matching rate with the "query-document fragment" is ≥80%, it can be judged as qualified. From macro-level topic browsing to micro-level evidence location, the large model enables precise segmentation of topics and evidence, and real-time synchronization of policy changes and security requirements. This annotation method provides structured knowledge support for subsequent scenarios such as intelligent question answering and fault tracing.

[0038] Optionally, the standardized question templates include at least one of the following: factual question templates, comparative question templates, analytical question templates, tutorial question templates, or summary question templates.

[0039] Among them, factual question templates can be used to directly ask about objective facts, focusing on objective facts or definitions, and the answer must directly cite data or standard descriptions (e.g., "What are the common power quality problems in the power grid supply process?"); comparative question templates can be used to compare the differences between two things, requiring a comparison of the characteristics, advantages and disadvantages, or applicable scenarios of two or more objects (e.g., "What are the technical differences between traditional power grids and smart grids in dealing with renewable energy integration?"); analytical question templates can be used to reason or explain causes, requiring the breakdown of the causes, mechanisms, or impacts of a problem, and the proposal of solutions (e.g., "What are the main causes of low voltage problems in rural power grids and what are the remediation measures?"); tutorial question templates can be used to guide specific operations, requiring the provision of step-by-step operation guidelines or technical implementation methods (e.g., "How to use a voltage comparator circuit to monitor the voltage stability of the power grid?"); and summary question templates can be used to summarize long content, requiring a summary of core points, achievements, or development directions (e.g., "Summarize the economic benefits of coordinated dispatch of energy storage systems and distributed energy resources"). Based on standardized question templates, the questions are standardized, improving the quality of the questions and facilitating subsequent retrieval and answering.

[0040] S120. Based on the question to be searched, determine at least one target coarse-grained keyword and at least one target fine-grained keyword from the initial coarse-grained keyword set and the initial fine-grained keyword set.

[0041] Among them, the target coarse-grained keywords can be core topic keywords closely related to the question to be searched, and the target fine-grained keywords can be evidence detail keywords closely related to the question to be searched.

[0042] S130. Based on the target coarse-grained keywords, the candidate document fragments are filtered to obtain the filtered target candidate document fragments.

[0043] For example, when filtering candidate document fragments based on target coarse-grained keywords, only candidate document fragments containing the target coarse-grained keywords can be retained. Multiple target candidate document fragments can be selected to reduce the interference of search results on segmentation evaluation. This is because the search mechanism may lead to three scenarios: ① the target candidate document fragment may be in the middle of the top-n list; ② the target candidate document fragment may be at the bottom of the top-n list; ③ the target candidate document fragment may not be retrieved. These scenarios affect the selection of target candidate document fragments. Therefore, we can explicitly identify and retain target candidate document fragments that can be used for evaluation by using a larger search recall and a coarse-grained keyword filtering mechanism, thereby reducing the interference of search on segmentation evaluation.

[0044] S140. Based on the target fine-grained keywords and the target candidate document fragments, determine the block evaluation results of the search question.

[0045] The segmented evaluation results for the search query can include categories such as poor, average, good, and excellent. For example, the segmented evaluation results for the search query can be determined comprehensively based on the keyword coverage and keyword density of the target fine-grained keywords.

[0046] This application embodiment uses a retrieval-enhanced large language model to match the retrieval question in the power grid domain with initial document fragments in a power grid domain retrieval corpus, thereby determining candidate document fragments. Based on the retrieval question, at least one target coarse-grained keyword and at least one target fine-grained keyword are determined from an initial coarse-grained keyword set and an initial fine-grained keyword set. Based on the target coarse-grained keyword, the candidate document fragments are filtered to obtain filtered target candidate document fragments. Based on the target fine-grained keyword and the target candidate document fragments, the segmentation evaluation result of the retrieval question is determined. The above scheme improves the accuracy of segmentation evaluation by determining the segmentation evaluation result of the retrieval question through target coarse-grained keywords and target fine-grained keywords.

[0047] Example 2

[0048] Figure 2 This is a flowchart of a segmented evaluation method for retrieval enhancement generation according to Embodiment 2 of this application. Based on the technical solutions of the above embodiments, this embodiment refines "determining the segmented evaluation result of the retrieval question based on the target fine-grained keywords and the target candidate document fragments" into "determining keyword coverage and keyword density based on the target fine-grained keywords and the target candidate document fragments; determining the segmented evaluation result of the retrieval question based on the keyword coverage and the keyword density." It should be noted that for parts not detailed in this embodiment, please refer to the relevant descriptions in other embodiments. For example...Figure 2 As shown, the method includes:

[0049] S210. By retrieving and enhancing the large language model, the search question in the power grid field is matched with the initial document fragments in the power grid field search corpus, and candidate document fragments are determined from the initial document fragments based on the matching results; wherein, the search corpus is obtained by structuring the documents to be processed in the power grid field, and the search corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set.

[0050] S220. Based on the question to be searched, determine at least one target coarse-grained keyword and at least one target fine-grained keyword from the initial coarse-grained keyword set and the initial fine-grained keyword set.

[0051] S230. Based on the target coarse-grained keywords, the candidate document fragments are filtered to obtain the filtered target candidate document fragments.

[0052] S240. Determine the keyword coverage and keyword density based on the target fine-grained keywords and the target candidate document fragments.

[0053] A high-quality text chunk should cover the core semantics relevant to the query; therefore, keyword coverage is introduced to measure the text chunk's ability to retain relevant keywords. Keyword coverage measures a document fragment's ability to retain target fine-grained keywords. This metric reflects how many target fine-grained keywords are actually contained in the document fragment; a higher coverage indicates that the document fragment is more likely to contain the information the user needs for their query. Besides covering as many keywords as possible, a high-quality text chunk should also ensure a high keyword density within the chunk to avoid sparse key information; therefore, keyword density is introduced. Keyword density measures the frequency of target fine-grained keywords appearing in the document fragment, reflecting the degree of focus of the semantic content.

[0054] Optionally, determining the keyword coverage and keyword density based on the target fine-grained keywords and the target candidate document fragments includes: determining the number of target keywords based on the number of times the target fine-grained keywords appear in the target candidate document fragments; determining the keyword coverage as the ratio of the number of target keywords to the total number of target fine-grained keywords; determining effective keywords based on the target candidate document fragments and the target fine-grained keywords; and determining the keyword density as the ratio of the total length of the effective keywords to the total length of the target candidate document fragments.

[0055] Specifically, the keyword coverage metric measures the proportion of target fine-grained keywords that actually appear in the candidate blocks of a given sample. Let the set of target fine-grained keywords for the i-th sample be denoted as . The set of keywords that actually appear in all target candidate document fragments in this sample is denoted as . Then the keyword coverage rate of the i-th sample is:

[0056] ;

[0057] in, Represents a set The number of keywords in the text (i.e., the number of target keywords); Represents a set The total number of keywords in the target (i.e., the total number of fine-grained keywords); This represents the keyword coverage rate of the i-th sample.

[0058] Keyword density is a metric used to measure the "density" of a specific set of keywords in a candidate text. Let the target fine-grained keyword set in the i-th sample be... , This indicates the result of splicing all candidate blocks appearing in this sample. If the effective keywords are in the i-th sample, then the keyword density of the i-th sample is:

[0059] ;

[0060] in, For effective keywords exist The number of times it appears in Indicates valid keywords Word length; for The length of all words, Keyword density. The higher the keyword density, the denser the keywords.

[0061] By introducing keyword density and keyword coverage, the structural rationality and semantic coverage of the segmentation stage are measured, and the effectiveness of the RAG system is evaluated more accurately.

[0062] S250. Determine the block evaluation result of the search question based on the keyword coverage and the keyword density.

[0063] Optionally, determining the segmented evaluation result of the retrieval question based on the keyword coverage and the keyword density includes: for each target candidate document segment, determining a comprehensive keyword contribution value based on the product of the keyword coverage and the keyword density; averaging the comprehensive keyword contribution values ​​of all target candidate document segments to obtain a comprehensive keyword average value; determining a comprehensive index score based on the comprehensive keyword average value, a scaling factor, and a preset distribution index function; and determining the segmented evaluation result of the retrieval question based on the comprehensive index score.

[0064] Specifically, the comprehensive index score is determined using the following formula:

[0065] ;

[0066] in, This represents the comprehensive index score; γ represents the scaling factor, which can be set to 100; the preset distribution exponential function is... N represents the total number of candidate document fragments; Keyword coverage; Keyword density. A comprehensive score is determined by combining keyword coverage and keyword density, reflecting the quality of the segmentation. The closer the comprehensive score is to 1, the stronger the segmentation's support for the retrieval query.

[0067] For example, when the comprehensive index score is <0.2, it indicates that the block evaluation result is poor, the block is weakly related to the retrieval question, and the block strategy needs to be redesigned; when the comprehensive index score is between 0.2 and 0.5, it indicates that the block evaluation result is average, some key information is missing, and relevant blocks need to be added and the block algorithm adjusted (such as semantic clustering); when the comprehensive index score is between 0.5 and 0.8, it indicates that the block evaluation result is good, the core information is covered, but there is a small amount of redundancy or omission, and the block size or keyword weight needs to be fine-tuned; when the comprehensive index score is between 0.8 and 1.0, it indicates that the block evaluation result is excellent, the block completely covers the key information, the matching degree is extremely high, and the current block strategy needs to be maintained.

[0068] This application embodiment uses a retrieval-enhanced large language model to match the retrieval question in the power grid domain with initial document fragments in a power grid domain retrieval corpus, thereby determining candidate document fragments. Based on the retrieval question, at least one target coarse-grained keyword and at least one target fine-grained keyword are determined from the initial coarse-grained keyword set and the initial fine-grained keyword set. Based on the target coarse-grained keyword, the candidate document fragments are filtered to obtain filtered target candidate document fragments. Based on the target fine-grained keyword and the target candidate document fragments, keyword coverage and keyword density are determined. Based on keyword coverage and keyword density, the segmentation evaluation result of the retrieval question is determined. The above scheme directly quantifies the segmentation quality using keyword coverage and keyword density, directly measuring the complete preservation of query semantics and information focus of the segmentation, avoiding the uninterpretability caused by indirectly inferring segmentation quality from retrieval or end-to-end result generation. Furthermore, each score can be traced back to the hit keywords, their corresponding original text positions, and their frequency of occurrence, thereby accurately locating specific defects such as "missing key points," "information sparsity," and "irrelevant redundancy."

[0069] Example 3

[0070] Figure 3 This is a schematic diagram of a segmented evaluation device for retrieval enhancement generation according to Embodiment 3 of this application. It is applicable to situations where explicit evaluation of the segmentation stage in a retrieval enhancement generation system is performed. This segmented evaluation device for retrieval enhancement generation can be implemented in hardware and / or software, and can be configured in a computer device, such as a server. Figure 3 As shown, the device includes:

[0071] The candidate document fragment determination module 310 is used to match the search question in the power grid field with the initial document fragments in the power grid field retrieval corpus by retrieval enhancement large language model, and determine candidate document fragments from the initial document fragments according to the matching results; wherein, the retrieval corpus is obtained by structuring the documents to be processed in the power grid field, and the retrieval corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set;

[0072] The target keyword determination module 320 is used to determine at least one target coarse-grained keyword and at least one target fine-grained keyword from the initial coarse-grained keyword set and the initial fine-grained keyword set according to the search question;

[0073] The target candidate document fragment determination module 330 is used to filter the candidate document fragments according to the target coarse-grained keywords to obtain the filtered target candidate document fragments.

[0074] The block evaluation result determination module 340 is used to determine the block evaluation result of the search question based on the target fine-grained keywords and the target candidate document fragments.

[0075] Optionally, the block evaluation result determination module 340 includes:

[0076] The keyword coverage and keyword density determination unit is used to determine the keyword coverage and keyword density based on the target fine-grained keywords and the target candidate document fragments;

[0077] The segmented evaluation result determination unit is used to determine the segmented evaluation result of the search question based on the keyword coverage and the keyword density.

[0078] Optional, keyword coverage and keyword density determination units, specifically used for:

[0079] The number of target keywords is determined based on the number of times the target fine-grained keywords appear in the target candidate document fragments;

[0080] The ratio of the number of target keywords to the total number of target fine-grained keywords is determined as the keyword coverage rate;

[0081] Based on the target candidate document fragments and the target fine-grained keywords, determine the effective keywords;

[0082] The ratio of the total length of the effective keywords to the total length of the target candidate document fragments is determined as the keyword density.

[0083] Optionally, the block evaluation result determination unit is specifically used for:

[0084] For each target candidate document fragment, the comprehensive keyword contribution value is determined based on the product of the keyword coverage and the keyword density.

[0085] The average value of the keyword contribution of all the target candidate document fragments is calculated to obtain the overall keyword average value;

[0086] The comprehensive index score is determined based on the comprehensive average value of the keywords, the scaling factor, and the preset distribution index function;

[0087] Based on the comprehensive index score, the segmented evaluation results of the question to be retrieved are determined.

[0088] Optionally, the device may also include:

[0089] The standardization issue determination module is used to determine the standardization issue based on the initial document fragment and the standardization type issue template;

[0090] The problem association sample determination module is used to associate the standardized problem with the initial document fragment to obtain problem association samples;

[0091] The keyword set determination module is used to annotate the problem-related samples according to preset annotation rules to obtain the coarse-grained keyword set and the fine-grained keyword set.

[0092] The standardized question templates include at least one of the following: factual question templates, comparative question templates, analytical question templates, tutorial question templates, or summary question templates.

[0093] This application embodiment uses a retrieval-enhanced large language model to match the retrieval question in the power grid domain with initial document fragments in a power grid domain retrieval corpus, thereby determining candidate document fragments. Based on the retrieval question, at least one target coarse-grained keyword and at least one target fine-grained keyword are determined from an initial coarse-grained keyword set and an initial fine-grained keyword set. Based on the target coarse-grained keyword, the candidate document fragments are filtered to obtain filtered target candidate document fragments. Based on the target fine-grained keyword and the target candidate document fragments, the segmentation evaluation result of the retrieval question is determined. The above scheme improves the accuracy of segmentation evaluation by determining the segmentation evaluation result of the retrieval question through target coarse-grained keywords and target fine-grained keywords.

[0094] The retrieval enhancement generation segmentation evaluation device provided in this application embodiment can execute the retrieval enhancement generation segmentation evaluation method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each retrieval enhancement generation segmentation evaluation method.

[0095] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0096] Example 4

[0097] Figure 4 This is a schematic diagram of the structure of an electronic device 410 implementing the retrieval enhancement generation block evaluation method of the embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0098] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0099] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as the block evaluation method for retrieving enhanced generation.

[0101] In some embodiments, the retrieval enhancement generation block evaluation method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the retrieval enhancement generation block evaluation method described above can be performed. Alternatively, in other embodiments, processor 411 can be configured for the retrieval enhancement generation block evaluation method by any other suitable means (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable retrieval enhancement generation chunk evaluation device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0107] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0108] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for evaluating the segmentation of retrieval enhancement generation, characterized in that, include: By retrieving an enhanced large language model, the search query in the power grid field is matched with initial document fragments in the power grid field search corpus, and candidate document fragments are determined from the initial document fragments based on the matching results. The search corpus is obtained by structuring the documents to be processed in the power grid field, and the search corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set. Based on the search question, at least one target coarse-grained keyword and at least one target fine-grained keyword are determined from the initial coarse-grained keyword set and the initial fine-grained keyword set; Based on the target coarse-grained keywords, the candidate document fragments are filtered to obtain the filtered target candidate document fragments; Based on the target fine-grained keywords and the target candidate document fragments, the block evaluation results of the search question are determined.

2. The method according to claim 1, characterized in that, The step of determining the segmented evaluation result of the search question based on the target fine-grained keywords and the target candidate document fragments includes: Based on the target fine-grained keywords and the target candidate document fragments, determine the keyword coverage and keyword density; Based on the keyword coverage and keyword density, the segmented evaluation results of the search query are determined.

3. The method according to claim 2, characterized in that, The step of determining keyword coverage and keyword density based on the target fine-grained keywords and the target candidate document fragments includes: The number of target keywords is determined based on the number of times the target fine-grained keywords appear in the target candidate document fragments; The ratio of the number of target keywords to the total number of target fine-grained keywords is determined as the keyword coverage rate; Based on the target candidate document fragments and the target fine-grained keywords, determine the effective keywords; The ratio of the total length of the effective keywords to the total length of the target candidate document fragments is determined as the keyword density.

4. The method according to claim 2, characterized in that, The step of determining the segmentation evaluation result of the search query based on the keyword coverage and the keyword density includes: For each target candidate document fragment, the comprehensive keyword contribution value is determined based on the product of the keyword coverage and the keyword density. The average value of the keyword contribution of all the target candidate document fragments is calculated to obtain the overall keyword average value; The comprehensive index score is determined based on the comprehensive average value of the keywords, the scaling factor, and the preset distribution index function; Based on the comprehensive index score, the segmented evaluation results of the question to be retrieved are determined.

5. The method according to claim 1, characterized in that, The method further includes: Based on the initial document fragment and the standardized type question template, determine the standardized questions; The standardized questions are associated with the initial document fragments to obtain question-associated samples; According to the preset annotation rules, the problem-related samples are annotated to obtain the coarse-grained keyword set and the fine-grained keyword set.

6. The method according to claim 5, characterized in that, The standardized question templates include at least one of the following: factual question templates, comparative question templates, analytical question templates, tutorial question templates, or summary question templates.

7. A retrieval enhancement generation block evaluation device, characterized in that, include: The candidate document fragment determination module is used to match the search query in the power grid field with the initial document fragments in the power grid field retrieval corpus by retrieval enhancement large language model, and determine candidate document fragments from the initial document fragments based on the matching results; wherein, the retrieval corpus is obtained by structuring the documents to be processed in the power grid field, and the retrieval corpus is associated with an initial coarse-grained keyword set and an initial fine-grained keyword set; The target keyword determination module is used to determine at least one target coarse-grained keyword and at least one target fine-grained keyword from the initial coarse-grained keyword set and the initial fine-grained keyword set according to the search question; The target candidate document fragment determination module is used to filter the candidate document fragments based on the target coarse-grained keywords to obtain the filtered target candidate document fragments. The block evaluation result determination module is used to determine the block evaluation result of the search question based on the target fine-grained keywords and the target candidate document fragments.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the chunked evaluation method for retrieval enhancement generation as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the block evaluation method for retrieval enhancement generation as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the chunking evaluation method for retrieval enhancement generation according to any one of claims 1-6.

Images