Method for evaluating document segmentation effect in large model retrieval enhancement generation

Abstract

The invention discloses a method for evaluating a document segmentation effect in large model retrieval enhancement generation, and relates to the technical field of document segmentation, and the method comprises the following steps: obtaining segmentation pairs obtained after segmentation processing of a to-be-evaluated document, inputting the segmentation pairs into a general semantic model according to a sequence to obtain a target evaluation value corresponding to each segmentation pair, determining target effect levels corresponding to all the target evaluation values based on the corresponding relationship between the evaluation values and the effect levels; the training process of the universal semantic model specifically comprises the following steps: segmenting a training document to obtain at least two original blocks; randomly segmenting any original block to obtain a preset number of slices; marking according to whether semantic relevance exists or not to obtain n groups of training samples; and calculating a target correlation score and a target separation score corresponding to any group of training samples, and determining an evaluation value corresponding to the training sample. The score of the segmentation effect corresponding to each document can be fed back, and document segmentation can be assisted.

Description

Background Art

[0002] Retrieval Augmented Generation is a technology applied in the field of knowledge content retrieval. It combines vector databases and large language models, and aims to enhance the generation capabilities of large models through retrieval technology. The construction process of the vector database is: document preparation → document segmentation → slice vectorization → storage. Document segmentation is to optimize the construction of the vector database and improve the generation quality of the large language model. In this technology, the large model will make reference answers based on the document segmentation content after retrieval sorting, so document segmentation is particularly important for the answer quality of the large model, and how to evaluate its segmentation effect is also a problem that needs to be solved. Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a method for evaluating the effect of document segmentation in large model retrieval enhancement generation, which is as follows:

[0004] 1) In the first aspect, the present invention provides a method for evaluating the effect of document segmentation in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0005] Obtain segmentation pairs obtained after segmentation of the document to be evaluated, input the segmentation pairs into the general semantic model in order, obtain the target evaluation value corresponding to each segmentation pair, and determine the target effect level corresponding to all target evaluation values ​​based on the corresponding relationship between the evaluation value and the effect level;

[0006] The training process of the general semantic model is specifically as follows:

[0007] Obtaining a training document, and segmenting the training document into at least two original segments;

[0008] Randomly divide any original block into slices to obtain a preset number of slices;

[0009] According to whether there is semantic correlation, any two original blocks, any two slices, and any combination of an original block and any slice are labeled to obtain n groups of training samples;

[0010] Calculate the target relevance score and target separability score corresponding to any set of training samples, and determine the evaluation value corresponding to the training sample based on the relevance score weight and the separability score weight.

[0011] The beneficial effects of the method for evaluating the effect of document segmentation in large model retrieval enhancement generation provided by the present invention are as follows:

[0012] By expanding and adjusting the training set during the training process of the general semantic model, the number of training sets can be increased while the balance of the labeled categories can be controlled. In addition, providing intuitive feedback on the score of the segmentation effect corresponding to each document can also help assist in document segmentation.

[0013] Based on the above solution, the present invention can also be improved as follows.

[0014] Furthermore, based on the correspondence between the evaluation value and the effect level, the process of determining the target effect level corresponding to all target evaluation values ​​is specifically as follows:

[0015] According to all target evaluation values ​​and the target relevance scores corresponding to each segmentation pair, the sum of all target evaluation values, i.e., the total evaluation value, is determined, and the target effect level corresponding to the total evaluation value in the corresponding relationship between the evaluation value and the effect level is determined.

[0016] Furthermore, the process of determining the sum of all target evaluation values ​​is as follows:

[0017]

[0018] Among them, score is the total evaluation value, rel_score(i) is the target relevance score corresponding to the i-th group of training samples, and weight_score(i) is the evaluation value corresponding to the i-th group of training samples.

[0019] Further, the method for determining the evaluation value corresponding to the training sample is:

[0020] weight_score=0.7*sep_score+0.3*rel_score;

[0021] Among them, weight_score is the evaluation value, sep_score is the target separation score, and rel_score is the target relevance score.

[0022] 2) In the second aspect, the present invention also provides a system for evaluating the effect of document segmentation in large model retrieval enhancement generation, and the specific technical solution is as follows:

[0023] The evaluation module is used to: obtain segmentation pairs obtained after segmentation processing of the document to be evaluated, input the segmentation pairs into the general semantic model in order, obtain the target evaluation value corresponding to each segmentation pair, and determine the target effect level corresponding to all target evaluation values ​​based on the corresponding relationship between the evaluation value and the effect level;

[0024] The training process of the general semantic model is specifically as follows:

[0025] Obtaining a training document, and segmenting the training document into at least two original segments;

[0026] Randomly divide any original block into slices to obtain a preset number of slices;

[0027] According to whether there is semantic correlation, any two original blocks, any two slices, and any combination of an original block and any slice are labeled to obtain n groups of training samples;

[0028] Calculate the target relevance score and target separability score corresponding to any set of training samples, and determine the evaluation value corresponding to the training sample based on the relevance score weight and the separability score weight.

[0029] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0030] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0031] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0033] Figure 1 A flowchart of a method for evaluating the effect of document segmentation in a large model retrieval enhancement generation according to an embodiment of the present invention;

[0034] Figure 2 The figure is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, a method for evaluating the effect of document segmentation in a large model retrieval enhancement generation according to an embodiment of the present invention, the specific technical solution is as follows:

[0037] S1, obtaining segmentation pairs obtained after segmentation processing of the document to be evaluated, inputting the segmentation pairs into the general semantic model in order, obtaining the target evaluation value corresponding to each segmentation pair, and determining the target effect level corresponding to all the target evaluation values ​​based on the corresponding relationship between the evaluation value and the effect level;

[0038] The training process of the general semantic model is specifically as follows:

[0039] Obtaining a training document, and segmenting the training document into at least two original segments;

[0040] Randomly divide any original block into slices to obtain a preset number of slices;

[0041] According to whether there is semantic correlation, any two original blocks, any two slices, and any combination of an original block and any slice are labeled to obtain n groups of training samples;

[0042] Calculate the target relevance score and target separability score corresponding to any set of training samples, and determine the evaluation value corresponding to the training sample based on the relevance score weight and the separability score weight.

[0043] The beneficial effects of the method for evaluating the effect of document segmentation in large model retrieval enhancement generation provided by the present invention are as follows:

[0044] By expanding and adjusting the training set during the training process of the general semantic model, the number of training sets can be increased while the balance of the labeled categories can be controlled. In addition, providing intuitive feedback on the score of the segmentation effect corresponding to each document can also help assist in document segmentation.

[0045] The cutting pair may be composed of two cut blocks or two slices or one cut block and one slice.

[0046] The segments are obtained by segmenting the sentences according to commas.

[0047] Slices are obtained by cutting each block again.

[0048] The specific process of segmenting the document to be evaluated is as follows:

[0049] The first method is to segment the document to be evaluated according to paragraphs. For any paragraph obtained after segmentation, segment the paragraph into sentences according to periods or semicolons, determine the sentence length of the segmented sentences in real time, and record them in association with the serial number identifiers corresponding to the sentences to generate a sentence length record table. Determine the length segmentation range corresponding to documents of different categories, determine the target length segmentation range based on the category of the current document to be evaluated, and divide the sentence lengths in the sentence length record table into levels according to the target length segmentation range, determine the serial number identifier corresponding to each level, and divide the sentences corresponding to each serial number identifier at the level into slices according to the number of divisions corresponding to the level, and obtain the slicing results corresponding to each serial number identifier.

[0050] The process of determining the length segmentation range corresponding to documents of different categories is as follows:

[0051] By retrieving different categories of historical documents in the historical database, the corresponding slicing results of each category of historical documents are determined, and the slicing results are integrated to determine the length of historical sentences corresponding to two slicing results, the length of historical sentences corresponding to three slicing results, four... And according to the statistical results, the corresponding table of generated segmentation lengths for each category of documents is determined.

[0052] Regarding the use of the segmentation length correspondence table, by determining the category of the document to be evaluated, the segmentation length correspondence table of the corresponding category is called. The length range corresponding to different numbers of slices can be obtained in the segmentation length correspondence table, that is, the target length segmentation range mentioned above. The sentence length is compared in real time to see which target length segmentation range it falls within, and the segmentation of the sentence corresponding to the sentence length is completed according to the corresponding number of segmentations.

[0053] The second method is to segment the document to be evaluated into paragraphs, determine a fixed segmentation length, and segment any paragraph according to the fixed segmentation length to obtain at least two segments. It should be further explained that in the process of segmentation according to the fixed segmentation length, if there are commas or semicolons in the segments obtained during the segmentation process, it is necessary to re-segment, that is, segment according to the commas or semicolons, and after segmentation, start segmenting again according to the fixed segmentation length from the first text content after the comma or semicolon. For each segmentation result, the slicing process is performed according to the slicing method in the first method.

[0054] Furthermore, based on the correspondence between the evaluation value and the effect level, the process of determining the target effect level corresponding to all target evaluation values ​​is specifically as follows:

[0055] According to all target evaluation values ​​and the target relevance scores corresponding to each segmentation pair, the sum of all target evaluation values, i.e., the total evaluation value, is determined, and the target effect level corresponding to the total evaluation value in the corresponding relationship between the evaluation value and the effect level is determined.

[0056] Furthermore, the process of determining the sum of all target evaluation values ​​is as follows:

[0057]

[0058] Among them, score is the total evaluation value, rel_score(i) is the target relevance score corresponding to the i-th group of training samples, and weight_score(i) is the evaluation value corresponding to the i-th group of training samples.

[0059] Further, the method for determining the evaluation value corresponding to the training sample is:

[0060] weight_score=0.7*sep_score+0.3*rel_score;

[0061] Among them, weight_score is the evaluation value, sep_score is the target separation score, and rel_score is the target relevance score.

[0062] Example 1, 1. Prepare training data and train a separate classification model

[0063] Through manual annotation, original document segments of 100 papers were prepared according to their semantic contents.

[0064] Training data format: [text block 1, text block 2 label 0 / 1]

[0065] The text block 1 / 2 can be a complete original block or a slice of the original block. The label 0 represents separation and 1 represents non-separation.

[0066] For example, a document is manually divided into text block A, text block B, text block C, and text block D according to content coherence, which means that there are semantic differences between any two adjacent text blocks A, B, C, and D.

[0067] During training, data processing is performed on these blocks:

[0068]

Text block A, text block B 0

[0069]

Text block A, text block C 0

[0070]

Text block A, text block D 0

[0071]

Text block B, text block C 0

[0072]

Text block B, text block D 0

[0073]

Text block C, text block D 0

[0074] [Part 1 of text block A, Part 2 of text block A 1]

[0075] [Part 2 of text block A, Part 3 of text block A 1]

[0076] [Part 1 of text block A, Part 3 1 of text block A]

[0077] [Partial 1 of text block A, Partial 3 0 of text block B]

[0078] [Partial 2 of text block A, Partial 2 0 of text block C]

[0079] [Partial 3 of text block A, Partial 1 0 of text block D]

[0080] In addition to the training samples directly constructed from the complete original blocks, each of the original blocks will be randomly divided into three slices that may have some overlapping content but are not exactly the same (for example, "Artificial intelligence is amazing" can be sliced ​​into three slices of "Artificial, intelligent, and intelligent are amazing"). Different slices are combined in pairs, including combinations of slices belonging to the same original block and combinations of slices belonging to different original blocks.

[0081] If slice A and slice B belong to the same original cut, the label is 1, and if they belong to different original cuts, the label is 0.

[0082] In particular, if they belong to different original blocks, the slices in each original block can only randomly match one of the slices in the remaining original blocks to control the category balance.

[0083] Based on this, a text block related judgment model is trained by connecting a fully connected layer with an output dimension of 2 on the basis of the general BERT.

[0084] By adding a softmax layer after the fully connected layer, the separability score sep_score between the two input blocks can be obtained, and the score range is 0 to 1.

[0085] 2. NSP calculation adjacency of general semantic model BERT

[0086] Based on the untrained general BERT, calculate the relevance score (nsp_score) of the current text block and the previous text block.

[0087] Since the model is pre-trained for context judgment tasks, it can directly determine whether there is context between two sentences.

[0088] In this task, two adjacent text blocks are separated by [SEP], [CLS] is added at the beginning of the first text block, and [SEP] is added at the end of the second text block.

[0089] For example: [CLS] Artificial intelligence is amazing [SEP] It accelerates the speed at which humans understand the world [SEP]

[0090] rel_score = bert_nsp_pre(C,S)

[0091] C represents the current text block, and S represents the previous text block of the current line.

[0092] Bert_nsp_pre is the same as above, which is the result of the output of Bert's final fully connected layer and the softmax layer. The score range is also 0 to 1.

[0093] 3. Score weighting

[0094] Considering the different weights of the two, separability has a greater impact in practice. After the experiment, the weight of the separability score in the combined evaluation function was increased, and the weight of the adjacency was reduced. The final evaluation function was:

[0095] weight_score=0.7*sep_score+0.3*rel_score

[0096] 4. Usage

[0097] When the entire document is provided with segmented text blocks, the text blocks are sent to the trained separability prediction model and the general BERT model in order of two adjacent blocks, and the corresponding scores are obtained and recorded until the last two adjacent text blocks are sent. The sum of all scores is divided by the total number of segmented text blocks in the document to obtain a final score, which is expressed as:

[0098]

[0099] Based on this, we can obtain a solution that can use scores to intuitively evaluate the effect of document segmentation. This score can be used to objectively evaluate the quality of document segmentation, which is convenient for experimental comparison.

[0100] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0101] The present invention also provides a system for evaluating the effect of document segmentation in large-model retrieval enhancement generation, and the specific technical solution is as follows:

[0102] The evaluation module is used to: obtain segmentation pairs obtained after segmentation processing of the document to be evaluated, input the segmentation pairs into the general semantic model in order, obtain the target evaluation value corresponding to each segmentation pair, and determine the target effect level corresponding to all target evaluation values ​​based on the corresponding relationship between the evaluation value and the effect level;

[0103] The training process of the general semantic model is specifically as follows:

[0104] Obtaining a training document, and segmenting the training document into at least two original segments;

[0105] Randomly divide any original block into slices to obtain a preset number of slices;

[0106] According to whether there is semantic correlation, any two original blocks, any two slices, and any combination of an original block and any slice are labeled to obtain n groups of training samples;

[0107] Calculate the target relevance score and target separability score corresponding to any set of training samples, and determine the evaluation value corresponding to the training sample based on the relevance score weight and the separability score weight.

[0108] It should be noted that the beneficial effects of the system for evaluating the effect of document segmentation in the large model retrieval enhancement generation provided in the above embodiment are the same as the beneficial effects of the method for evaluating the effect of document segmentation in the large model retrieval enhancement generation, which will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0109] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0110] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a method for evaluating the document segmentation effect in a large model retrieval enhancement generation provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for realizing the functions of the device, which will not be repeated here.

[0111] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0112] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0113] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0114] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0115] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0116] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0117] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for evaluating the effect of document segmentation in large model retrieval enhancement generation, characterized in that: include: Obtain segmentation pairs obtained after segmentation of the document to be evaluated, input the segmentation pairs into the general semantic model in order, obtain the target evaluation value corresponding to each segmentation pair, and determine the target effect level corresponding to all target evaluation values ​​based on the corresponding relationship between the evaluation value and the effect level; The training process of the general semantic model is specifically as follows: Obtaining a training document, and segmenting the training document into at least two original segments; Randomly divide any original block into slices to obtain a preset number of slices; According to whether there is semantic correlation, any two original blocks, any two slices, and any combination of an original block and any slice are labeled to obtain n groups of training samples; Calculate the target relevance score and target separability score corresponding to any set of training samples, and determine the evaluation value corresponding to the training sample based on the relevance score weight and the separability score weight.

2. According to the method for evaluating the effect of document segmentation in large model retrieval enhancement generation according to claim 1, it is characterized in that: Based on the correspondence between evaluation values ​​and effect levels, the process of determining the target effect levels corresponding to all target evaluation values ​​is as follows: According to all target evaluation values ​​and the target relevance scores corresponding to each segmentation pair, the sum of all target evaluation values, i.e., the total evaluation value, is determined, and the target effect level corresponding to the total evaluation value in the corresponding relationship between the evaluation value and the effect level is determined.

3. According to claim 2, the method for evaluating the effect of document segmentation in large model retrieval enhancement generation is characterized in that: The process of determining the sum of all target evaluation values ​​is as follows: Among them, score is the total evaluation value, rel_score(i) is the target relevance score corresponding to the i-th group of training samples, and weight_score(i) is the evaluation value corresponding to the i-th group of training samples.

4. According to claim 1, the method for evaluating the effect of document segmentation in the large model retrieval enhancement generation is characterized in that: The method for determining the evaluation value corresponding to the training sample is: weight_score=0.7*sep_score+0.3*rel_score; Among them, weight_score is the evaluation value, sep_score is the target separation score, and rel_score is the target relevance score.

5. A system for evaluating the effect of document segmentation in large-scale model retrieval enhancement generation, characterized in that: include: The evaluation module is used to: obtain segmentation pairs obtained after segmentation processing of the document to be evaluated, input the segmentation pairs into the general semantic model in order, obtain the target evaluation value corresponding to each segmentation pair, and determine the target effect level corresponding to all target evaluation values ​​based on the corresponding relationship between the evaluation value and the effect level; The training process of the general semantic model is specifically as follows: Obtaining a training document, and segmenting the training document into at least two original segments; Randomly divide any original block into slices to obtain a preset number of slices; According to whether there is semantic correlation, any two original blocks, any two slices, and any combination of an original block and any slice are labeled to obtain n groups of training samples; Calculate the target relevance score and target separability score corresponding to any set of training samples, and determine the evaluation value corresponding to the training sample based on the relevance score weight and the separability score weight.

6. The system for evaluating the effect of document segmentation in large model retrieval enhancement generation according to claim 5, characterized in that: Based on the correspondence between evaluation values ​​and effect levels, the process of determining the target effect levels corresponding to all target evaluation values ​​is as follows: According to all target evaluation values ​​and the target relevance scores corresponding to each segmentation pair, the sum of all target evaluation values, i.e., the total evaluation value, is determined, and the target effect level corresponding to the total evaluation value in the corresponding relationship between the evaluation value and the effect level is determined.

7. The system for evaluating the effect of document segmentation in large model retrieval enhancement generation according to claim 5, characterized in that: The process of determining the sum of all target evaluation values ​​is as follows: Among them, score is the total evaluation value, rel_score(i) is the target relevance score corresponding to the i-th group of training samples, and weight_score(i) is the evaluation value corresponding to the i-th group of training samples.

8. The system for evaluating the effect of document segmentation in large model retrieval enhancement generation according to claim 5, characterized in that: The method for determining the evaluation value corresponding to the training sample is: weight_score=0.7*sep_score+0.3*rel_score; Among them, weight_score is the evaluation value, sep_score is the target separation score, and rel_score is the target relevance score.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images