Text semantic chunking method, device, equipment, medium and product

Abstract

The present application discloses a text semantic chunking method, apparatus, device, medium and product. By performing weighted summation on the embedding vectors of each clause within a sliding window, the overall semantic features of the clauses within the window can be reflected, reducing the influence of noise. Calculating the similarity between the weighted average embedding vectors of adjacent sliding windows can more accurately identify the semantically continuous parts in the document. Adjusting the similarity array based on the target coefficient of variation enables the segmentation effect of different documents to remain relatively stable. Dynamically adjusting the similarity threshold according to the target coefficient of variation and segmentation information allows the segmentation process to adapt to the structures and contents of different documents. The segmentation information indicates the range of the number of clauses included in the segmentation, making the segmentation result more in line with actual requirements and avoiding overly long or short segments. Dividing the target similarity array according to the dynamic similarity threshold and determining the corresponding clauses as the segments of the document to be processed improves the accuracy and rationality of text chunking.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to methods, devices, equipment, media and products for text semantic segmentation. Background Art

[0002] Text segmentation technology is an important part of natural language processing. Its purpose is to segment long text into shorter text blocks with certain semantic integrity to facilitate subsequent processing (such as retrieval, generation, classification, etc.).

[0003] In related technologies, a fixed similarity threshold is used to segment the text of all documents. However, this method has poor discrimination when segmenting documents of different types and topics, and may segment text blocks that are too long or too short, and cannot adapt to the dynamic segmentation of document content of different topics.

[0004] Therefore, how to improve the accuracy and rationality of text segmentation is an urgent problem that needs to be solved. Summary of the Invention

[0005] The present application provides a text semantic segmentation method, apparatus, device, medium and product to at least solve the problem of low accuracy and rationality of text segmentation in related technologies.

[0006] This application provides a text semantic segmentation method, which includes:

[0007] Obtaining the weighted average embedding vectors of each sliding window corresponding to the document to be processed; the weighted average embedding vector of any sliding window is obtained by weighted summing the embedding vectors of each clause of the document to be processed within the window;

[0008] Obtaining a similarity array of the document to be processed; the similarity array includes: similarities between weighted average embedding vectors of adjacent sliding windows;

[0009] Adjust the similarity array based on the target variation coefficient to obtain the target similarity array;

[0010] Determining a dynamic similarity threshold based on the target coefficient of variation and segmentation information, wherein the segmentation information is used to indicate the range of the number of clauses contained in the segment;

[0011] Dividing the target similarity array according to the dynamic similarity threshold to obtain a plurality of sub-similarity arrays;

[0012] The clauses corresponding to the respective sub-similarity arrays are respectively determined as segments of the document to be processed.

[0013] The present application also provides a text semantic segmentation device, comprising:

[0014] An acquisition module, configured to acquire weighted average embedding vectors of each sliding window corresponding to a document to be processed; the weighted average embedding vector of any sliding window is obtained by performing weighted summation on the embedding vectors of each clause in the window of the document to be processed;

[0015] A calculation module, configured to acquire a similarity array of the document to be processed; the similarity array includes: similarities between weighted average embedding vectors of each pair of adjacent sliding windows;

[0016] An adjustment module, configured to adjust the similarity array based on a target coefficient of variation to obtain a target similarity array;

[0017] An analysis module, configured to determine a dynamic similarity threshold according to the target coefficient of variation and segmentation information; the segmentation information is used to indicate the range of the number of clauses included in a segment;

[0018] A division module, configured to divide the target similarity array according to the dynamic similarity threshold to obtain a plurality of sub-similarity arrays;

[0019] A determination module, configured to respectively determine the clauses corresponding to each sub-similarity array as segments of the document to be processed.

[0020] This application also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above text semantic chunking methods when executing the computer program.

[0021] This application also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above text semantic chunking methods are implemented.

[0022] This application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any of the above text semantic chunking methods are implemented.

[0023] Through this application, weighted average embedding vectors of each sliding window corresponding to a document to be processed are obtained. Among them, the weighted average embedding vector of any sliding window is obtained by weighted summation of the embedding vectors of each clause in the window of the document to be processed. A similarity array of the document to be processed is obtained. The similarity array includes: similarities between the weighted average embedding vectors of each pair of adjacent sliding windows. The similarity array is adjusted based on a target coefficient of variation to obtain a target similarity array. A dynamic similarity threshold is determined according to the target coefficient of variation and segmentation information, where the segmentation information is used to indicate the range of the number of clauses included in a segment. The target similarity array is divided according to the dynamic similarity threshold to obtain multiple sub-similarity arrays. The clauses corresponding to each sub-similarity array are respectively determined as segments of the document to be processed. By performing weighted summation on the embedding vectors of each clause within a sliding window, the overall semantic features of the clauses within the window can be reflected, reducing the influence of noise. Calculating the similarities between the weighted average embedding vectors of adjacent sliding windows can more accurately identify the semantically continuous parts in the document. Adjusting the similarity array based on the target coefficient of variation enables the segmentation effect of different documents to remain relatively stable. Dynamically adjusting the similarity threshold according to the target coefficient of variation and segmentation information enables the segmentation process to adapt to the structures and contents of different documents. The segmentation information indicates the range of the number of clauses included in a segment, making the segmentation result more in line with actual requirements and avoiding overly long or short segments. Dividing the target similarity array according to the dynamic similarity threshold and determining the corresponding clauses as segments of the document to be processed improves the accuracy and rationality of text chunking. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure.

[0025] To more clearly illustrate the embodiments of this application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 It is a schematic flowchart of a text semantic chunking method provided by an embodiment of this application;

[0027] Figure 2 It is a schematic structural diagram of a text semantic chunking device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0029] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0030] In order to enable those skilled in the art of this technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0031] Term Explanation:

[0032] Chunk: Or known as chunking, it refers to the process of decomposing large chunks of text into smaller segments. This is particularly important in the Retrieval-Augmented Generation (RAG) system, which can help optimize the accuracy of the content recalled from the vector database.

[0033] RAG: Retrieval-Augmented Generation, which is a technology that combines Information Retrieval (IR) and Natural Language Generation (NLG). It enhances the text generation ability of the model by retrieving relevant information from an external knowledge base and using it as context input to a large language model (LLM).

[0034] Vector: Commonly represented by v in mathematical form, and its Chinese meaning is vector. In the present application, it means converting the target text chunk into a vector form using a certain machine learning encoder model. Vector is related to the meaning of Embedding and is generally used as the output of Embedding.

[0035] Embedding: Embedding refers to the process of mapping original, usually discrete text or image data (such as phrases, sentences, documents, pictures, etc., which are the "chunks" mentioned above) into a continuous, low-dimensional vector space. These generated vectors are called Embedding vectors. Each Embedding vector is regarded as the semantic representation of its corresponding phonetic, morphological, graphic, and text elements, capable of capturing semantic relationships, syntactic structures, and context dependencies between words. For example, similar or related words are closer in the Embedding space, reflecting their semantic commonalities or correlations.

[0036] L2 norm: Also known as the Euclidean norm, the L2 norm of a vector is a commonly used measure of the length of a vector and is particularly important in mathematics and engineering, especially in fields such as signal processing, linear algebra, and machine learning. The L2 norm is based on the concept of distance in Euclidean space and can be understood as the straight-line distance from the origin to the position indicated by the vector.

[0037] L2 regularization: Also known as Ridge regularization or weight decay, it is a commonly used regularization technique that limits the magnitude of model parameters by adding a penalty term proportional to the square of the weight parameters to the loss function. The main role of L2 regularization is to reduce the complexity of the model by penalizing large weight values, thereby preventing overfitting.

[0038] Common text chunking methods include: chunking by a fixed length, but this is likely to split text with the same semantics in the middle. Based on this, there is also a fixed-length chunking method that retains a certain amount of overlapping redundant words. Further, there is a sentence-based segmentation method, such as segmenting according to punctuation marks, etc., which is also likely to separate sentences with the same semantics. This method has no essential difference from the fixed-length segmentation method. In addition, there are some special chunking methods for document formats, which perform specific processing for different formats; currently, the most ideal is the semantic chunking method, which divides text chunks according to the degree of semantic similarity. However, existing technologies use a fixed similarity threshold for text segmentation of all documents, with poor discrimination when segmenting documents of different types and topics, possibly resulting in text chunks that are too long or too short, and when changing parameters such as sentence length, it is necessary to recalculate the embedding vectors of all document-related sentence clusters using the Embedding model, and the repeated calculation also wastes a large amount of time and computing power, reducing the availability of business services.

[0039] Based on the above problems, embodiments of the present application provide a text semantic chunking method, and the method is described in detail in combination with the execution process of the text semantic chunking method.

[0040] Refer to Figure 1 As shown, the text semantic chunking method provided by the embodiments of the present invention includes the following steps:

[0041] S11. Obtain the weighted average embedding vectors of each sliding window corresponding to the document to be processed.

[0042] Among them, the weighted average embedding vector of any sliding window is obtained by weighted summing the embedding vectors of each clause in the window of the document to be processed.

[0043] Specifically, calculate to obtain the weighted average embedding vectors of each sliding window corresponding to the document to be processed.

[0044] In some embodiments, the above step S11 (obtain the weighted average embedding vectors of each sliding window corresponding to the document to be processed) can be implemented by the following steps:

[0045] (1) Obtain the document to be processed and split the document to be processed into multiple clauses.

[0046] Specifically, obtain the document to be processed, read all the content of the document to be processed, and split the document to be processed into independent sentences according to punctuation marks, that is, split the document to be processed into multiple clauses.

[0047] (2) Input the multiple clauses into a target embedding model for embedding processing to obtain the embedding vectors of the multiple clauses.

[0048] Specifically, perform embedding processing on all independent clauses to obtain independent embedding vectors of all clauses of the document to be processed. Embedding processing is the process of mapping independent clauses of the document to be processed into a continuous vector space.

[0049] Aiming at the disadvantage that the window size needs to be recalculated every time it changes, the weighted vector coding method with variable clause length based on independent clause embedding vectors provided by the embodiments of the present disclosure only needs to calculate the independent clause embedding vectors once, and can realize dynamic modification of the window size without recalculating the independent clause embedding vectors, and the calculation and coding efficiency is significantly improved compared with the method of recalculating the window sentence embedding vectors in the related art.

[0050] (3) Obtain a target weight combination.

[0051] Among them, the target weight combination includes: the weights corresponding to the embedding vectors of multiple clauses of the sliding window.

[0052] The size of the sliding window can be set according to the actual application scenario, and no specific limitation is provided here. For example, when the size of the sliding window is 3, that is, each sliding window contains 3 consecutive clauses.

[0053] In some embodiments, the above step (3) can be implemented in the following manner:

[0054] 1) Determine whether the target database includes the weight combination corresponding to the target embedding model and the target window size.

[0055] Among them, the target database includes: multiple embedding model identifiers and multiple weight combinations corresponding to multiple sliding window sizes; the target window size is the size of the sliding window. The embedding model identifier includes but is not limited to the name of the embedding model, the number of the embedding model, etc.

[0056] Specifically, since the target database includes: multiple embedding model identifiers and multiple weight combinations corresponding to multiple sliding window sizes, the query condition is the target embedding model identifier and the target sliding window size. Thus, query in the target database according to the target embedding model and the target window size to determine whether there is a weight combination corresponding to the target embedding model identifier and the target sliding window size in the target database.

[0057] 2) If the target database includes the combined weight corresponding to the target embedding model and the target window size, then use the combined weight as the target combined weight.

[0058] Specifically, if the target database includes the combined weight corresponding to the target embedding model and the target window size, then use this combined weight as the target combined weight. In this way, when calculating the embedding vector of the sentence within the sliding window, there is no need to repeatedly use the embedding model for embedding processing. Directly read the existing target combined weight from the target database, and then perform simple mathematical operations with the independent embedding vector. This process is significantly faster than using the embedding model for calculation.

[0059] Through the calculation and preservation of the combined weight matrix, the repeated calculation during each window adjustment is avoided, the consumption of computing resources is greatly reduced, and the efficiency of text processing is improved.

[0060] 3) If the target database does not include the combined weight corresponding to the target embedding model and the target window size, then obtain the target weight combination based on the target window size and the embedding vectors of the multiple clauses.

[0061] Specifically, initially, there is no combined weight information in the target database of the system. Through the following calculation method, calculate and obtain the combined weight information.

[0062] In some embodiments, step 3) above can be implemented through the following steps:

[0063] a. Obtain the target embedding vector and the initial weight combination of the multiple clauses.

[0064] Among them, the target embedding vector is a combined embedding vector obtained by combining multiple clauses within the target window into a target clause and then performing embedding processing on the combined clause.

[0065] Optionally, step a) above can be performed through the following steps:

[0066] Combine multiple clauses within the target window into a target clause;

[0067] According to the target embedding model, perform embedding processing on the target clause to obtain the target embedding vector of the target clause;

[0068] Initialize the weights of multiple clauses within the target window to obtain the initial weight combination of the multiple clauses.

[0069] Among them, based on the weight constraint conditions, initialize the weights of multiple clauses within the target window to obtain the initial weight combination of the multiple clauses. The weight constraint conditions include equality constraints and boundary constraints. The equality constraint means that the sum of the weights of each clause within the target window is 1; the boundary constraint means that the weights of each clause within the target window are all between 0 and 1, that is, all weights satisfy .

[0070] Specifically, combine multiple clauses within the target window into a target clause, perform embedding processing on the target clause according to the target embedding model to obtain the target embedding vector of the target clause; initialize the weights of multiple clauses within the target window to obtain the initial weight combination of the multiple clauses.

[0071] Exemplarily, assume that the initial weights are set to a uniform distribution, that is, , represents the size of the sliding window. In this embodiment, can take a value of 3. Therefore, initialize the weights of multiple clauses within the target window to obtain the initial weight combination of the multiple clauses as .

[0072] b. According to the target embedding vector and the embedding vectors of the multiple clauses, optimize the initial weight combination to obtain the target weight combination.

[0073] In some embodiments, step b) above can be performed through the following steps:

[0074] According to the initial weight combination of the multiple clauses, perform weighted average processing on the embedding vectors of the multiple clauses within the target window to obtain the initial weighted average embedding vector;

[0075] Adjust the initial weight combination according to the initial weighted average embedding vector and the target embedding vector to obtain the current weight combination of the multiple clauses;

[0076] Perform weighted average processing on the embedding vectors of the multiple clauses within the target window according to the current weight combination of the multiple clauses to obtain the current weighted average embedding vector;

[0077] When the current weighted average embedding vector and the target embedding vector meet the preset convergence condition, obtain the target weight combination.

[0078] Specifically, first perform weighted average processing on the embedding vectors of the multiple clauses within the target window according to the initial weight combination of the multiple clauses to obtain the initial weighted average embedding vector, then adjust the initial weight combination according to the initial weighted average embedding vector and the target embedding vector to obtain the current weight combination of the multiple clauses within the target window, and then perform weighted average processing on the embedding vectors of the multiple clauses within the target window according to the current weight combination of the multiple clauses within the target window to obtain the current weighted average embedding vector, until the current weighted average embedding vector and the target embedding vector meet the preset convergence condition, and obtain the target weight combination.

[0079] Suppose the size of the sliding window is 3, the independent embedding vectors of all clauses within a group of windows and a target embedding vector , where each group of embedding vectors v in E is a multi-dimensional vector, generally 512-dimensional or 768-dimensional. A set of weights is found through the following method such that the weighted average embedding vector is as close as possible to the target embedding vector T. Among them, the weighted average embedding vector is:

[0080] . Among them, W·E is matrix multiplication, which means applying the weight vector W to each column of the embedding matrix E and summing the results.

[0081]

[0082] Among them, represents the value of the objective function, represents the Euclidean distance between the weighted average embedding vector and the target embedding vector, is a constant, represents the -th current weight of the clause. represents the weighted average embedding vector, Represents the target embedding vector. The objective function is used to measure the difference between the weighted average embedding vector and the target embedding vector.

[0083] Specifically, the first term (before the plus sign) in the above formula is the Euclidean distance between the weighted average embedding vector and the target embedding vector, and the second term (after the plus sign) is the L2 regularization term, which is used to prevent overfitting. By restricting the magnitude of the weights, the model becomes smoother, thereby improving the generalization ability. The parameter is a constant used to control the strength of regularization. In this embodiment can take a value of 0.01.

[0084] By minimizing the objective function , the optimal weight vector is found, that is, the target weight combination is found. A preset optimization function is used to optimize the objective function. Among them, the preset optimization function includes but is not limited to the Trust Region Method or SLSQP (Sequential Least Squares Programming Method). Specifically, first, an initial point (initial solution) is selected. According to the information of the current point, the search direction is calculated and the current point is updated. It is checked whether the preset convergence conditions are met (such as the gradient is small enough, the change in the objective function value is very small, etc.). If the convergence conditions are met, the iteration stops and the optimal solution is returned; otherwise, the iteration continues. A numerical optimization method is used to find the optimal weight vector and save it in the target database for subsequent use. This process is a one-time task, and the saved weight vector file can be used continuously.

[0085] (4) Based on the target weight combination and the embedding vectors of the multiple clauses, obtain the weighted average embedding vectors of each sliding window of the document to be processed.

[0086] Optionally, the above step (4) can be implemented in the following manner:

[0087] Obtain an embedding vector matrix composed of the embedding vectors of multiple clauses of any sliding window;

[0088] Perform a dot product operation on the target weight combination and the embedding vector matrix to obtain the weighted average embedding vector of any sliding window.

[0089] Specifically, each clause within the sliding window has a corresponding embedding vector. These vectors can form a matrix. By performing a dot product operation on the target weight combination and the embedding vector matrix, a new vector is obtained. This new vector is the weighted average of all the word embedding vectors within the sliding window, representing the semantic information of the entire sliding window. After the user selects the size of the sliding window, the dot product operation is performed on the target weight combination and the embedding vector matrix corresponding to the clauses within the sliding window to obtain the weighted average embedding vector of the current sliding window. The window is then slid sequentially to obtain the weighted average embedding vectors of all the sliding windows in the entire document to be processed.

[0090] S12. Obtain the similarity array of the document to be processed.

[0091] Among them, the similarity array includes: the similarities between the weighted average embedding vectors of each pair of adjacent sliding windows.

[0092] Specifically, the similarity array of the document to be processed is obtained by calculating the similarities between the weighted average embedding vectors of each pair of adjacent sliding windows.

[0093] Exemplarily, the cosine similarity method can be used to calculate the similarities between the weighted average embedding vectors of each pair of adjacent sliding windows. The similarity between two vectors is measured by calculating the cosine value of the angle between them. The value range of the cosine similarity is [-1, 1]. The closer this value is to 1, the more similar the vectors are. Additionally, the Euclidean distance can be used to measure the straight-line distance between two vectors, or other reasonable similarity calculation methods can be used to measure the similarity between two vectors.

[0094] S13. Adjust the similarity array based on the target coefficient of variation to obtain the target similarity array.

[0095] Among them, the target coefficient of variation is specified by the user to make the similarity fluctuation level of the basis for dividing document paragraphs in the same batch reach the specified level. For example, in this embodiment, the target coefficient of variation can take the value of 0.35.

[0096] Specifically, assuming there is a document, the similarity array of this document is adjusted based on the target coefficient of variation to obtain a set of target similarity arrays. Assuming there are multiple documents, the similarity arrays of multiple documents are adjusted based on the target coefficient of variation to obtain multiple sets of target similarity arrays.

[0097] In some embodiments, before performing the above step S13 (adjusting the similarity array based on the target coefficient of variation to obtain the target similarity array), the following steps can also be performed:

[0098] Determine the target coefficient of variation based on the coefficient of variation of the similarity array; the value of the target coefficient of variation includes: greater than the coefficient of variation of the similarity array; or; less than the coefficient of variation of the similarity array.

[0099] Specifically, the target coefficient of variation is determined according to the coefficient of variation of the similarity array. When there is at least one similarity array, the target coefficient of variation can be determined according to the coefficient of variation of the at least one similarity array, and the value of the target coefficient of variation includes: greater than the coefficient of variation of the at least one similarity array; or; less than the coefficient of variation of the at least one similarity array.

[0100] Exemplarily, which value to specifically select depends on whether it is desired to increase or decrease the dispersion degree of the similarity array. For example, if it is desired that the similarity values be more concentrated, the target coefficient of variation can be selected to be less than the coefficient of variation of at least one similarity array; conversely, if it is desired that the similarity values be more dispersed, the target coefficient of variation can be selected to be greater than the coefficient of variation of at least one similarity array. According to the determined target coefficient of variation, the similarity calculation method or data processing method can be adjusted to achieve the desired dispersion degree. For example, it can be achieved by adjusting the parameters of the similarity calculation, using different similarity measurement methods, or preprocessing the data (such as standardization, normalization).

[0101] By calculating the coefficient of variation of the similarity array and determining the target coefficient of variation according to actual requirements, the dispersion degree of the similarity values can be better controlled, thereby improving the accuracy and effectiveness of the similarity calculation.

[0102] In some embodiments, before performing the above step S13 (adjust the similarity array based on the target coefficient of variation to obtain the target similarity array), the following steps can also be performed:

[0103] Calculate the standard deviation and mean of the similarity array;

[0104] Calculate the coefficient of variation of the similarity array according to the ratio of the standard deviation to the mean of the similarity array.

[0105] Among them, CV, Coefficient of Variation, that is, the coefficient of variation, is a statistic used to measure the relative volatility of a data set, representing the ratio of the standard deviation to the mean. Since the coefficient of variation is the ratio of the standard deviation to the mean, it is a dimensionless quantity and is suitable for comparing data with different dimensions or scales. The coefficient of variation reflects the degree of fluctuation of the data relative to its mean and is suitable for comparing the stability or dispersion degree of different data sets.

[0106] Specifically, the coefficient of variation is obtained by statistically analyzing the target similarity array. That is, first, add up all the data values in the similarity array, then divide by the number of data points to calculate the average value of the similarity array. Then calculate the square of the difference between each data value and the average value, add up all the squared values, divide by the number of data points, and take the square root to obtain the standard deviation of the similarity array. Finally, divide the standard deviation by the average value and multiply by 100% to obtain the coefficient of variation of the similarity array.

[0107] Exemplarily, initially, assume that two sets of similarity arrays of two documents are obtained, and the data are as follows:

[0108] The first set of data: data1 = [0.912, 0.865, 0.413, 0.675, 0.755, 0.322, 0.876, 0.821, 0.367, 0.688];

[0109] The second set of data: data2 = [0.612, 0.765, 0.513, 0.775, 0.715, 0.522, 0.776, 0.621, 0.867, 0.668].

[0110] Through the above calculation steps, the coefficient of variation of the first set of data is approximately 31.62%, and the coefficient of variation of the second set of data is approximately 16.20%.

[0111] Optionally, the above step S13 can be implemented in the following manner:

[0112] Based on the mean and standard deviation of the similarity array and the target coefficient of variation, calculate each value in the similarity array to obtain the target similarity array.

[0113] Optionally, the above step can be implemented in the following manner:

[0114] Calculate each value corresponding to the similarity array in the target similarity array according to the adjustment formula:

[0115]

[0116] Wherein, represents the adjusted th value in the target similarity array, represents the th value in the similarity array before adjustment, represents the target coefficient of variation, represents the mean of the similarity array, represents the standard deviation of the similarity array.

[0117] Specifically, assuming there are multiple documents, the similarity arrays of the multiple documents are adjusted based on the target coefficient of variation to obtain multiple groups of target similarity arrays. Based on the mean and standard deviation of each similarity array and the target coefficient of variation, each value in each similarity array is calculated to obtain the target similarity array.

[0118] Exemplarily, according to the above formula to adjust the formula, the original data is transformed sequentially with the entire array as a unit. After the transformation of data1 and data2, they are respectively:

[0119] Target similarity array 1: [0.9379, 0.8859, 0.3856, 0.6756, 0.7642, 0.2849, 0.8981, 0.8372, 0.3347, 0.6900];

[0120] Target similarity array 2: [0.5291, 0.8597, 0.3152, 0.8813, 0.7517, 0.3347, 0.8835, 0.5486, 1.0801, 0.6501].

[0121] Through the above coefficient of variation calculation steps, the coefficient of variation of the first group of data is approximately 35.01%, and the coefficient of variation of the second group of data is approximately 35.01%.

[0122] S14. Determine the dynamic similarity threshold according to the target coefficient of variation and the segmentation information.

[0123] Among them, the segmentation information is used to indicate the range of the number of clauses included in the segment. The segmentation information can be understood as the user's segmentation preference value, and the user's segmentation preference value reflects the preference degree of different document segmentations. For example, the user's segmentation preference value can be large paragraph segmentation, medium paragraph segmentation, and small paragraph segmentation. Among them, large paragraph segmentation, medium paragraph segmentation, and small paragraph segmentation can be represented by 0, 1, and 2 respectively. Regarding the specific division of large paragraph segmentation, medium paragraph segmentation, and small paragraph segmentation, it can be determined by setting the range of the number of clauses included in the segment.

[0124] Specifically, determine the dynamic similarity threshold according to the target coefficient of variation and the segmentation information. The dynamic similarity threshold is generated based on the target coefficient of variation and the user's segmentation preference value, and is used to adapt to the semantic distribution of different documents.

[0125] Exemplarily, the dynamic similarity threshold can take a value of 0.5, or other reasonable values.

[0126] S15. Divide the target similarity array according to the dynamic similarity threshold to obtain multiple sub-similarity arrays.

[0127] Specifically, the target similarity array is divided according to the dynamic similarity threshold to obtain multiple sub-similarity arrays.

[0128] Exemplarily, taking the adjusted target similarity array 1 [0.9379, 0.8859, 0.3856, 0.6756, 0.7642, 0.2849, 0.8981, 0.8372, 0.3347, 0.6900] as an example for division, and dividing according to the dynamic similarity threshold of 0.5, we get: Group 1: [0.9379, 0.8859]; Group 2: [0.3856, 0.6756, 0.7642]; Group 3: [0.2849, 0.8981, 0.8372]; Group 4: [0.3347, 0.6900].

[0129] S16. Respectively determine the clauses corresponding to each sub-similarity array as the segments of the document to be processed.

[0130] Specifically, segment according to the text sentence order corresponding to the grouped data. That is, respectively determine the clauses corresponding to each sub-similarity array as the segments of the document to be processed.

[0131] Combining the dynamic similarity threshold and the dynamic clause length vector encoding solves the deficiencies of the fixed threshold and fixed window size in the traditional method, and significantly improves the flexibility and accuracy of semantic chunking.

[0132] The text semantic chunking method provided by the embodiments of the present disclosure obtains the weighted average embedding vectors of each sliding window corresponding to the document to be processed, where the weighted average embedding vector of any sliding window is obtained by weighted summing the embedding vectors of each clause in the document to be processed within the window, obtains the similarity array of the document to be processed, and the similarity array includes: the similarity between the weighted average embedding vectors of each adjacent sliding window, adjusts the similarity array based on the target coefficient of variation to obtain the target similarity array, determines the dynamic similarity threshold according to the target coefficient of variation and the segmentation information, where the segmentation information is used to indicate the range of the number of clauses included in the segmentation, divides the target similarity array according to the dynamic similarity threshold to obtain multiple sub-similarity arrays, and respectively determines the clauses corresponding to each sub-similarity array as the segmentation of the document to be processed. By weighted summing the embedding vectors of each clause within the sliding window, the overall semantic features of the clauses within the window can be reflected, the influence of noise can be reduced, calculating the similarity between the weighted average embedding vectors of adjacent sliding windows can more accurately identify the semantically continuous parts in the document, adjusting the similarity array based on the target coefficient of variation makes the segmentation effect of different documents relatively stable, dynamically adjusting the similarity threshold according to the target coefficient of variation and the segmentation information enables the segmentation process to adapt to the structures and contents of different documents, the segmentation information indicates the range of the number of clauses included in the segmentation, making the segmentation result more in line with the actual requirements and avoiding overly long or short segmentations. Dividing the target similarity array according to the dynamic similarity threshold and determining the corresponding clauses as the segmentation of the document to be processed improves the accuracy and rationality of text chunking.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation manner.

[0134] Figure 2 FIG. is a schematic structural diagram of a text semantic chunking device 200 provided by the present disclosure, as Figure 2 shown, the device of this embodiment includes: an acquisition module 210, a calculation module 220, an adjustment module 230, an analysis module 240, a division module 250, and a determination module 260, where

[0135] The acquisition module 210 is configured to obtain the weighted average embedding vectors of each sliding window corresponding to the document to be processed; the weighted average embedding vector of any sliding window is obtained by weighted summing the embedding vectors of each clause in the document to be processed within the window;

[0136] A calculation module 220, configured to obtain a similarity array of the document to be processed; the similarity array includes: similarities between weighted average embedding vectors of each adjacent sliding window;

[0137] An adjustment module 230, configured to adjust the similarity array based on a target coefficient of variation to obtain a target similarity array;

[0138] An analysis module 240, configured to determine a dynamic similarity threshold according to the target coefficient of variation and segmentation information; the segmentation information is used to indicate a range of the number of clauses included in a segment;

[0139] A division module 250, configured to divide the target similarity array according to the dynamic similarity threshold to obtain a plurality of sub-similarity arrays;

[0140] A determination module 260, configured to respectively determine clauses corresponding to each sub-similarity array as segments of the document to be processed.

[0141] As an optional implementation manner of an embodiment of the present disclosure, the obtaining module 210 includes:

[0142] A splitting unit, configured to obtain a document to be processed and split the document to be processed into a plurality of clauses;

[0143] A processing unit, configured to input the plurality of clauses into a target embedding model for embedding processing to obtain embedding vectors of the plurality of clauses;

[0144] An obtaining unit, configured to obtain a target weight combination; the target weight combination includes: weights corresponding to embedding vectors of a plurality of clauses of the sliding window;

[0145] A calculation unit, configured to obtain weighted average embedding vectors of each sliding window of the document to be processed based on the target weight combination and the embedding vectors of the plurality of clauses.

[0146] As an optional implementation manner of an embodiment of the present disclosure, the obtaining unit is specifically configured to:

[0147] Determine whether a weight combination corresponding to the target embedding model and a target window size is included in a target database; the target database includes: a plurality of embedding model identifiers and a plurality of weight combinations corresponding to a plurality of sliding window sizes; the target window size is the size of the sliding window;

[0148] If the target database includes a combined weight corresponding to the target embedding model and the target window size, use the combined weight as the target combined weight;

[0149] If the combination weight corresponding to the target embedding model and the target window size is not included in the target database, obtain a target weight combination based on the target window size and the embedding vectors of the multiple clauses.

[0150] As an optional implementation manner of an embodiment of the present disclosure, the obtaining unit is further specifically configured to:

[0151] Obtain a target embedding vector and an initial weight combination of the multiple clauses; the target embedding vector is a combined embedding vector obtained by combining multiple clauses within the target window into a target clause and then performing embedding processing.

[0152] Optimize the initial weight combination according to the target embedding vector and the embedding vectors of the multiple clauses to obtain a target weight combination.

[0153] As an optional implementation manner of an embodiment of the present disclosure, the obtaining unit is further specifically configured to:

[0154] Combine multiple clauses within the target window into a target clause;

[0155] Perform embedding processing on the target clause according to the target embedding model to obtain a target embedding vector of the target clause;

[0156] Initialize the weights of multiple clauses within the target window to obtain an initial weight combination of the multiple clauses.

[0157] As an optional implementation manner of an embodiment of the present disclosure, the obtaining unit is further specifically configured to:

[0158] Perform weighted average processing on the embedding vectors of multiple clauses within the target window according to the initial weight combination of the multiple clauses to obtain an initial weighted average embedding vector;

[0159] Adjust the initial weight combination according to the initial weighted average embedding vector and the target embedding vector to obtain the current weight combination of the multiple clauses;

[0160] Perform weighted average processing on the embedding vectors of multiple clauses within the target window according to the current weight combination of the multiple clauses to obtain a current weighted average embedding vector;

[0161] When the current weighted average embedding vector and the target embedding vector meet a preset convergence condition, obtain a target weight combination.

[0162] As an optional implementation manner of an embodiment of the present disclosure, the adjusting the initial weight combination according to the initial weighted average embedding vector and the target embedding vector to obtain the current weight combination of the multiple clauses includes:

[0163]

[0164] Among them, represents the value of the objective function, represents the Euclidean distance between the weighted average embedding vector and the target embedding vector, is a constant, represents the current weight of the sub-clause, represents the weighted average embedding vector,

[0165] As an optional implementation manner of an embodiment of the present disclosure, the calculation unit is specifically configured to:

[0166] Obtain an embedding vector matrix composed of embedding vectors of multiple sub-clauses of any sliding window;

[0167] Perform a dot product operation on the target weight combination and the embedding vector matrix to obtain the weighted average embedding vector of any sliding window.

[0168] As an optional implementation manner of an embodiment of the present disclosure, the device further includes a coefficient calculation module, and the coefficient calculation module is used to:

[0169] Calculate the standard deviation and mean of the similarity array;

[0170] Calculate the coefficient of variation of the similarity array according to the ratio of the standard deviation to the mean of the similarity array.

[0171] As an optional implementation manner of an embodiment of the present disclosure, the adjustment module is specifically configured to:

[0172] Based on the mean and standard deviation of the similarity array and the target coefficient of variation, calculate each value in the similarity array to obtain a target similarity array.

[0173] As an optional implementation manner of an embodiment of the present disclosure, the adjustment module is specifically configured to:

[0174] Calculate each value corresponding to the similarity array in the target similarity array according to the adjustment formula:

[0175]

[0176] Among them, represents the adjusted represents the th value before adjustment in the similarity array, represents the target coefficient of variation, represents the mean value of the similarity array, represents the standard deviation of the similarity array.

[0177] For the description of the features in the corresponding embodiment of the text semantic chunking device 200, reference can be made to the relevant description of the corresponding embodiment of the text semantic chunking method, which will not be elaborated here one by one.

[0178] The text semantic chunking device provided by the embodiments of the present disclosure obtains the weighted average embedding vectors of each sliding window corresponding to the document to be processed. Among them, the weighted average embedding vector of any sliding window is obtained by weighted summing the embedding vectors of each clause in the window of the document to be processed. It obtains the similarity array of the document to be processed. The similarity array includes: the similarity between the weighted average embedding vectors of each adjacent sliding window. It adjusts the similarity array based on the target coefficient of variation to obtain the target similarity array. According to the target coefficient of variation and the segmentation information, it determines the dynamic similarity threshold, where the segmentation information is used to indicate the range of the number of clauses included in the segment. It divides the target similarity array according to the dynamic similarity threshold to obtain multiple sub-similarity arrays, and respectively determines the clauses corresponding to each sub-similarity array as the segments of the document to be processed. By weighted summing the embedding vectors of each clause in the sliding window, it can reflect the overall semantic features of the clauses in the window, reduce the influence of noise, calculate the similarity between the weighted average embedding vectors of adjacent sliding windows, and can more accurately identify the semantically continuous parts in the document. Adjusting the similarity array based on the target coefficient of variation makes the segmentation effect of different documents relatively stable. Dynamically adjusting the similarity threshold according to the target coefficient of variation and the segmentation information enables the segmentation process to adapt to the structures and contents of different documents. The segmentation information indicates the range of the number of clauses included in the segment, making the segmentation result more in line with the actual needs and avoiding overly long or short segments. Dividing the target similarity array according to the dynamic similarity threshold and determining the corresponding clauses as the segments of the document to be processed improves the accuracy and rationality of text chunking.

[0179] The embodiments of the present application also provide an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above embodiments of the text semantic chunking method.

[0180] The embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. Among them, the computer program is configured to execute the steps in any of the above embodiments of the text semantic chunking method when running.

[0181] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media that can store computer programs, such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0182] An embodiment of the present application also provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments of the text semantic chunking method are implemented.

[0183] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments of the text semantic chunking method are implemented.

[0184] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0185] The above provides a detailed introduction to a text semantic chunking method provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A text semantic chunking method, characterized in that The method includes: Obtaining weighted average embedding vectors of each sliding window corresponding to the document to be processed; the weighted average embedding vector of any sliding window is obtained by performing weighted summation on the embedding vectors of each clause in the window of the document to be processed; Obtaining a similarity array of the document to be processed; the similarity array includes: similarities between weighted average embedding vectors of each pair of adjacent sliding windows; Calculating the standard deviation and mean of the similarity array; Calculating the coefficient of variation of the similarity array according to the ratio of the standard deviation to the mean of the similarity array; Calculating each value in the similarity array based on the mean and standard deviation of the similarity array and the target coefficient of variation to obtain a target similarity array; Determining a dynamic similarity threshold according to the target coefficient of variation and segmentation information; the segmentation information is used to indicate the range of the number of clauses included in the segment; the dynamic similarity threshold is a reasonable value selected by the user according to the target coefficient of variation and the segmentation information; Dividing the target similarity array according to the dynamic similarity threshold to obtain a plurality of sub-similarity arrays; Respectively determining the clauses corresponding to each sub-similarity array as segments of the document to be processed; The calculating each value in the similarity array based on the mean and standard deviation of the similarity array and the target coefficient of variation to obtain a target similarity array includes: Calculating each value corresponding to the similarity array in the target similarity array according to an adjustment formula; Among them, represents the th adjusted value in the target similarity array, represents the th value in the similarity array before adjustment, represents the target coefficient of variation, represents the mean of the similarity array, represents the standard deviation of the similarity array.

2. The text semantic chunking method according to claim 1, wherein The obtaining weighted average embedding vectors of each sliding window corresponding to the document to be processed includes: Obtaining the document to be processed, and splitting the document to be processed into a plurality of clauses; Inputting the plurality of clauses into a target embedding model for embedding processing to obtain embedding vectors of the plurality of clauses; Obtaining a target weight combination; the target weight combination includes: weights corresponding to the embedding vectors of the plurality of clauses in the sliding window; Based on the target weight combination and the embedding vectors of the plurality of clauses, obtaining weighted average embedding vectors of each sliding window of the document to be processed.

3. The text semantic chunking method according to claim 2, wherein The obtaining the target weight combination includes: Judging whether the target database includes a weight combination corresponding to the target embedding model and the target window size; the target database includes: a plurality of embedding model identifiers and a plurality of weight combinations corresponding to a plurality of sliding window sizes; the target window size is the size of the sliding window; If the target database includes a combined weight corresponding to the target embedding model and the target window size, then using the combined weight as the target combined weight; If the target database does not include a combined weight corresponding to the target embedding model and the target window size, then obtaining a target weight combination based on the target window size and the embedding vectors of the plurality of clauses.

4. The text semantic chunking method according to claim 3, wherein Obtaining a target weight combination based on the target window size and the embedding vectors of the plurality of clauses includes: Obtaining a target embedding vector and an initial weight combination of the plurality of clauses; the target embedding vector is a combined embedding vector obtained by combining a plurality of clauses within the target window into a target clause and then performing embedding processing; Optimizing the initial weight combination according to the target embedding vector and the embedding vectors of the plurality of clauses to obtain a target weight combination.

5. The text semantic chunking method according to claim 4, wherein The obtaining of the target embedding vector and the initial weight combination of the plurality of clauses includes: Combining a plurality of clauses within the target window into a target clause; Performing embedding processing on the target clause according to the target embedding model to obtain a target embedding vector of the target clause; Initializing the weights of a plurality of clauses within the target window to obtain an initial weight combination of the plurality of clauses.

6. The text semantic chunking method according to claim 4, wherein The optimizing the initial weight combination according to the target embedding vector and the embedding vectors of the plurality of clauses to obtain a target weight combination includes: Performing weighted average processing on the embedding vectors of the plurality of clauses within the target window according to the initial weight combination of the plurality of clauses to obtain an initial weighted average embedding vector; Adjusting the initial weight combination according to the initial weighted average embedding vector and the target embedding vector to obtain a current weight combination of the plurality of clauses; Performing weighted average processing on the embedding vectors of the plurality of clauses within the target window according to the current weight combination of the plurality of clauses to obtain a current weighted average embedding vector; When the current weighted average embedding vector and the target embedding vector satisfy a preset convergence condition, obtaining a target weight combination.

7. The text semantic chunking method according to claim 6, wherein The adjusting the initial weight combination according to the initial weighted average embedding vector and the target embedding vector to obtain a current weight combination of the plurality of clauses includes: Among them, represents the value of the objective function, represents the Euclidean distance between the weighted average embedding vector and the target embedding vector, is a constant, represents the current weight of the subclause, represents the weighted average embedding vector, and represents the target embedding vector. The objective function is used to measure the difference between the weighted average embedding vector and the target embedding vector.

8. The text semantic chunking method according to claim 2, wherein The obtaining of the weighted average embedding vectors of each sliding window of the document to be processed based on the target weight combination and the embedding vectors of the plurality of clauses includes: Obtaining an embedding vector matrix composed of the embedding vectors of a plurality of clauses of any sliding window; Performing a dot product operation on the target weight combination and the embedding vector matrix to obtain a weighted average embedding vector of any sliding window.

9. A text semantic chunking device, characterized in that, The device includes: An obtaining module, configured to obtain weighted average embedding vectors of each sliding window corresponding to a document to be processed; the weighted average embedding vector of any sliding window is obtained by performing weighted summation on the embedding vectors of each clause of the document to be processed within the window; A calculating module, configured to obtain a similarity array of the document to be processed; the similarity array includes: similarities between the weighted average embedding vectors of each adjacent sliding window; A coefficient calculation module for calculating the standard deviation and mean of the similarity array; calculating the coefficient of variation of the similarity array according to the ratio of the standard deviation to the mean of the similarity array; An adjustment module for calculating each value in the similarity array based on the mean and standard deviation of the similarity array and a target coefficient of variation to obtain a target similarity array; An analysis module for determining a dynamic similarity threshold according to the target coefficient of variation and segmentation information; the segmentation information is used to indicate the number range of clauses included in the segmentation; A division module for dividing the target similarity array according to the dynamic similarity threshold to obtain a plurality of sub-similarity arrays; A determination module for respectively determining the clauses corresponding to each sub-similarity array as the segments of the document to be processed; The adjustment module is specifically configured to: Calculate each value corresponding to the similarity array in the target similarity array according to an adjustment formula: Among them, represents the th adjusted value in the target similarity array, represents the th value in the similarity array before adjustment, represents the target coefficient of variation, represents the mean of the similarity array, represents the standard deviation of the similarity array.

10. An electronic device, characterized in that, Including: A memory for storing a computer program; A processor for implementing the steps of the text semantic chunking method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that A computer program is stored in the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the text semantic chunking method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that The computer program, when executed by a processor, implements the steps of the text semantic chunking method according to any one of claims 1 to 8.

Images