Text segmentation processing method and device for large language model, equipment and storage medium

Abstract

The invention provides a text segmentation processing method and device for a large language model, equipment and a storage medium, and relates to the technical field of retrieval enhancement. The method comprises the following steps: acquiring a text list, generating a plurality of semantic clusters in combination with a preset recursive clustering algorithm, generating a summary text fragment corresponding to each semantic cluster through a preset summary algorithm to form a new text list, and iteratively updating the semantic clusters until the semantic clusters are stable; and reconstructing a preset hierarchical structure according to the summarized text fragment and the text list generated each time, and obtaining hierarchical structure index information corresponding to the preset hierarchical structure so as to be used for retrieval of different granularities of a retrieval enhancement system. According to the method, the obtained text fragments are subjected to recursive clustering, the summarized text fragments of each iteration are summarized and obtained and are stored according to the preset hierarchical structure, and retrieval of different granularities is realized based on the summarized text fragments, so that the semantic coherence of the text after segmentation processing of the retrieval enhancement generation system is improved, and the layering sense of the output text is enhanced; and the output performance is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of retrieval enhancement, and particularly relates to a text segmentation processing method and device of a large language model, equipment and a storage medium. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) system is a technical architecture combining information retrieval and large language model (LLM), which retrieves relevant information fragments from an external knowledge base and provides them as context to LLM to enhance the accuracy, factual consistency and explainability of the content generated by LLM. The output performance of the RAG system is highly dependent on the quality of the pre-segmented text.

[0003] The current text segmentation scheme of the retrieval augmented generation system is usually based on fixed-length sliding window segmentation processing, simple paragraph segmentation processing or punctuation-based segmentation processing of long texts, and then the long texts are uniformly divided into several text segments of similar size for subsequent vectorization processing and storage.

[0004] However, the current text segmentation scheme of the retrieval augmented generation system does not usually pay attention to the deep semantic structure and logical coherence of the text during text segmentation processing, which easily destroys a complete argumentation process or concept description into fragmented information, and the output text segmentation processing result is flat, which cannot reflect the inherent hierarchical structure of the original text such as chapters and paragraphs, thereby leading to poor semantic coherence of the segmented text and difficulty in adapting to different granularity query requirements. SUMMARY

[0005] The main purpose of the present application is to provide a text segmentation processing method, device, equipment and storage medium of a large language model, which aims to improve the semantic coherence of the segmented text of the retrieval augmented generation system and enhance the hierarchical feeling of the output text, and optimize the output performance of the retrieval augmented generation system.

[0006] In a first aspect, the present application provides a text segmentation processing method of a large language model, comprising: obtaining a text list containing a plurality of text segments; wherein each text segment comprises text content and corresponding vector data; using a preset recursive clustering algorithm to generate a plurality of semantic clusters according to each vector data in the text list, wherein each semantic cluster comprises at least one text segment corresponding to the vector data; generating a summary text segment corresponding to each semantic cluster according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, the summary text segment constituting a new text list, and iteratively updating the semantic cluster by using the new text list until the semantic cluster is stable. reconstructing a preset hierarchy according to the summary text segment generated each time and the text list, and obtaining hierarchy index information corresponding to the preset hierarchy, the hierarchy index information being used to retrieve different granularities of the retrieval system.

[0007] In an optional implementation, the generating a plurality of semantic clusters according to each vector data in the text list includes: performing dimension reduction on each vector data in the text list by using a preset dimension reduction algorithm, obtaining a plurality of reduced dimension vector data, and constructing a to-be-clustered vector set according to the plurality of reduced dimension vector data. generating a plurality of semantic clusters according to the to-be-clustered vector set and a preset clustering algorithm, each semantic cluster including at least one text segment corresponding to a reduced dimension vector data.

[0008] In an optional implementation, the preset clustering algorithm includes a preset clustering strategy, and the generating a plurality of semantic clusters according to the to-be-clustered vector set and a preset clustering algorithm includes: determining a plurality of to-be-clustered data points according to the to-be-clustered vector set, wherein each to-be-clustered data point corresponds to a reduced dimension vector data; generating a corresponding number of semantic clusters according to a preset number of semantic clusters, and assigning at least one to-be-clustered data point to each semantic cluster according to the preset clustering strategy, wherein the preset clustering strategy is a distribution strategy that minimizes the sum of squares of distances from all to-be-clustered data points to cluster centers of corresponding semantic clusters.

[0009] In an optional implementation, the iteratively updating the semantic cluster by using the new text list includes: when the to-be-clustered data points corresponding to the semantic cluster are updated, the cluster center of the semantic cluster is updated correspondingly, and a new semantic cluster is obtained, wherein the cluster center of the semantic cluster is a mean vector of the to-be-clustered data points corresponding to the semantic cluster.

[0010] In an optional implementation, before the generating a summary text segment corresponding to each semantic cluster according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, the method further includes: truncate each of the text contents as a plurality of truncated text contents according to a preset text length, and connect the plurality of truncated text contents corresponding to the same text content through a preset connector to obtain clustered text contents; obtain a preset summary algorithm calling instruction according to the clustered text contents and a preset prompt word algorithm; the preset summary algorithm comprises a text abstraction summary algorithm and a vector generation algorithm; the generation of the summary text segment corresponding to each of the semantic clusters according to the text content of the text segment in the semantic cluster and the preset summary algorithm comprises: according to the preset summary algorithm calling instruction, calling the text abstraction summary algorithm to generate corresponding summary text contents according to the clustered text contents; generating a summary vector data corresponding to the summary text content according to the summary text content and the vector generation algorithm; generating a summary text segment corresponding to each of the semantic clusters according to the summary text content and the corresponding summary vector data.

[0011] In an optional implementation, before the obtaining of the text list, the method further comprises: receiving a document processing request, wherein the document processing request comprises a document identifier, a user identifier and a knowledge base identifier; constructing a database query statement according to the document processing request and a preset query statement construction rule, and querying a target document corresponding to the document identifier from a knowledge base corresponding to the knowledge base identifier through the database query statement; extracting a plurality of text segments in the target document, and constructing the text list according to the text segments.

[0012] In an optional implementation, the preset hierarchical structure is a tree hierarchical structure, the text segments in the text list are leaf nodes of the tree hierarchical structure, the summary text segment corresponding to the semantic cluster obtained by the last iteration update is a root node of the tree hierarchical structure, and the summary text segments corresponding to the semantic clusters obtained by the rest iteration updates are intermediate nodes of the tree hierarchical structure.

[0013] In a second aspect, the present application provides a text segmentation processing device, comprising: an obtaining module configured to obtain a text list comprising a plurality of text segments; wherein each of the text segments comprises text content and corresponding vector data; a generating module configured to generate a plurality of semantic clusters according to each of the vector data in the text list by using a preset recursive clustering algorithm, wherein each of the semantic clusters comprises at least one text segment corresponding to the vector data. an updating module configured to generate a summary text segment corresponding to each semantic cluster according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, the summary text segment constituting a new text list, and configured to update the semantic cluster iteratively using the new text list until the semantic cluster is stable; a structure module configured to reconstruct a preset hierarchy according to the summary text segment generated each time and the text list, and to obtain hierarchy index information corresponding to the preset hierarchy, the hierarchy index information being used to retrieve different granularities of the retrieval system.

[0014] In a third aspect, the present application provides an electronic device, comprising a processor, a storage medium and a bus, the storage medium storing machine readable instructions executable by the processor, the processor and the storage medium communicating through the bus when the electronic device is running, and the processor executing the machine readable instructions to perform the steps of the method according to any one of the preceding embodiments.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to perform the steps of the method according to any one of the preceding embodiments.

[0016] The present application has the following beneficial effects: The text segmentation processing method for a large language model provided in this application includes: obtaining a text list containing multiple text fragments; wherein each text fragment includes: text content and corresponding vector data; using a preset recursive clustering algorithm, generating multiple semantic clusters based on each vector data in the text list, wherein each semantic cluster includes: at least one text fragment corresponding to vector data; generating a summary text fragment corresponding to each semantic cluster based on the text content of the text fragments in the semantic cluster and a preset summarization algorithm, wherein the summary text fragments constitute a new text list, and iteratively updating the semantic clusters using the new text list until the semantic clusters are stable; reconstructing a preset hierarchical structure based on the summarization text fragments and the text list generated each time, and obtaining hierarchical structure index information corresponding to the preset hierarchical structure, wherein the hierarchical structure index information is used for retrieval enhancement system retrieval at different granularities. In this embodiment, multiple semantic clusters are generated by recursively clustering the text fragments in the obtained text list. The text content of the text fragments in each semantic cluster is summarized by a preset summarization algorithm to obtain the summary text fragments corresponding to each semantic cluster. The summary text fragments are used to form a new text list for iteratively updating the semantic clusters. After the recursive clustering is completed, the summary text fragments obtained in each iteration and the text fragments in the text list are stored according to a preset hierarchical structure, and the hierarchical structure index information corresponding to the preset hierarchical structure is obtained. This enables the retrieval enhancement system to perform retrieval at different granularities through the hierarchical structure index information, thereby improving the semantic coherence of the text after segmentation processing by the retrieval enhancement generation system and enhancing the hierarchical sense of the output text. This optimizes the output performance of the retrieval enhancement generation system and provides a solid foundation for accurate retrieval and high-quality generation of the Large Language Model (LLM). Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of a text segmentation method for a large language model provided in an embodiment of this application; Figure 2 A schematic flowchart of a text segmentation processing method for a large language model provided in another embodiment of this application; Figure 3 A schematic flowchart of a text segmentation processing method for a large language model provided in another embodiment of this application; Figure 4 A recursive clustering iteration complete flow diagram is provided for an embodiment of the present application. Figure 5 A structure diagram of a text segmentation processing device is provided for an embodiment of the present application. Figure 6 A structure diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0021] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, thus, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. The term “comprises”, “comprising”, or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without more limitations, the element defined by the statement “comprises a” does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

[0022] The following will be a detailed description of some embodiments of the present application in combination with the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0023] The current text segmentation scheme of retrieval enhancement generation system is usually fixed-length sliding window segmentation processing, simple paragraph segmentation processing or punctuation-based segmentation processing on long text, and then the long text is uniformly divided into several text segments with similar size for subsequent vectorization processing and storage.

[0024] However, with the current text segmentation scheme of the retrieval enhancement generation system, the text segmentation process usually does not pay attention to the deep semantic structure and logical coherence within the text. The segmentation processing mode based on a fixed-length sliding window segmentation processing, simple paragraph segmentation processing or punctuation symbol segmentation processing easily breaks the complete argumentation process or concept description corresponding to a long text into fragmented information. There may be a large difference between such fragmented information and the ideas or meanings expressed in the long text. The content generated by the large language model based on the fragmented information may not be accurate or even seriously deviate from the central idea of the long text. Moreover, the text segmentation processing result output by the current text segmentation scheme of the retrieval enhancement generation system is flat and cannot reflect the inherent hierarchical structure of the original text such as chapters and paragraphs. Therefore, the semantic coherence of the segmented text is poor, and it is difficult to adapt to different granularity query requirements.

[0025] To solve the above problems, the main purpose of the present application is to provide a large language model text segmentation processing method, device, equipment and storage medium, which aims to improve the semantic coherence of the segmented text of the retrieval enhancement generation system and enhance the hierarchical feeling of the output text, and optimize the output performance of the retrieval enhancement generation system.

[0026] Figure 1 The large language model text segmentation processing method flowchart provided by an embodiment of the present application, for example, the execution subject of the method can be a computer, a server or other device with computing processing capability, but is not limited thereto, for example, as shown in the figure, the method comprises: Figure 1 S101, obtaining a text list containing a plurality of text segments; wherein each text segment comprises: text content and corresponding vector data.

[0027] For example, the above-mentioned text list can be a text segment set. Assuming that each of the above-mentioned text segments can be represented as (content, vector), wherein content represents text content, and vector represents corresponding vector data, the above-mentioned text list can be represented as chunks=[(content1, vector1), (content2, vector2), (content3, vector3)……], wherein the above-mentioned chunks are the above-mentioned text list, the above-mentioned content1, content2, content3, etc. are the text content in different text segments, and the above-mentioned vector1, vector2, vector3, etc. are the vector data corresponding to the above-mentioned text content content1, content2, content3, etc. respectively.

[0028] ​Of course, the above is only one possible representation of the text list, text segment, text content, and corresponding vector data, and the actual representation can be different from the above example, which is not limited herein.

[0029] S102, using a preset recursive clustering algorithm, generating a plurality of semantic clusters according to each of the above vector data in the above text list, wherein each of the above semantic clusters comprises: at least one vector data corresponding to the above text segment.

[0030] For example, the specific number of the above semantic clusters can be obtained by evaluating the model quality of the model corresponding to different clustering numbers through the Bayesian information criterion, and then determining the optimal clustering number under the current recursive iteration round according to the model quality, that is, the specific number of the above semantic clusters. It can be understood that the optimal clustering number under each recursive iteration round can be the same or different, but it can be understood that the determination method of the number of the above semantic clusters is not limited to the method of obtaining the model quality of the model corresponding to different clustering numbers through the Bayesian information criterion, and then determining the optimal clustering number under the current recursive iteration round according to the model quality.

[0031] S103, generating a summary text segment corresponding to each of the above semantic clusters according to the text content of the above text segment in the above semantic cluster and a preset summarization algorithm, and the above summary text segment constitutes a new text list, and the above semantic cluster is iteratively updated using the above new text list until the above semantic cluster is stable.

[0032] For example, the above generating a summary text segment corresponding to each of the above semantic clusters according to the text content of the above text segment in the above semantic cluster and a preset summarization algorithm can be, for example, referring to the text content of the text segment corresponding to the vector data in each semantic cluster to form a string, and summarizing the formed string through a preset summarization algorithm to generate corresponding summary text content and vector data corresponding to the summary text content. A pair of summary text content and corresponding vector data constitutes one of the above summary text segments. The summary text segment can be similar to the above text segment representation, for example, the summary text segment can be represented as (content', vector'), wherein the above content' is the above summary text content, and the above vector' is the vector data corresponding to the above summary text content. It can be understood that one summary text segment corresponds to one semantic cluster.

[0033] The above summary text segment constitutes a new text list, for example, the summary text segment obtained after each round of recursive iteration can constitute a new text list. The new text list can be similar to the above text segment representation, for example, the new text list can be represented as chunks'=[(content1', vector1'), (content2', vector2'), (content3', vector3')……], wherein, the above chunks' is the new text list, the above content1', content2', content3', etc. is the summary text content in different summary text segments, and the above vector1', vector2', vector3', etc. is the corresponding vector data of the above summary text content content1', content2', content3', etc.

[0034] Of course, the above content is only one possible representation of the new text list, the summary text segment, the summary text content, and the corresponding vector data. The actual representation can be different from the above example content, which is not limited herein.

[0035] The above semantic cluster is updated iteratively using the above new text list, for example, the preset recursive clustering algorithm can be used again to generate a plurality of semantic clusters according to each of the above vector data in the above new text list, wherein each of the above semantic clusters includes: at least one vector data corresponding to the above summary text segment, wherein the new text list used in each recursive iteration round is a new text list constituted by the summary text segment obtained after the completion of the last recursive iteration round.

[0036] The above semantic cluster is stable, for example, after a preset number of recursive iterations, for example, after 5 recursive iterations, the semantic cluster is considered stable; or, the semantic cluster is stable, which means that the content in the generated semantic cluster no longer changes after an indefinite number of recursive iterations. When the semantic cluster is stable, the recursive clustering stops and no longer performs recursive iteration. Of course, the specific method for determining the stability of the semantic cluster can be adjusted and determined according to the actual situation, which is not limited herein.

[0037] S104, reconstructing a preset hierarchical structure according to each generated summary text segment and the above text list, and obtaining hierarchical structure index information corresponding to the above preset hierarchical structure, wherein the hierarchical structure index information is used for retrieving different granularities of the retrieval system.

[0038] For example, assuming that the text list chunks=[(content1, vector1), (content2, vector2), (content3, vector3) …] is used for the first round of recursive iteration, the new text list obtained after the first round of recursive iteration is chunks'=[(content1', vector1'), (content2', vector2'), (content3', vector3') …], the new text list obtained after the first round of recursive iteration using the new text list chunks' is chunks''=[(content1'', vector1''), (content2'', vector2''), (content3'', vector3'') …], and so on until the semantic clusters are stable and the recursive clustering stops, and no further recursive iteration is performed. Then, according to the text list chunks and the new text list chunks', chunks'', etc. containing the summary text segments, a preset hierarchy is reconstructed, and each text list / new text list corresponds to a layer in the preset hierarchy, but this is not a limitation.

[0039] The text segmentation processing method of the large language model provided in the embodiments comprises: obtaining a text list comprising a plurality of text segments. Each text segment comprises text content and corresponding vector data. A preset recursive clustering algorithm is used to generate a plurality of semantic clusters according to each vector data in the text list, wherein each semantic cluster comprises at least one text segment corresponding to the vector data. A summary text segment corresponding to each semantic cluster is generated according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, and the summary text segment constitutes a new text list. The semantic clusters are iteratively updated using the new text list until the semantic clusters are stable. A preset hierarchical structure is reconstructed according to each generated summary text segment and the text list, and hierarchical structure index information corresponding to the preset hierarchical structure is obtained. The hierarchical structure index information is used for searching of different granularities of a search enhancement system. In the embodiments, the text segments in the obtained text list are recursively clustered to generate a plurality of semantic clusters, and the text content of the text segments in each semantic cluster is summarized to obtain a summary text segment corresponding to each semantic cluster by using a preset summarization algorithm. The summary text segments are used to constitute a new text list for iteratively updating the semantic clusters. After recursive clustering is completed, the summary text segments obtained in each iteration are stored according to a preset hierarchical structure together with the text segments in the text list, and hierarchical structure index information corresponding to the preset hierarchical structure is obtained. The search enhancement system can realize search of different granularities through the hierarchical structure index information, thereby improving the semantic coherence of the segmented text of the search enhancement generation system and enhancing the hierarchical feeling of the output text, optimizing the output performance of the search enhancement generation system, and providing a solid foundation for accurate search and high-quality generation of the large language model.

[0040] Optionally, on the basis of the above Figure 1 The embodiments comprise: The vector data in the text list is dimensionally reduced by using a preset dimension reduction algorithm to obtain a plurality of dimensionally reduced dimension vector data, and a to-be-clustered vector set is constructed according to the plurality of dimension vector data.

[0041] The plurality of semantic clusters are generated according to the to-be-clustered vector set and a preset clustering algorithm, and each semantic cluster comprises at least one text segment corresponding to the dimension vector data.

[0042] For example, if each vector data in the text list is regarded as a set, the set composed of each vector data in the text list can be represented as:

[0043] Wherein, the above Z is the set composed of each of the above vector data in the text list, and the above z i is the i-th vector data in the set Z, such as z1, z2, z3, that is, vector1, vector2, vector3 in the above embodiment, and N is the total amount of vector data in the set Z, i is the minimum of 1 and the maximum of N. The above refers to a real number vector space of D dimensions.

[0044] It can be understood that after at least one round of recursive iteration, a new text list is obtained, and the above dimension reduction is performed on each of the above vector data in the text list by using the preset dimension reduction algorithm, for example, the dimension reduction can be performed on each of the above vector data in the new text list by using the preset dimension reduction algorithm, and at this time, Z1, Z2, Z3 are content1', content2', content3', etc. That is, before the semantic cluster is updated by using the new text list, the dimension reduction is also performed on each of the above vector data in the new text list.

[0045] The above to-be-clustered vector set corresponding to the set Z composed of each of the above vector data in the text list can be represented as, for example:

[0046] Wherein, the above Y is the above to-be-clustered vector set, and the above y i is the i-th dimension-reduced vector data in the to-be-clustered vector set Y. Since the above dimension-reduced vector data is obtained by dimension reduction of the vector data in the text list, the number of dimension-reduced vector data is the same as the number of vector data in the text list, that is, N is also the total amount of dimension-reduced vector data in the to-be-clustered vector set Y, i is the minimum of 1 and the maximum of N, and the above refers to a real number vector space of d dimensions, and d << D, that is, the dimension of the dimension-reduced vector data y i is much smaller than the dimension of the vector data z i in the text list, for example, d=5 and D=50, but not limited thereto, and the specific much smaller standard or degree can be determined according to actual conditions, which is not limited herein.

[0047] The preset dimension reduction algorithm can be, for example, a Zhuiyi-SubManifold Approximation and Projection (zhuiyi-smap) algorithm, which can include configuration parameters such as a neighbor number n neighbors = 15 and a minimum distance parameter min dist = 0.1, but the types and values of the configuration parameters can be adjusted and determined according to actual conditions, and the preset dimension reduction algorithm can further include an optimization objective represented by the following loss function:

[0048] wherein L is the loss function, and the optimization objective of the preset dimension reduction algorithm is to minimize the value of the loss function L. i and j are sample indexes for identifying the vector data and the dimension reduction vector data, p is a distance between the vector data and the dimension reduction vector data, q is a sample similarity in the set Z, and r is a sample similarity in the set Y. ij q is a sample similarity in the set Z, and r is a sample similarity in the set Y. ij The smaller the value of the loss function L, the more similar the semantic proximity relationship between the dimension reduction vector data in the set Y and the vector data in the set Z, that is, the dimension reduction does not significantly destroy the semantic proximity relationship of the vector data in the set Z.

[0049] Further, based on the above embodiment, the preset clustering algorithm includes a preset clustering strategy, and the plurality of semantic clusters are generated according to the set of to-be-clustered vectors and the preset clustering algorithm, including: A plurality of to-be-clustered data points are determined according to the set of to-be-clustered vectors, wherein each to-be-clustered data point corresponds to a dimension reduction vector data.

[0050] According to the preset number of semantic clusters, a corresponding number of semantic clusters are generated, and at least one to-be-clustered data point is assigned to each semantic cluster according to the preset clustering strategy, wherein the preset clustering strategy is a distribution strategy that minimizes the sum of squared distances from all to-be-clustered data points to the cluster center of the corresponding semantic cluster.

[0051] Further, based on the above embodiment, the semantic clusters are iteratively updated using the new text list, including: When the to-be-clustered data points corresponding to the semantic clusters are updated, the cluster center of the semantic cluster is updated, and a new semantic cluster is obtained, wherein the cluster center of the semantic cluster is the mean vector of the to-be-clustered data points corresponding to the semantic cluster.

[0052] For example, the preset clustering strategy can be represented by the following objective function: , wherein, the above J is an objective function, the above C r represents the rth semantic cluster, the rth semantic cluster C r may be represented as a set including all the to-be-clustered data points assigned to the semantic cluster C r , the above μ r represents the cluster center of the semantic cluster C r , that is, the mean vector of the corresponding to-be-clustered data points of the semantic cluster C r . The above k is the number of semantic clusters in the current recursive iteration round.

[0053] After obtaining the to-be-clustered vector set and determining the preset clustering strategy, the above generating a plurality of semantic clusters according to the to-be-clustered vector set and the preset clustering algorithm may be represented as: , wherein, the above label(i) may represent the semantic cluster to which the ith to-be-clustered data point is assigned, for example, label(1)=1, which may represent that the 1st to-be-clustered data point is assigned to the 1st semantic cluster C1, the above arg min is a parameter operator for finding the parameter r when the corresponding expression takes the minimum value, that is, sequentially finding the semantic cluster C r with the smallest sum of squared distances of each to-be-clustered data point to the cluster center, thereby realizing the assignment of each to-be-clustered data point.

[0054] The above cluster center of the semantic cluster is updated when the above to-be-clustered data point corresponding to the semantic cluster is updated, and a new semantic cluster is obtained, for example, the cluster center of the semantic cluster may be updated by the following formula: , Figure 2 The text segmentation processing method flowchart of the large language model provided by another embodiment of the application is shown in FIG. 4, please refer to Figure 2 Optionally, based on the above Figure 1 embodiment, before the above generating a summary text segment corresponding to each semantic cluster according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, the method further comprises: truncating each of the above text content into a plurality of truncated text contents according to a preset text length, and connecting the plurality of truncated text contents corresponding to the same text content through a preset connector to obtain a clustered text content.

[0055] obtaining a preset summarization algorithm calling instruction according to the clustered text content and a preset prompt word algorithm.

[0056] The above preset summarization algorithm comprises a text abstract summarization algorithm and a vector generation algorithm.

[0057] The above-mentioned summary text fragments corresponding to each semantic cluster are generated based on the text content of the text fragments in the semantic clusters and a preset summary algorithm, including: S201. According to the above-mentioned preset summary algorithm call instruction, the above-mentioned text abstraction summary algorithm is called to generate the corresponding summary text content based on the above-mentioned clustered text content.

[0058] For example, the aforementioned text abstraction and summarization algorithm may refer to a large language model, and the corresponding preset summarization algorithm invocation command may be a large language model invocation command, but it is not limited to this. Since large language models and other text abstraction and summarization algorithms may have context length limitations, each segment of the aforementioned text content will be truncated into multiple truncated text contents according to the preset text length to adapt to the context length limitations of large language models and other text abstraction and summarization algorithms. The aforementioned preset connector may be a newline character, but it is not limited to this.

[0059] The above-mentioned text abstraction and summarization algorithm is invoked according to the above-mentioned preset summary algorithm invocation instruction, and the corresponding summary text content is generated based on the above-mentioned clustered text content. For example, it can refer to invoking a large language model or other text abstraction and summarization algorithm to abstract and summarize the clustered text content according to the preset summary algorithm invocation instruction, through asynchronous rate limiter control, to obtain the corresponding summary text content.

[0060] S202. Based on the above summary text content and the above vector generation algorithm, generate summary vector data corresponding to the summary text content.

[0061] For example, the above summary text content and corresponding summary vector data can be in a similar form to the text content and corresponding vector data in the above text fragment.

[0062] S203. Based on the above summary text content and the corresponding summary vector data, generate summary text fragments corresponding to each of the above semantic clusters.

[0063] For example, the form of this summary text fragment could be, for instance, Figure 1 The example uses (content', vector'), but is not limited thereto.

[0064] Figure 3 A flowchart illustrating a text segmentation method for a large language model provided in another embodiment of this application is shown below. Figure 3 As shown, optionally, before obtaining the text list, the method further includes: S301. Receive a document processing request, wherein the document processing request includes: document identifier, user identifier, and knowledge base identifier.

[0065] Exemplarily, the document processing request may be from an upstream system, and the document identifier, the user identifier, and the knowledge base identifier may correspond to a unique document, a unique user, and a unique knowledge base, respectively, that is, according to the document identifier, the user identifier, and the knowledge base identifier, the corresponding document, the user, and the knowledge base to which the document belongs can be determined, and the document processing request may include other information in addition to the document identifier, the user identifier, and the knowledge base identifier, and is not limited to the document identifier, the user identifier, and the knowledge base identifier.

[0066] S302, constructing a database query statement according to the document processing request and the preset query statement, and querying and obtaining a target document corresponding to the document identifier from a knowledge base corresponding to the knowledge base identifier through the database query statement.

[0067] It can be understood that the database query statement may include the document identifier, the user identifier, and the knowledge base identifier, but the specific information included may be determined according to the information in the document processing request, and is not limited to the document identifier, the user identifier, and the knowledge base identifier.

[0068] S303, extracting a plurality of text segments in the target document, and constructing a text list according to the text segments.

[0069] Exemplarily, the text list may refer to Figure 1 chunks = [(content1, vector1), (content2, vector2), (content3, vector3)…] in the embodiment, where (content1, vector1) and the like are the text segments, but the specific form is not limited thereto.

[0070] In addition, on the basis of any of the foregoing embodiments, the preset hierarchical structure is a tree hierarchical structure, the text segments in the text list are leaf nodes of the tree hierarchical structure, the summary text segment corresponding to the semantic cluster obtained by the last iteration update is a root node of the tree hierarchical structure, and the summary text segments corresponding to the semantic clusters obtained by the remaining iteration updates are intermediate nodes of the tree hierarchical structure.

[0071] When the hierarchical structure index information corresponding to the preset hierarchical structure is obtained, the document identifier, the user identifier, the knowledge base identifier, and the like, and the hierarchical information and node types of the text segments / summary text segments in each text list / new text list may also be obtained, and all of them can be obtained and stored together to facilitate subsequent retrieval of the system based on the hierarchical structure index information corresponding to the preset hierarchical structure and the other information.

[0072] To more clearly illustrate the complete workflow of the multiple recursive iterations in the above recursive clustering process, Figure 4 For a schematic diagram of the complete recursive clustering iteration process provided in an embodiment of this application, please refer to... Figure 4 Based on the above embodiments, the recursive clustering iteration process of this application includes: S401, Get the text list.

[0073] For example, if this is the first round of recursive iteration, the above text list can be, for instance, used as a way to... Figure 3 In the embodiment, steps S301-S303 obtain the text list, which is also the initial text list. If it is not the first round of recursive iteration, the above text list may refer to a new text list composed of summarized text fragments in the previous recursive iteration round.

[0074] S402. Dimensionally reduce each of the vector data in the above text list using a preset dimensionality reduction algorithm to obtain multiple dimensionality-reduced vector data, and construct a set of vectors to be clustered based on the multiple dimensionality-reduced vector data.

[0075] S403. Based on the above set of vectors to be clustered, determine multiple data points to be clustered, wherein each of the above data points to be clustered corresponds to one of the above-mentioned dimensionality reduction vector data.

[0076] S404. Generate a corresponding number of semantic clusters according to the preset number of semantic clusters, and allocate at least one data point to be clustered to each semantic cluster according to the preset clustering strategy, wherein the preset clustering strategy is an allocation strategy that minimizes the sum of squared distances from all the data points to be clustered to the cluster center of the corresponding semantic cluster.

[0077] For example, the aforementioned preset clustering strategy may refer to the above... Figure 1 The objective function in the embodiment is not limited to this. If it is the first round of recursive iteration, after allocating at least one of the above-mentioned data to be clustered to each of the above-mentioned semantic clusters, the corresponding number of the above-mentioned semantic clusters are generated according to the preset number of semantic clusters. If it is not the first round of recursive iteration, the above-mentioned allocation of at least one of the above-mentioned data to be clustered to each of the above-mentioned semantic clusters is to update multiple semantic clusters. It can be understood that the number of updated semantic clusters can change or remain unchanged, that is, the number of updated semantic clusters in each round of recursive iteration is not limited to the preset number of semantic clusters.

[0078] S405. According to the preset text length, each of the above-mentioned text contents is truncated into multiple truncated text contents, and multiple truncated text contents corresponding to the same text content are connected by a preset connector to obtain clustered text contents.

[0079] S406, obtaining a preset summarization algorithm calling instruction according to the clustered text content and the preset prompt word algorithm.

[0080] S407, calling the text abstract summarization algorithm according to the preset summarization algorithm calling instruction, and generating corresponding summary text content according to the clustered text content.

[0081] S408, generating a summary vector data corresponding to the summary text content according to the summary text content and the vector generation algorithm.

[0082] For example, the summary text content and the corresponding summary vector data can be in the form of (content', vector') similar to the text content and the corresponding vector data in the text segment of the embodiment. Figure 1

[0083] S409, generating a summary text segment corresponding to each semantic cluster according to the summary text content and the corresponding summary vector data.

[0084] For example, the summary text segment can be in the form of (content', vector') in the embodiment, but is not limited thereto. Figure 1

[0085] S410, the summary text segment constitutes a new text list.

[0086] For example, the summary text segment constitutes a new text list, that is, the new text list includes: a plurality of summary text segments, and each summary text segment includes: summary text content and corresponding vector data.

[0087] S411, determining whether the semantic cluster is stable.

[0088] For example, whether the semantic cluster is stable can be determined directly after a preset number of recursive iterations, for example, 5 recursive iterations. Alternatively, whether the semantic cluster is stable can also mean that the content in the generated semantic cluster no longer changes after a certain number of recursive iterations, and then the semantic cluster is determined to be stable. However, how to determine whether the semantic cluster is stable is not limited to the above example.

[0089] If the semantic cluster is determined to be unstable, the new text list constituted in the step S410 is returned to the step S401 to start a new recursive iteration. If the semantic cluster is stable, the step S412 is executed.

[0090] S412, reconstructing a preset hierarchical structure according to the generated summary text segment and the text list each time.

[0091] ​​For example, the preset hierarchy can be a tree hierarchy, the text segments in the initial text list are leaf nodes of the tree hierarchy, the summary text segment in the new text list obtained in the last round of recursive iteration is a root node of the tree hierarchy, and the summary text segments in the new text list obtained in the other rounds of recursive iteration are intermediate nodes of the tree hierarchy. Of course, the preset hierarchy is not limited to a tree hierarchy.

[0092] S413, obtain hierarchy index information corresponding to the preset hierarchy.

[0093] In obtaining the hierarchy index information corresponding to the preset hierarchy, the document identifier, the user identifier, the knowledge base identifier, and other related information can also be obtained at the same time, as well as the hierarchy information and node type of the text segments in the text list / new text list and the summary text segments, which can be obtained and stored together to facilitate subsequent retrieval of the retrieval enhancement system based on the hierarchy index information corresponding to the preset hierarchy and the other information.

[0094] Figure 5 A structural schematic diagram of a text segmentation processing device provided by an embodiment of the present application is shown in the figure. The text segmentation processing device can execute the text segmentation processing method of the large language model, and the device can be integrated into a computer, a server, or other devices with computing processing capability, as shown in the figure. The device can include: Figure 5 An obtaining module 510 is configured to obtain a text list containing a plurality of text segments. Each text segment includes text content and corresponding vector data.

[0095] A generating module 520 is configured to generate a plurality of semantic clusters according to each vector data in the text list by using a preset recursive clustering algorithm. Each semantic cluster includes at least one text segment corresponding to the vector data.

[0096] An updating module 530 is configured to generate a summary text segment corresponding to each semantic cluster according to the text content of the text segment in the semantic cluster and a preset summarization algorithm. The summary text segment constitutes a new text list, and the semantic cluster is iteratively updated by using the new text list until the semantic cluster is stable.

[0097] A structure module 540 is configured to reconstruct a preset hierarchy according to each generated summary text segment and the text list, and obtain hierarchy index information corresponding to the preset hierarchy. The hierarchy index information is used for different granularity retrieval of a retrieval enhancement system.

[0098] ​The text segmentation processing method of the large language model provided in the embodiments of the present application includes: obtaining a text list containing a plurality of text segments. Each text segment includes text content and corresponding vector data. A preset recursive clustering algorithm is used to generate a plurality of semantic clusters according to each vector data in the text list, wherein each semantic cluster includes at least one text segment corresponding to the vector data. A summary text segment corresponding to each semantic cluster is generated according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, and the summary text segment constitutes a new text list. The semantic clusters are iteratively updated using the new text list until the semantic clusters are stable. A preset hierarchical structure is reconstructed according to each generated summary text segment and the text list, and hierarchical structure index information corresponding to the preset hierarchical structure is obtained. The hierarchical structure index information is used for retrieval of different granularities of a retrieval enhancement system. In the embodiments, the text segments in the obtained text list are recursively clustered to generate a plurality of semantic clusters, and the text content of the text segments in each semantic cluster is summarized by a preset summarization algorithm to obtain a summary text segment corresponding to each semantic cluster. The summary text segments constitute a new text list for iterative updating of the semantic clusters. After recursive clustering, the summary text segments obtained in each iteration are stored according to a preset hierarchical structure together with the text segments in the text list, and hierarchical structure index information corresponding to the preset hierarchical structure is obtained. The retrieval enhancement system can realize retrieval of different granularities through the hierarchical structure index information, thereby improving the semantic coherence of the segmented text of the retrieval enhancement generation system and enhancing the hierarchical feeling of the output text, optimizing the output performance of the retrieval enhancement generation system, and providing a solid foundation for accurate retrieval and high-quality generation of the large language model.

[0099] Optionally, the generation module 520 is specifically configured to perform dimension reduction on each vector data in the text list by a preset dimension reduction algorithm, obtain a plurality of dimension-reduced dimension vector data, and construct a to-be-clustered vector set according to the plurality of dimension-reduced dimension vector data. The plurality of semantic clusters are generated according to the to-be-clustered vector set and a preset clustering algorithm, and each semantic cluster includes at least one text segment corresponding to the dimension-reduced dimension vector data.

[0100] Optionally, the preset clustering algorithm includes a preset clustering strategy.

[0101] The generation module 520 is specifically configured to determine a plurality of to-be-clustered data points according to the to-be-clustered vector set, wherein each to-be-clustered data point corresponds to a dimension-reduced vector data. According to a preset semantic cluster number, a corresponding number of semantic clusters are generated, and at least one to-be-clustered data point is assigned to each semantic cluster according to a preset clustering strategy. The preset clustering strategy is a distribution strategy that minimizes the sum of squared distances from all to-be-clustered data points to the cluster center of the corresponding semantic cluster.

[0102] Optionally, the generation module 520 is specifically configured to update the cluster center of the semantic cluster when the to-be-clustered data point corresponding to the semantic cluster is updated, and obtain a new semantic cluster corresponding to the updated cluster center of the semantic cluster. The cluster center of the semantic cluster is the mean vector of the to-be-clustered data point corresponding to the semantic cluster.

[0103] Optionally, the update module 530 can also be configured to truncate each piece of text content into a plurality of truncated text contents according to a preset text length, and connect a plurality of truncated text contents corresponding to the same piece of text content through a preset connector to obtain clustered text content. According to the clustered text content and a preset prompt word algorithm, a preset summary algorithm calling instruction is obtained.

[0104] The preset summary algorithm includes a text abstraction summary algorithm and a vector generation algorithm.

[0105] The update module 530 is specifically configured to call the text abstraction summary algorithm according to the preset summary algorithm calling instruction, generate corresponding summary text content according to the clustered text content, generate summary vector data corresponding to the summary text content according to the summary text content and the vector generation algorithm, and generate a summary text segment corresponding to each semantic cluster according to the summary text content and the corresponding summary vector data.

[0106] Optionally, the obtaining module 510 can also be configured to receive a document processing request, wherein the document processing request includes a document identifier, a user identifier, and a knowledge base identifier. According to a database query statement construction rule of the document processing request and a preset query statement, a database query statement is constructed, and a target document corresponding to the document identifier is obtained from a knowledge base corresponding to the knowledge base identifier through the database query statement. A plurality of text segments in the target document are extracted, and the text list is constructed according to the text segments.

[0107] Optionally, the preset hierarchy is a tree hierarchy, the text segments in the text list are leaf nodes of the tree hierarchy, the summary text segment corresponding to the semantic cluster obtained by the last iteration update is a root node of the tree hierarchy, and the summary text segments corresponding to the semantic clusters obtained by the remaining iteration updates are intermediate nodes of the tree hierarchy.

[0108] The apparatus is configured to perform the method provided by the foregoing embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0109] Figure 6 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. The electronic device can be a computer, a server, or other device with computing processing capability, such as the device 600 shown in the figure. Figure 6 As shown in the figure, the device 600 includes: A processor 610, a storage medium 620, and a bus 630. The processor 610 and the storage medium 620 are communicatively connected through the bus 630.

[0110] The storage medium 620 stores machine-readable instructions executable by the processor 610. When the electronic device is running, the processor 610 executes the machine-readable instructions to perform the text segmentation processing method of the large language model.

[0111] It should be understood that, Figure 6 The structure shown in the figure is only a structural schematic diagram of the electronic device. The electronic device can further include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 6 The components shown in the figure can be implemented in hardware, software, or a combination thereof. Figure 6 Figure 6 The embodiments of the present application also provide a computer-readable storage medium. The computer-readable medium stores a computer program. The computer program can be executed by a processor to implement the text segmentation processing method of the large language model described in the foregoing method embodiments.

[0112] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable medium stores a computer program. The computer program can be executed by a processor to implement the text segmentation processing method of the large language model described in the foregoing method embodiments.

[0113] ​The computer readable storage medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or any suitable combination of the foregoing. The computer readable storage medium is non-transitory. The computer readable storage medium includes a tangible computer readable medium. The tangible computer readable medium includes a non-transitory computer readable medium. The computer readable storage medium includes a tangible computer readable medium that is not a signal. The computer readable storage medium includes a computer readable storage medium that is not a transitory propagating signal.

[0114] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0115] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0116] If the functions are implemented in the form of software function modules and sold or used as independent exhibits, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of software exhibits, which are stored in a storage medium and include a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0117] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the contents of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for text segmentation of a large language model, the method comprising: The method comprises the following steps: acquiring a text list comprising a plurality of text segments; wherein each of the text segments comprises text content and corresponding vector data; generating a plurality of semantic clusters according to each of the vector data in the text list by using a preset recursive clustering algorithm; wherein each of the semantic clusters comprises at least one text segment corresponding to the vector data; generating a summary text segment corresponding to each of the semantic clusters according to the text content of the text segment in the semantic cluster and a preset summarization algorithm; the summary text segment constitutes a new text list; and the semantic clusters are iteratively updated by using the new text list until the semantic clusters are stable; reconstructing a preset hierarchical structure according to the summary text segment generated each time and the text list, and acquiring hierarchical structure index information corresponding to the preset hierarchical structure; the hierarchical structure index information is used for retrieving different granularities of a retrieval system.

2. The method of claim 1, wherein, The method of generating a plurality of semantic clusters according to each of the vector data in the text list by using a preset recursive clustering algorithm comprises the following steps: dimensionally reducing each of the vector data in the text list by using a preset dimension reduction algorithm, acquiring a plurality of dimensionally reduced vector data, and constructing a to-be-clustered vector set according to the plurality of dimensionally reduced vector data; generating a plurality of the semantic clusters according to the to-be-clustered vector set and a preset clustering algorithm; each of the semantic clusters comprises at least one text segment corresponding to the dimensionally reduced vector data.

3. The method of claim 2, wherein, The preset clustering algorithm comprises a preset clustering strategy; the method of generating a plurality of the semantic clusters according to the to-be-clustered vector set and a preset clustering algorithm comprises the following steps: determining a plurality of to-be-clustered data points according to the to-be-clustered vector set; each of the to-be-clustered data points corresponds to one of the dimensionally reduced vector data; generating a corresponding number of the semantic clusters according to a preset number of semantic clusters, and assigning at least one of the to-be-clustered data points to each of the semantic clusters according to the preset clustering strategy; the preset clustering strategy is a distribution strategy that minimizes the sum of squared distances from all the to-be-clustered data points to the cluster center of the corresponding semantic cluster.

4. The method of claim 3, wherein, The method of iteratively updating the semantic clusters by using the new text list comprises the following steps: when the to-be-clustered data point corresponding to the semantic cluster is updated, the cluster center of the semantic cluster is updated correspondingly, and a new semantic cluster is acquired; the cluster center of the semantic cluster is the mean vector of the to-be-clustered data point corresponding to the semantic cluster.

5. The method of claim 1, wherein, Before the method of generating a summary text segment corresponding to each of the semantic clusters according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, the method further comprises the following steps: truncating each of the text content into a plurality of truncated text contents according to a preset text length, and connecting a plurality of the truncated text contents corresponding to the same text content by using a preset connector to acquire clustered text content; acquiring a preset summarization algorithm calling instruction according to the clustered text content and a preset prompt word algorithm; The preset summarization algorithm comprises a text abstraction summarization algorithm and a vector generation algorithm. The summary text segment corresponding to each semantic cluster is generated according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, and the summary text segment constitutes a new text list. According to the preset summarization algorithm calling instruction, the text abstract summarization algorithm is called to generate corresponding summary text content according to the clustering text content; According to the summary text content and the vector generation algorithm, a summary vector data corresponding to the summary text content is generated; According to the summary text content and the corresponding summary vector data, a summary text segment corresponding to each semantic cluster is generated.

6. The method of claim 1, wherein, Before the text list is obtained, the method further comprises: receiving a document processing request, wherein the document processing request comprises a document identifier, a user identifier, and a knowledge base identifier; constructing a database query statement according to the document processing request and a preset query statement construction rule, and querying a target document corresponding to the document identifier from a knowledge base corresponding to the knowledge base identifier through the database query statement; extracting a plurality of text segments in the target document, and constructing the text list according to the text segments.

7. The method according to any one of claims 1 to 6, characterized in that, The preset hierarchical structure is a tree hierarchical structure, the text segments in the text list are leaf nodes of the tree hierarchical structure, the summary text segment corresponding to the semantic cluster obtained by the last iteration update is a root node of the tree hierarchical structure, and the summary text segments corresponding to the semantic clusters obtained by the remaining iteration updates are intermediate nodes of the tree hierarchical structure.

8. A text segmentation processing apparatus characterized by comprising: comprises: an acquisition module configured to acquire a text list comprising a plurality of text segments, wherein each text segment comprises text content and corresponding vector data; a generation module configured to generate a plurality of semantic clusters according to each vector data in the text list by using a preset recursive clustering algorithm, wherein each semantic cluster comprises at least one text segment corresponding to the vector data; an update module configured to generate a summary text segment corresponding to each semantic cluster according to the text content of the text segment in the semantic cluster and a preset summarization algorithm, wherein the summary text segment constitutes a new text list, and the semantic clusters are iteratively updated by using the new text list until the semantic clusters are stable; a structure module configured to reconstruct a preset hierarchical structure according to the summary text segment generated each time and the text list, and acquire hierarchical structure index information corresponding to the preset hierarchical structure, wherein the hierarchical structure index information is used for retrieving different granularities of a retrieval system.

9. An electronic device, comprising: comprises: a processor, a storage medium, and a bus, the storage medium stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine readable instructions to perform the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the method according to any one of claims 1-7.

Images