Topic clustering text segmentation method, device, medium and program product based on large language model

Abstract

The present invention relates to the field of natural language processing and analysis, and provides a method, device, medium and program product for text segmentation based on a large language model for topic clustering, wherein the method comprises: segmenting unstructured text into paragraphs; extracting topic content from the segmented paragraphs using a large model based on prompt learning; and dynamically segmenting the text based on topic clustering according to the extracted topic content. The present invention extracts topic content from paragraph-level texts through prompt learning combined with a large language model, determines the central topic of the text to be segmented, and divides sentence-level texts using a clustering method with the topic content as the center, thereby forming text blocks with the topic content as the center, which can realize the text segmentation task in the RAG retrieval stage, ensure the semantic integrity of the text segmentation, and improve the accuracy of RAG retrieval.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and analysis, and in particular to a topic clustering text segmentation method, device, medium and program product based on a large language model. Background Art

[0002] At present, the application of large models is developing rapidly, and has a wide range of application scenarios in natural language understanding, natural language generation, etc. In text generation, in order to solve the illusion of model generation results and make the generation results explainable and traceable, (RAG) retrieval-enhanced generation has become one of the mainstream technical methods for large model applications. Its essence is to retrieve and obtain the knowledge set related to the instructions according to the user input instructions, and solve user problems based on the obtained candidate knowledge set in combination with the large model through prompt learning and thinking chain. In the retrieval stage, whether a comprehensive and high-quality knowledge candidate set can be obtained according to the user's instructions will directly affect the quality of the model generation results. The key step of retrieval is to block the unstructured document data and divide the text into text blocks of appropriate size, so as to improve the accuracy of the retrieval stage. Therefore, it is of great research value to establish an efficient and accurate text semantic segmentation method based on unstructured text data.

[0003] Currently, the segmentation strategies for retrieval enhancement generation are mainly divided into rule-based and semantic information-based. The main rule-based methods include recursive character segmentation, fixed length segmentation, fixed character segmentation, etc., but they have certain limitations, including the difficulty in controlling the segmentation granularity, resulting in over-fine segmentation or over-coarse segmentation, strong dependence on delimiters, resulting in critical delimiter selection and sensitivity to text types, difficulty in handling semantic coherence, resulting in semantic breaks and context loss, and poor scalability, resulting in high requirements for text quality. There are model-based text semantic segmentation methods such as Cross-segment-attention and seqModel in text semantic segmentation, but they have problems such as strong data dependence, high model fine-tuning costs, and limited generalization capabilities. Summary of the invention

[0004] In order to segment unstructured document data more accurately and reasonably, the present invention aims to solve the problems of semantic disconnection, context loss, and poor generalization ability of segmentation methods in the text segmentation process, and to segment unstructured document data more accurately and reasonably, the present invention provides a topic clustering text segmentation method, device, medium, and program product based on a large language model. The method uses a clustering method based on the large model to generate topics, divides the unstructured document at the paragraph level, extracts topic content from the paragraph-level text through prompt learning combined with the large language model, and then uses the extracted topic content as the center point to calculate the distance between the sentence level and the center point in the paragraph by clustering. A class of continuous sentences that are close to each center point is used as a segmented cluster, thereby achieving the text segmentation task in the RAG retrieval stage, ensuring the semantic integrity of the text segmentation, and improving the accuracy of the RAG retrieval.

[0005] The present invention provides a topic clustering text segmentation method based on a large language model, comprising:

[0006] Segment unstructured text into paragraphs;

[0007] A large model based on prompt learning is used to extract the topic content from the segmented paragraphs;

[0008] Perform dynamic text segmentation based on topic clustering according to the extracted topic content.

[0009] In some embodiments, the extracting the subject content of the segmented paragraphs using the large model based on prompt learning includes:

[0010] Build a large model topic content extraction prompt template;

[0011] Based on the prompt template, the topic content of each paragraph is extracted.

[0012] In some embodiments, the prompt template is constructed using a chain thinking approach.

[0013] In some embodiments, the extracting the topic content from the segmented paragraphs using the large model based on prompt learning also includes:

[0014] Disambiguate similar topics from the extracted topics.

[0015] In some embodiments, performing similar topic content disambiguation on the extracted topic content includes:

[0016] The bge-m3 word embedding model is used to vectorize the extracted topic content;

[0017] The cosine similarity of the vectorized subject content is calculated to determine whether the extracted subject content has similar conflicts. If the calculated similarity value is greater than the set threshold, the similar subject content is merged.

[0018] In some embodiments, the text dynamic segmentation based on topic clustering according to the extracted topic content includes:

[0019] Perform sentence-level segmentation on the segmented paragraphs;

[0020] Perform text vectorization on the segmented sentences;

[0021] Calculate the distance between each sentence and the topic content;

[0022] Cluster the sentences and body contents according to the calculated distances;

[0023] A recursive calculation method is used to determine in order whether consecutive sentences are closest to a certain topic center point. After a set of matches are calculated, they are removed from the overall candidates until the last sentence is judged.

[0024] In some embodiments, sentences and main contents are clustered using the nearest neighbor principle based on the calculated distances.

[0025] The present invention also provides an electronic device, comprising:

[0026] at least one processor; and a memory communicatively coupled to the at least one processor;

[0027] The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the above method by executing the instructions stored in the memory.

[0028] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store instructions, and when the instructions are executed, the above method is implemented.

[0029] The present invention also provides a computer program product, which, when called by a computer, enables the computer to execute the above method.

[0030] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0031] The present invention extracts subject content from paragraph-level texts through prompt learning combined with a large language model, determines the central theme of the text to be segmented, and divides sentence-level texts by clustering based on the subject content, thereby forming text blocks centered on the subject content. This can realize the text segmentation task in the RAG retrieval stage, ensure the semantic integrity of the text segments, and improve the accuracy of RAG retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a schematic diagram of the overall process of the topic clustering text segmentation method based on a large language model in an embodiment of the present invention.

[0033] Figure 2 Schematic diagram of the flow of the topic clustering algorithm in an embodiment of the present invention.

[0034] Figure 3 Schematic diagram of spatial feature distribution before and after text segmentation in an embodiment of the present invention.

[0035] Figure 4 The present invention is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 This is a schematic diagram of the overall process of the topic clustering text segmentation method based on a large language model. Figure 2 The following is a flow chart of the topic clustering algorithm, which is implemented in the following steps:

[0039] Step S1, segmenting the unstructured text into paragraphs.

[0040] Text segmentation is performed at the sentence level. Usually, paragraph text contains the complete semantic structure of an event. Therefore, in this embodiment, the unstructured text is segmented into paragraphs before the text semantic segmentation is performed. The paragraph segmentation is performed based on paragraph separators (such as: \n) and the like.

[0041] See Table 1, and take the original text paragraph in Table 1 as an example to perform paragraph segmentation.

[0042] Step S2, using a large model based on prompt learning to extract the topic content of the segmented paragraphs.

[0043] In text semantic segmentation, the goal is to determine the correlation between sentence levels, that is, to determine whether two consecutive sentences belong to the same semantic structure. In order to better distinguish the relationship between sentence levels, a large model based on prompt learning is used to extract the topic content of the segmented paragraphs based on event content. Specifically:

[0044] (1) Construct a large model topic content extraction prompt template. The prompt model is constructed using the Chain of Thought (COT) method, using the [task] + {…} + [solution] + {…} + [candidate article] + {…} method;

[0045] (2) Extract the topic content of each paragraph based on the prompt template.

[0046] (3) Disambiguate similar topics from the extracted topics. The bge-m3 word embedding model is used to vectorize the extracted topics. The cosine similarity of the vectorized topics is then calculated to determine whether there is a similarity conflict between the extracted topics. If the calculated similarity value is greater than the set threshold, the similar topics are merged.

[0047] Taking the Qwen2-72B large language model as the base model, the thinking chain prompt method of [task] + {…} + [solution] + {…} + [candidate article] + {…} is adopted to extract the topic content of the segmented paragraphs. The generated results are shown in the topic content part in Table 1. Through manual judgment, it is found that the extracted topic content basically summarizes the original text content.

[0048] Step S3, performing dynamic text segmentation based on topic clustering according to the extracted topic content.

[0049] According to the extracted subject content, the nearest neighbor principle is adopted with the subject content as the center, that is, the text data with the same semantics are close in the feature space. Based on this strategy, the text blocks with the subject content as the cluster center can be segmented. Specifically:

[0050] (1) Sentence-level segmentation of the segmented paragraphs. In some embodiments, the paragraphs are segmented at the sentence level using separators (.;?!) to form a series of continuous text sentence sets.

[0051] (2) Performing text vectorization on the segmented sentences. In some embodiments, in order to calculate the distance between each sentence and the topic content, the bge-m3 word embedding model is used to perform text vectorization on the segmented sentences; in addition, the text vectors can be further normalized if necessary.

[0052] (3) Calculate the distance between each sentence and the topic content. In some embodiments, the distance from the sample point to the cluster center is calculated using the Euclidean distance, expressed as:

[0053]

[0054] Among them, (x 1 ,x 2 ...x n ) and (y 1 ,y 2 ...y n ) represent the vector representation of the sentence and the topic content respectively;

[0055] (4) Clustering the sentences and the subject content according to the calculated distance. In some embodiments, the nearest neighbor principle is used for clustering, specifically, according to the calculated distance between each sentence and the subject content in the feature space, the consecutive sentences closest to the subject content are selected as one class.

[0056] (5) For all recursive calculation methods, determine in order whether the consecutive sentences are closest to one of the topic center points, calculate and match a group of them and remove them from the overall candidates until the last sentence is judged.

[0057] After the above steps, the segmentation results are shown in Table 1. The segmented texts are basically in the same semantic environment through manual judgment, which means that the method based on topic content clustering can segment the text well while ensuring the semantic coherence of the text.

[0058] In order to analyze the effectiveness of the method of the present invention more intuitively and effectively, Figure 3 This is the spatial feature distribution diagram before and after text segmentation in the present invention. Based on the bge-m3 word embedding model, the text data before and after segmentation is vectorized and reduced to the same dimension through principal component analysis (PCA), so that it is mapped in the same three-dimensional space, where the "origin" represents the original sentence, the "asterisk" represents the text block, and the "inverted triangle" represents the cluster center theme content. Figure 3It can be intuitively found that the topic clustering text segmentation method based on the large language model of the present invention can effectively segment the original text into text blocks close to the topic content, thereby realizing the text segmentation task in the RAG retrieval stage, ensuring the semantic integrity of the text blocks, and improving the accuracy of RAG retrieval to recall related texts.

[0059] Table 1, text segmentation example:

[0060]

[0061]

[0062]

[0063] Based on the same technical concept, the embodiment of the present application also provides an electronic device, which can implement the topic clustering text segmentation method based on the large language model provided in the above embodiment of the present application. In one embodiment, the electronic device can be a server, or a terminal device or other electronic device. Figure 4 As shown, the electronic device may include:

[0064] At least one processor, and a memory connected to the at least one processor. The specific connection medium between the processor and the memory is not limited in the embodiment of the present application. Figure 4 The example in this article is that the processor and memory are connected through a bus. Figure 4 The connections between other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into address bus, data bus, control bus, etc. Figure 4 In the diagram, only one thick line is used, but this does not mean that there is only one bus or only one type of bus. Alternatively, a processor can also be called a controller, and there is no limitation on the name.

[0065] In an embodiment of the present application, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can execute a topic clustering text segmentation method based on a large language model discussed above. The processor can implement Figure 4 The functions of each module in the device shown.

[0066] Among them, the processor is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory and calling data stored in the memory, various functions of the device and processing data.

[0067] In an optional design, the processor may include one or more processing units, and the processor may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor. In some embodiments, the processor and the memory may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0068] The processor may be a general-purpose processor, such as a CPU, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of a topic clustering text segmentation method based on a large language model disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0069] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory can include at least one type of storage medium, for example, it can include flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access s Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic storage, disk, optical disk, etc. The memory is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0070] By designing and programming the processor, the code corresponding to the topic clustering text segmentation method based on a large language model introduced in the above embodiment can be fixed into the chip, so that the chip can perform the steps of the topic clustering text segmentation method based on a large language model in the above embodiment when running. How to design and program the processor is a technology well known to those skilled in the art and will not be described here.

[0071] Based on the same inventive concept, an embodiment of the present application also provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes a topic clustering text segmentation method based on a large language model discussed above.

[0072] In some optional embodiments, the present application also provides various aspects of a topic clustering text segmentation method based on a large language model, which can also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of a topic clustering text segmentation method based on a large language model according to various exemplary embodiments of the present application described above in this specification.

[0073] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of a unit described above can be further divided into multiple units to be embodied. In addition, although the operation of the method of the present application is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0074] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a server, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] Program code for performing the operations of the present application may be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0077] Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A topic clustering text segmentation method based on a large language model, characterized in that: include: Segment unstructured text into paragraphs; A large model based on prompt learning is used to extract the topic content from the segmented paragraphs; Perform dynamic text segmentation based on topic clustering according to the extracted topic content.

2. The topic clustering text segmentation method based on a large language model according to claim 1, characterized in that: The method of using a large model based on prompt learning to extract topic content from the segmented paragraphs includes: Build a large model topic content extraction prompt template; Based on the prompt template, the topic content of each paragraph is extracted.

3. The topic clustering text segmentation method based on a large language model according to claim 2 is characterized in that: The prompt template is constructed using a chain thinking approach.

4. The topic clustering text segmentation method based on a large language model according to claim 2 or 3, characterized in that: The method of extracting the subject content of the segmented paragraphs by using a large model based on prompt learning also includes: Disambiguate similar topics from the extracted topics.

5. The topic clustering text segmentation method based on a large language model according to claim 4 is characterized in that: The extracting subject content is subjected to similar subject content disambiguation, including: The bge-m3 word embedding model is used to vectorize the extracted topic content; The cosine similarity of the vectorized subject content is calculated to determine whether the extracted subject content has similar conflicts. If the calculated similarity value is greater than the set threshold, the similar subject content is merged.

6. The topic clustering text segmentation method based on a large language model according to claim 1, characterized in that: The text dynamic segmentation based on topic clustering according to the extracted topic content includes: Perform sentence-level segmentation on the segmented paragraphs; Perform text vectorization on the segmented sentences; Calculate the distance between each sentence and the topic content; Cluster the sentences and body contents according to the calculated distances; A recursive calculation method is used to determine in order whether consecutive sentences are closest to a certain topic center point. After a set of matches are calculated, they are removed from the overall candidates until the last sentence is judged.

7. The topic clustering text segmentation method based on a large language model according to claim 6 is characterized in that: According to the calculated distance, the nearest neighbor principle is used to cluster the sentences and main contents.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method as described in any one of claims 1 to 7 by executing the instructions stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that When the computer program product is called by a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images