Intelligent text segmentation method and system based on large language model

Abstract

The invention relates to an intelligent text segmentation method and system based on a large language model, and the method is characterized in that the method comprises the following steps: receiving original text data, cleaning and normalizing the data, and obtaining a preprocessed text; performing deep semantic analysis on the preprocessed text by using a pre-trained large language model; identifying important semantic units in the preprocessed text, and performing text segmentation according to semantic boundaries to obtain text segments; the segmented text segments are converted into vector representation; and the generated vectors are used for semantic matching, and a generation or retrieval task is executed, so that an intelligent word segmentation result better fits the overall semantics and context of a text, and the correlation of word segmentation is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an intelligent text segmentation method and system based on a large language model. Background Art

[0002] In the field of natural language processing (NLP), text segmentation technology is a basic and critical task, which is widely used in many fields such as information retrieval, text summarization, language modeling, and coreference resolution. The purpose of text segmentation is to divide the text into several semantically independent units to better understand the text content and structure.

[0003] Traditional text segmentation methods mainly rely on fixed-length segmentation or sentence boundary-based segmentation strategies. However, these methods have obvious limitations when processing complex texts. They often cannot fully understand the deep semantic content of the text, resulting in inaccurate semantic matching after text vectorization, which in turn affects the performance of downstream tasks. The present invention aims to propose an innovative intelligent text segmentation method by utilizing the deep semantic understanding ability of a large language model, in order to improve the effect of semantic matching in applications such as RAG and enhance the overall performance. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the present invention utilizes a large language model for word segmentation, breaking through the limitations of traditional word segmentation technology and achieving accurate word segmentation with deep semantic understanding and contextual relevance. This method uses deep learning technology to understand the context and semantics of the text, especially when dealing with new vocabulary or ambiguous expressions, showing higher accuracy than traditional rule-based word segmentation. At the same time, the context capture capability of the large language model makes the word segmentation results more consistent with the overall semantics and context of the text, effectively improving the relevance of the word segmentation. To achieve the above objectives, the technical solutions adopted by the present invention include:

[0005] An intelligent text segmentation method based on a large language model, characterized by comprising:

[0006] S1. Input and preprocessing: receiving raw text data, cleaning and normalizing the data to obtain preprocessed text;

[0007] S2, semantic understanding: Use the pre-trained large language model to perform in-depth semantic analysis on the pre-processed text;

[0008] S3, text segmentation: identifying important semantic units in the preprocessed text, and segmenting the text according to semantic boundaries to obtain text fragments;

[0009] S4, vector generation: converting the segmented text segments into vector representations;

[0010] S5. Semantic matching: Use the generated vectors for semantic matching and perform generation or retrieval tasks.

[0011] Specifically, clean the input text data, including removing useless punctuation, numbers, special characters, and performing word segmentation. Split the text into separate words or phrases. You can use natural language processing libraries (such as NLTK, spaCy) or word segmentation tools (such as jieba, StanfordNLP). Select a suitable pre-trained model, and input the pre-processed text into the model according to the task requirements (such as BERT is suitable for understanding tasks, GPT is suitable for generation tasks) to obtain the contextual representation of the text.

[0012] The model will generate a semantic representation of the text based on the context information and understand the deeper meaning of the text. Based on the results of semantic analysis, the model will identify important semantic units in the text: topics, arguments, and evidence. Through the semantic representation output by the model, the model will be used to extract the core topics, arguments, and supporting evidence in the text. At the same time, the model's context information will be used to identify the start and end positions of each semantic unit. The text will be segmented using a custom algorithm based on the identified semantic boundaries, such as using text similarity analysis. If the task is to generate relevant text, the model's generative capabilities will be used to generate text that matches the input vector. If the task is to retrieve relevant documents, the vector will be used for similarity calculation to retrieve the document that is most similar to the input vector from the database.

[0013] Exemplarily, the S1 input and preprocessing includes preprocessing the text using regular word segmentation and statistical word segmentation.

[0014] Specifically, rule-based word segmentation: This method relies on linguistic rules and part-of-speech tagging to determine word boundaries. For example, in Chinese, text can be accurately divided into words by identifying changes in parts of speech (such as nouns, verbs, etc.). This method can improve the accuracy of word segmentation by applying grammatical rules and linguistic knowledge, especially when dealing with structured or grammatically clear texts.

[0015] Statistical word segmentation: This method trains the word segmentation model based on statistical data from a large-scale corpus. By analyzing the co-occurrence frequency and contextual information of words, the statistical word segmentation method optimizes the word segmentation effect. The machine learning model adjusts parameters during the training process to adapt to the text in a specific field, and is often used to process unstructured text or text data in a specific field. In addition to the above-mentioned word segmentation rules, specific dictionaries and rule bases can also be introduced. In specific fields and multi-language scenarios, professional rule bases containing professional terms and common phrases can be introduced, or word segmentation rules designed in multiple languages ​​can be introduced to achieve an adaptive word segmentation strategy.

[0016] Exemplarily, the large language model used in the S2 semantic understanding step is BERT or GPT for understanding tasks.

[0017] Exemplarily, the S3 text segmentation includes but is not limited to segmentation based on recognized semantic unit boundaries or segmentation based on syntactic analysis.

[0018] Exemplarily, the vectorization method used in the S4 vectorization process is Word2Vec or GloVe, which is used to convert text fragments into vectors.

[0019] Specifically, use methods such as TF-IDF (Term Frequency-Inverse Document Frequency) or Word2Vec to convert text into vector form. TF-IDF can reflect the importance of words in the document and their rarity in the entire corpus, while Word2Vec can capture the semantic relationship between words and convert text into a vector of fixed length. This step usually requires selecting an appropriate dimension to balance the richness of features and the complexity of calculation. For each pair of document vectors, calculate the cosine similarity between them. The formula for calculating cosine similarity is:

[0020] Where A and B are the vector representations of the two documents, respectively, and ||A|| and ||B|| are their norms. The similarity between documents is evaluated based on the value of cosine similarity. The value range of cosine similarity is between -1 and 1, and the closer the value is to 1, the more similar the documents are.

[0021] Exemplarily, if the task of the S5 semantic matching is to generate relevant text, the model's generation capability is used to generate text; if the task is to retrieve relevant documents, vectors are used to calculate similarity.

[0022] Specific vector similarity calculations may also include Jaccard similarity: applicable to calculating similarities between sets, and can be used to evaluate the similarity of text sets.

[0023] The method may also include that after the user inputs the query text and converts it into a vector, the system will calculate the cosine similarity between the query vector and each document vector in the database, and sort the search results according to the similarity. A user-friendly interface may also be provided to allow the user to enter a query, display the sorted search results, and provide a document summary or key information to help the user quickly determine the relevance of the document. In order to improve the retrieval efficiency, indexing techniques such as inverted indexing and optimization algorithms such as local sensitive hashing (LSH) may be used to reduce the amount of calculation and increase the retrieval speed.

[0024] In order to support multi-language text similarity retrieval, the system can integrate multi-language word segmentation and vectorization tools to ensure that texts in different languages ​​can be accurately processed. It can also provide a feedback mechanism to allow users to evaluate the relevance of retrieval results, and the system can further optimize the retrieval algorithm based on these feedbacks.

[0025] The present invention also provides an intelligent text segmentation system based on a large language model, which is characterized by comprising:

[0026] Input and preprocessing modules for receiving raw text data, cleaning and normalizing the data;

[0027] Semantic analysis module, used for in-depth semantic analysis;

[0028] Text segmentation module, used to identify semantic units and perform text segmentation;

[0029] A vector generation module, which converts text snippets into vector representations;

[0030] Semantic matching module, used to perform generation or retrieval tasks.

[0031] The present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

[0032] The present invention also provides an electronic device, characterized in that it includes a processor and a memory;

[0033] The memory is used to store computer programs;

[0034] The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

[0035] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.

[0036] The beneficial effects of the present invention are:

[0037] Improve semantic matching accuracy: This paper uses the deep semantic understanding ability of the large language model to break through the limitations of traditional methods that rely on fixed length or sentence structure. This intelligent segmentation method can accurately identify semantic units in the text and achieve more accurate text segmentation, thereby significantly improving the semantic matching accuracy of the text after vectorization.

[0038] Enhanced RAG performance: In the RAG framework, the intelligent text segmentation method of the present invention can provide more semantically consistent text fragments, which is crucial for the retrieval and generation stages. Using the method of the present invention, more relevant and accurate text can be generated to the query, thereby significantly enhancing the performance of the entire RAG system.

[0039] Strong adaptability: The text segmentation method of the present invention is not limited to a specific language or field, and shows excellent adaptability. Due to the generalization ability of the large language model, the present method can be easily applied to text segmentation tasks in different languages ​​and fields without large-scale retraining or adjustment.

[0040] Easy to operate: The present invention simplifies the operation process of text segmentation. Users only need to provide the original text and select a suitable large language model to automatically complete intelligent text segmentation. This process does not require complex parameter adjustment or rule setting, making it easy for non-professionals to use.

[0041] Resource saving: Compared with traditional exhaustive search or iterative optimization methods, the method of the present invention greatly reduces the demand for computing resources. The efficient processing capability of the large language model makes the text segmentation process fast and resource-efficient, thus saving computing resources and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of the intelligent text segmentation method based on a large language model of the present invention.

[0043] Figure 2 The figure is a schematic diagram of the structure of the intelligent text segmentation system based on the large language model of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0046] The first aspect of the present invention relates to a process of steps as follows Figure 1 The intelligent text segmentation method based on the large language model shown includes the following steps:

[0047] S1. Input and preprocessing: receiving raw text data, cleaning and normalizing the data to obtain preprocessed text;

[0048] S2, semantic understanding: Use the pre-trained large language model to perform in-depth semantic analysis on the pre-processed text;

[0049] S3, text segmentation: identifying important semantic units in the preprocessed text, and segmenting the text according to semantic boundaries to obtain text fragments;

[0050] S4, vector generation: converting the segmented text segments into vector representations;

[0051] S5. Semantic matching: Use the generated vectors for semantic matching and perform generation or retrieval tasks.

[0052] This embodiment relates to an intelligent text segmentation technology, which is particularly suitable for the RAG (Retrieval-Augmented Generation) task of the internal knowledge base of CITIC Bank. This technology aims to improve the semantic accuracy of text segmentation by deeply integrating the semantic understanding ability of a large language model, thereby enhancing the semantic matching and retrieval effect in the RAG task.

[0053] Intelligent segmentation technology: The intelligent segmentation method used in this embodiment is different from the traditional method. It performs deep semantic analysis on the text based on a large language model to identify and segment text units that maintain semantic integrity.

[0054] Semantic matching enhancement: Intelligently segmented text fragments are expected to significantly improve the relevance and accuracy of generated text in the retrieval and generation stages of the RAG task.

[0055] Model selection and fine-tuning: Select a suitable pre-trained open source large language model and fine-tune it to suit the specific RAG task requirements.

[0056] Intelligent segmentation algorithm development: embed an algorithm that uses the fine-tuned model output to determine the semantic unit boundaries of the text and achieve intelligent text segmentation, which can be a text similarity analysis algorithm.

[0057] RAG process integration: Integrate the text fragments obtained by intelligent segmentation into the RAG process to optimize the performance of the retrieval and generation stages.

[0058] The technical effects obtained include:

[0059] Improved semantic matching: The semantic matching of intelligently segmented text segments in the RAG task will be improved by about 20%.

[0060] Enhanced quality of generated text: In question-answering or summary generation tasks, the quality of text generated by the RAG model using intelligent segmentation is expected to improve by 15%, mainly manifested in higher information coverage and accuracy.

[0061] Improved user satisfaction: In the simulated user satisfaction survey, user satisfaction will increase to over 90%.

[0062] An embodiment is also provided for calculating the similarity of a text set. Specifically, a text clustering task can be used as an embodiment to show how to use Jaccard similarity to determine the similarity between text sets and perform clustering accordingly.

[0063] The specific implementation steps are as follows: Collect the text datasets that need to be clustered. These data can be news articles, social media posts, academic papers, etc. Preprocess the collected texts, including removing stop words, punctuation, word segmentation (for non-English texts), stemming, etc.

[0064] Convert the text into a feature vector. One approach is to use the Bag of Words (BoW) model, where the text is represented as a multi-set of its words, and the weight of each element can be its frequency of occurrence in the text. In order to calculate the Jaccard similarity, it is necessary to build a vocabulary index that lists all the unique words that appear in the document.

[0065] For each pair of documents, the Jaccard similarity between them is calculated using the following formula: Among them, A and B are the vocabulary sets of two documents, A∩B | is the size of their intersection, | A∪B | is the size of their union.

[0066] Construct a similarity matrix, where each row and column represents a document, and the elements in the matrix are the Jaccard similarities of the corresponding document pairs. Select an appropriate clustering algorithm, such as hierarchical clustering, K-means clustering, etc. Hierarchical clustering does not require the number of clusters to be specified in advance, while K-means clustering does.

[0067] Based on the similarity matrix and the selected clustering algorithm, a clustering process is performed to group the text collection. To evaluate the quality of the clustering results, indicators such as the Silhouette Coefficient can be used, or the rationality of the clustering results can be manually checked.

[0068] The clustering results are visualized, such as using scatter plots, dendrograms, etc., to help users intuitively understand the clustering of texts. Based on the clustering results and evaluation feedback, it may be necessary to return to the feature extraction step, adjust the vocabulary weights, or select different feature representation methods to optimize the clustering effect. Through the above steps, the text clustering task can effectively group large amounts of text data according to content similarity, help users quickly identify patterns and topics in text collections, and provide support for information management and knowledge discovery.

[0069] The intelligent text segmentation technology of this embodiment is expected to bring significant performance improvements to RAG tasks, especially in scenarios that require fine semantic understanding. By achieving more accurate semantic matching, this technology can not only improve the relevance of text generation, but also significantly improve user satisfaction, thereby playing a key role in various natural language processing (NLP) applications.

[0070] Another aspect of the present invention also relates to an intelligent text segmentation method system based on a large language model, the structure of which is as follows: Figure 2 As shown, including:

[0071] Input and preprocessing modules for receiving raw text data, cleaning and normalizing the data;

[0072] Semantic analysis module, used for in-depth semantic analysis;

[0073] Text segmentation module, used to identify semantic units and perform text segmentation;

[0074] A vector generation module, which converts text snippets into vector representations;

[0075] Semantic matching module, used to perform generation or retrieval tasks.

[0076] By using this system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.

[0077] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0078] An embodiment of the present invention also provides an electronic device for executing the above method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores data and related computer programs required for executing the method, and the processor calls the data and programs in the memory to execute all steps of the implementation method and obtains the corresponding technical effect.

[0079] Preferably, the electronic device may include a bus architecture, the bus may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same component, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor when performing operations.

[0080] In addition, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operation control) used may include a microprocessor or other processor device and / or a logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices, which may store the above-mentioned related data information, and may also store a program for executing related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it may be a key or a touch input device; the power supply is used to provide power to the electronic device; the display is used to display display objects such as images and text, for example, it may be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processor, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0081] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0083] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0085] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An intelligent text segmentation method based on a large language model, characterized in that: The following steps are involved: S1. Input and preprocessing: receiving raw text data, cleaning and normalizing the data to obtain preprocessed text; S2, semantic understanding: Use the pre-trained large language model to perform in-depth semantic analysis on the pre-processed text; S3, text segmentation: identifying important semantic units in the preprocessed text, and segmenting the text according to semantic boundaries to obtain text fragments; S4, vector generation: converting the segmented text segments into vector representations; S5. Semantic matching: Use the generated vectors for semantic matching and perform generation or retrieval tasks.

2. The method according to claim 1, characterized in that The S1 input and preprocessing includes preprocessing the text using regular word segmentation and statistical word segmentation.

3. The method according to claim 1, characterized in that The large language model used in the S2 semantic understanding step is BERT or GPT for understanding tasks.

4. The method according to claim 1, characterized in that The S3 text segmentation includes but is not limited to segmentation based on recognized semantic unit boundaries or segmentation based on syntactic analysis.

5. The method according to claim 1, characterized in that The vectorization method used in the S4 vectorization process is Word2Vec or GloVe, which is used to convert text fragments into vectors.

6. The method according to any one of claims 1 to 5, characterized in that If the task of the S5 semantic matching is to generate relevant text, the model's generation capability is used to generate text; if the task is to retrieve relevant documents, vectors are used to calculate similarity.

7. An intelligent text segmentation system based on a large language model, characterized in that: include: Input and preprocessing modules for receiving raw text data, cleaning and normalizing the data; Semantic analysis module, used for in-depth semantic analysis; Text segmentation module, used to identify semantic units and perform text segmentation; A vector generation module, which converts text snippets into vector representations; Semantic matching module, used to perform generation or retrieval tasks.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images