A Smart Text Segmentation Method and System Based on a Large Language Model

Abstract

This invention relates to an intelligent text segmentation method and system based on a large language model. The method comprises the following steps: receiving raw text data and cleaning and standardizing the data to obtain preprocessed text; performing deep semantic analysis on the preprocessed text using a pre-trained large language model; identifying important semantic units in the preprocessed text and segmenting the text according to semantic boundaries to obtain text fragments; converting the segmented text fragments into vector representations; and using the generated vectors for semantic matching to perform generation or retrieval tasks. This method enables intelligent word segmentation results to better fit the overall semantics and context of the text, effectively improving the relevance of word segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an intelligent text segmentation method and system based on a large language model. Background Technology

[0002] In the field of Natural Language Processing (NLP), text segmentation is a fundamental and crucial task, widely applied in information retrieval, text summarization, language modeling, and semantic dereference. The purpose of text segmentation is to divide text into several semantically independent units to facilitate a better understanding of the text's content and structure.

[0003] Traditional text segmentation methods primarily rely on fixed-length segmentation or sentence boundary-based segmentation strategies. However, these methods have significant limitations when processing complex text. They often fail to fully understand the deep semantic content of the text, leading to inaccurate semantic matching after text vectorization, which in turn affects the performance of downstream tasks. This invention aims to propose an innovative intelligent text segmentation method by leveraging the deep semantic understanding capabilities of large language models, in order to improve semantic matching performance and enhance overall performance in applications such as RAG. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention utilizes a large language model for word segmentation, overcoming the limitations of traditional word segmentation techniques and achieving accurate word segmentation with deep semantic understanding and contextual relevance. This method, through deep learning technology, can understand the context and semantics of text, exhibiting higher accuracy than traditional rule-based word segmentation, especially when handling novel words or ambiguous expressions. Simultaneously, the context-capturing capability of the large language model makes the segmentation results more closely aligned with the overall semantics and context of the text, effectively improving the relevance of the segmented words. To achieve the above objectives, the technical solution adopted by this invention includes:

[0005] A smart text segmentation method based on a large language model, characterized in that it includes:

[0006] S1. Input and Preprocessing: Receive raw text data, clean and normalize the data to obtain preprocessed text;

[0007] S2. Semantic Understanding: Using a pre-trained large language model to perform deep semantic analysis on the pre-processed text;

[0008] S3. Text segmentation: Identify important semantic units in the preprocessed text and segment the text according to semantic boundaries to obtain text fragments;

[0009] S4. Vector Generation: Convert the segmented text fragments into vector representations;

[0010] S5. Semantic matching: Use the generated vectors to perform semantic matching, and execute generation or retrieval tasks.

[0011] Specifically, the input text data is cleaned, including removing useless punctuation, numbers, and special characters, and performing word segmentation. The text is split into individual words or phrases. Natural language processing libraries (such as NLTK and spaCy) or word segmentation tools (such as jieba and StanfordNLP) can be used. A suitable pre-trained model is selected, and based on task requirements (e.g., BERT is suitable for understanding tasks, and GPT is suitable for generation tasks), the pre-processed text is input into the model to obtain the contextual representation of the text.

[0012] The model generates semantic representations of text based on contextual information, understanding the deeper meaning of the text. Based on the results of semantic analysis, it identifies important semantic units in the text: topic, argument, and supporting evidence. Through the semantic representation output by the model, it extracts the core topic, argument, and supporting evidence from the text. Simultaneously, it uses the model's contextual information to identify the start and end positions of each semantic unit. Based on the identified semantic boundaries, it uses a custom algorithm to segment the text, such as using text similarity analysis. If the task is to generate relevant text, it utilizes the model's generation capabilities to generate text that matches the input vector. If the task is to retrieve relevant documents, it uses vectors to calculate similarity and retrieves the documents most similar to the input vector from the database.

[0013] For example, the S1 input and preprocessing includes preprocessing the text using rule-based word segmentation and statistical word segmentation.

[0014] Specifically, rule-based word segmentation relies on linguistic rules and part-of-speech tagging to determine word boundaries. For example, in Chinese, text can be accurately segmented into words by recognizing changes in part of speech (such as nouns, verbs, etc.). This method improves segmentation accuracy by applying grammatical rules and linguistic knowledge, especially when processing structured or grammatically explicit text.

[0015] Statistical segmentation: This method trains a segmentation model based on statistical data from a large-scale corpus. By analyzing word co-occurrence frequencies and contextual information, statistical segmentation optimizes segmentation performance. Machine learning models adjust parameters during training to adapt to domain-specific text, and are commonly used to process unstructured text or domain-specific text data. In addition to the segmentation rules mentioned above, specific dictionaries and rule bases can be introduced. For specific domains and multilingual scenarios, specialized rule bases containing technical terms and common phrases can be introduced, or segmentation rules designed according to multiple languages ​​can be implemented to achieve adaptive segmentation strategies.

[0016] For example, the large language model used in the S2 semantic understanding step is BERT or GPT for understanding the task.

[0017] For example, the S3 text segmentation includes, but is not limited to, segmentation based on the boundaries of identified semantic units or segmentation based on syntactic analysis.

[0018] For example, the vectorization method used in the S4 vectorization process is Word2Vec or GloVe, which is used to convert text fragments into vectors.

[0019] Specifically, methods such as TF-IDF (Term Frequency-Inverse Document Frequency) or Word2Vec are used to convert the text into vector form. TF-IDF reflects the importance of words in a document and their rarity in the entire corpus, while Word2Vec captures the semantic relationships between words, converting the text into fixed-length vectors. This step typically requires choosing an appropriate dimension to balance feature richness and computational complexity. For each pair of document vectors, their cosine similarity is calculated. The formula for calculating cosine similarity is:

[0020] Here, A and B are the vector representations of the two documents, and ||A|| and ||B|| are their norms. The similarity between documents is assessed based on the cosine similarity value. The cosine similarity value ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity.

[0021] For example, if the task of S5 semantic matching is to generate relevant text, then the model's generation capability is used to generate text; if the task is to retrieve relevant documents, then vectors are used to calculate similarity.

[0022] Specific vector similarity calculations can also include Jaccard similarity: suitable for calculating the similarity between sets, and can be used to evaluate the similarity of text sets.

[0023] This method may also include, after the user inputs query text and converts it into a vector, the system calculates the cosine similarity between the query vector and each document vector in the database, and sorts the search results according to the similarity score. It can also provide a user-friendly interface that allows users to input queries, displays the sorted search results, and provides document summaries or key information to help users quickly determine the relevance of documents. To improve retrieval efficiency, indexing techniques, such as inverted indexes, and optimization algorithms, such as Locality Sensitive Hash (LSH), can be used to reduce computational load and increase retrieval speed.

[0024] To support multilingual text similarity retrieval, the system can integrate multilingual word segmentation and vectorization tools to ensure accurate processing of text in different languages. It can also provide a feedback mechanism, allowing users to evaluate the relevance of search results, and the system can further optimize its retrieval algorithm based on this feedback.

[0025] This invention also provides an intelligent text segmentation system based on a large language model, characterized in that it includes:

[0026] The input and preprocessing module is used to receive raw text data, clean and normalize the data;

[0027] The semantic analysis module is used for in-depth semantic analysis;

[0028] The text segmentation module is used to identify semantic units and perform text segmentation;

[0029] The vector generation module is used to convert text fragments into vector representations;

[0030] The semantic matching module is used to perform generation or retrieval tasks.

[0031] The present invention also provides a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements any one of the methods 1 to 6 above.

[0032] The present invention also provides an electronic device, characterized in that it includes a processor and a memory;

[0033] The memory is used to store computer programs;

[0034] The processor is configured to execute any one of the methods 1 to 6 above by invoking a stored computer program.

[0035] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0036] The beneficial effects of this invention are as follows:

[0037] Improving semantic matching accuracy: This invention leverages the deep semantic understanding capabilities of large language models, overcoming the limitations of traditional methods that rely on fixed lengths or sentence structures. This intelligent segmentation method can accurately identify semantic units in text, achieving more precise text segmentation and thus significantly improving the semantic matching accuracy of the vectorized text.

[0038] Enhancing RAG Performance: Within the RAG framework, the intelligent text segmentation method of this invention provides more semantically accurate text fragments, which is crucial for both the retrieval and generation stages. Using the method of this invention, more relevant and accurate text can be generated, thereby significantly enhancing the performance of the entire RAG system.

[0039] High adaptability: The text segmentation method of this invention is not limited to a specific language or domain, demonstrating excellent adaptability. Due to the generalization ability of large language models, this method can be easily applied to text segmentation tasks in different languages ​​and domains without the need for large-scale retraining or tuning.

[0040] Easy to use: This invention simplifies the text segmentation process. Users only need to provide the original text and select a suitable large language model to automatically complete intelligent text segmentation. This process does not require complex parameter adjustments or rule settings, making it easy for non-professionals to use.

[0041] Resource Saving: Compared to traditional exhaustive search or iterative optimization methods, the method of this invention significantly reduces the demand for computing resources. The efficient processing capabilities of the large language model enable the text segmentation process to be fast and resource-efficient, thereby saving computing resources and time. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the intelligent text segmentation method based on a large language model according to the present invention.

[0043] Figure 2 This is a schematic diagram of the intelligent text segmentation system based on a large language model according to the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0045] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] The first aspect of this invention relates to a process flow as follows: Figure 1 The intelligent text segmentation method based on a large language model, as shown, includes the following steps:

[0047] S1. Input and Preprocessing: Receive raw text data, clean and normalize the data to obtain preprocessed text;

[0048] S2. Semantic Understanding: Using a pre-trained large language model to perform deep semantic analysis on the pre-processed text;

[0049] S3. Text segmentation: Identify important semantic units in the preprocessed text and segment the text according to semantic boundaries to obtain text fragments;

[0050] S4. Vector Generation: Convert the segmented text fragments into vector representations;

[0051] S5. Semantic matching: Use the generated vectors to perform semantic matching, and execute generation or retrieval tasks.

[0052] This embodiment relates to an intelligent text segmentation technology, particularly suitable for the RAG (Retrieval-Augmented Generation) task in CITIC Bank's internal knowledge base. This technology aims to improve the semantic accuracy of text segmentation by deeply integrating the semantic understanding capabilities of a large language model, thereby enhancing the semantic matching and retrieval performance in the RAG task.

[0053] Intelligent segmentation technology: The intelligent segmentation method used in this embodiment is different from the traditional method. It is based on a large language model to perform deep semantic analysis on the text, identify and segment text units that maintain semantic integrity.

[0054] Semantic matching enhancement: Intelligently segmented text fragments are expected to significantly improve the relevance and accuracy of the generated text during the retrieval and generation stages of the RAG task.

[0055] Model selection and fine-tuning: Select a suitable pre-trained open-source large language model and fine-tune it to adapt to the specific RAG task requirements.

[0056] Intelligent text segmentation algorithm development: Embed an algorithm that uses the fine-tuned model output to determine the semantic unit boundaries of the text, thereby achieving intelligent text segmentation. This could be a text similarity analysis algorithm.

[0057] RAG workflow integration: Integrates intelligently segmented text fragments into the RAG workflow to optimize performance in the retrieval and generation stages.

[0058] The technical effects achieved include:

[0059] Improved semantic matching accuracy: The semantic matching accuracy of the intelligently segmented text fragments in the RAG task will be improved by about 20%.

[0060] Enhanced text quality: In question answering or summarizing tasks, the text quality generated by the RAG model with intelligent segmentation is expected to improve by 15%, mainly in terms of higher information coverage and accuracy.

[0061] Increased user satisfaction: In simulated user satisfaction surveys, user satisfaction will increase to over 90%.

[0062] An embodiment is also provided for calculating the similarity of text sets. Specifically, a text clustering task can be used as an example to demonstrate how to use Jaccard similarity to determine the similarity between text sets and to perform clustering accordingly.

[0063] The specific implementation steps are as follows: Collect the text dataset that needs to be clustered. This data can be news articles, social media posts, academic papers, etc. Preprocess the collected text, including removing stop words and punctuation, performing word segmentation (for non-English text), and stemming.

[0064] The text is converted into feature vectors. One approach is to use the Bag of Words (BoW) model, where the text is represented as a multiple set of its words, and the weight of each element can be its frequency of occurrence in the text. To calculate Jaccard similarity, a vocabulary index needs to be built, listing all unique words that appear in all documents.

[0065] For each pair of documents, calculate their Jaccard similarity using the following formula: Here, A and B are the vocabulary sets of two documents, |A∩B| is the size of their intersection, and |A∪B| is the size of their union.

[0066] Construct a similarity matrix where each row and column represents a document, and the elements of the matrix are the Jaccard similarity scores of corresponding document pairs. Choose a suitable clustering algorithm, such as hierarchical clustering or K-means clustering. Hierarchical clustering does not require pre-specifying the number of clusters, while K-means clustering does.

[0067] Based on the similarity matrix and the selected clustering algorithm, a clustering process is performed to group the text sets. The quality of the clustering results can be evaluated using metrics such as the silhouette coefficient, or by manually checking the reasonableness of the clustering results.

[0068] Visualizing the clustering results, such as using scatter plots or dendrograms, helps users intuitively understand the text clustering. Based on the clustering results and evaluation feedback, it may be necessary to return to the feature extraction step, adjust word weights, or select different feature representation methods to optimize the clustering effect. Through the above steps, text clustering tasks can effectively group large amounts of text data according to content similarity, helping users quickly identify patterns and themes in text sets, providing support for information management and knowledge discovery.

[0069] The intelligent text segmentation technology in this embodiment is expected to bring significant performance improvements to RAG tasks, especially in scenarios requiring fine semantic understanding. By achieving more accurate semantic matching, this technology can not only improve the relevance of generated text but also significantly enhance user satisfaction, thus playing a key role in various natural language processing (NLP) applications.

[0070] Another aspect of this invention relates to an intelligent text segmentation method system based on a large language model, the structure of which is as follows: Figure 2 As shown, it includes:

[0071] The input and preprocessing module is used to receive raw text data, clean and normalize the data;

[0072] The semantic analysis module is used for in-depth semantic analysis;

[0073] The text segmentation module is used to identify semantic units and perform text segmentation;

[0074] The vector generation module is used to convert text fragments into vector representations;

[0075] The semantic matching module is used to perform generation or retrieval tasks.

[0076] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0077] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0078] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device includes at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0079] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0080] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0081] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0085] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intelligent text segmentation method based on a large language model, characterized in that, Includes the following steps: S1. Input and Preprocessing: Receive raw text data, clean and normalize the data to obtain preprocessed text; S2. Semantic Understanding: Using a pre-trained large language model to perform deep semantic analysis on the pre-processed text; S3. Text segmentation: Identify important semantic units in the preprocessed text and segment the text according to semantic boundaries to obtain text fragments; S4. Vector Generation: Convert the segmented text fragments into vector representations; S5. Semantic matching: Use the generated vectors to perform semantic matching, and execute generation or retrieval tasks; The S2 further includes identifying important semantic units in the text based on the results of semantic analysis: theme, argument, and evidence identification, and extracting the core theme, argument, and supporting evidence in the text through the semantic representation output by the large language model; The S3 also This includes using contextual information from a large language model to identify the start and end positions of each semantic unit and using a custom algorithm to segment the text based on the identified semantic boundaries.

2. The method according to claim 1, characterized in that, The S1 input and preprocessing includes preprocessing the text using rule-based word segmentation and statistical word segmentation.

3. The method according to claim 1, characterized in that, The large language model used in the S2 semantic understanding step is BERT or GPT, which is used for understanding the task.

4. The method according to claim 1, characterized in that, The S3 text segmentation includes, but is not limited to, segmentation based on the boundaries of identified semantic units or segmentation based on syntactic analysis.

5. The method according to claim 1, characterized in that, The S4 vectorization process uses Word2Vec or GloVe as the vectorization method to convert text fragments into vectors.

6. The method according to any one of claims 1 to 5, characterized in that, If the task of S5 semantic matching is to generate relevant text, then the model's generation capability is used to generate text; if the task is to retrieve relevant documents, then vectors are used to calculate similarity.

7. An intelligent text segmentation system based on a large language model, characterized in that, include: The input and preprocessing module is used to receive raw text data, clean and normalize the data; The semantic analysis module is used for in-depth semantic analysis; The semantic analysis module identifies important semantic units in the text based on the results of semantic analysis: theme, argument, and evidence identification, and extracts the core theme, argument, and supporting evidence from the text through the semantic representation output by the large language model; The text segmentation module is used to identify semantic units and perform text segmentation; The text segmentation module is also used to identify the start and end positions of each semantic unit using the context information of the large language model and to segment the text using a custom algorithm based on the identified semantic boundaries. The vector generation module is used to convert text fragments into vector representations; The semantic matching module is used to perform generation or retrieval tasks.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the method of any one of claims 1 to 6 by invoking a stored computer program.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images