A text segmentation method, device and medium for large language model
Through the text segmentation method for large language models, the segmentation method is selected according to the text type to ensure that the segmented text blocks are suitable for model input, which solves the problem of text segmentation in the existing technology and improves the learning task Efficiency and accuracy, and learn text block relationships through neural network models to make up for the impact of inappropriate segmentation.
Patent Information
- Application Number
- CN202410571033.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-05-09
AI Technical Summary
The prior art has failed to address the text segmentation requirements of large language models, resulting in text segmentation inappropriateness and affecting the model's learning task efficiency.
Provide a text slicing method for large language models, select matching slicing methods according to text type, ensure that the slicing text blocks do not exceed the model input capacity limit, and adopt directory structure, regularization rules, page slicing and Natural language processing technology performs slicing, and priority is slicing from high to low.
Ensure that the segmented text blocks are suitable for large language model input, improve the efficiency and accuracy of learning tasks, reduce the impact of inappropriate segmentation on model answers, learn the intrinsic relationships of text blocks through the memory mechanism neural network model, and automatically recognize them. Combining text blocks of meaning.
Smart Images

Figure CN118536497B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of intelligent learning technology, and in particular to a text segmentation method, device and medium for a large language model. Background Art
[0002] As a powerful language processing tool, the large language model has been gradually applied in the field of intelligent learning to assist in completing specific learning tasks, such as intelligent question and answer, language learning, and test paper grading. Online learning platforms or users can upload knowledge texts on their own as a reference for the large language model to perform specific learning tasks. When the knowledge text is large, it needs to be uploaded and stored in the knowledge base in blocks.
[0003] In the prior art, patent CN112733545A discloses a text segmentation method device, a computer device and a storage medium, and patent CN112329548A discloses a document chapter segmentation method device and a storage medium, but neither of them considers the impact of the large language model on text segmentation, and does not provide an adaptive text segmentation method for the application characteristics and requirements of the large language model. Summary of the invention
[0004] The embodiments of the present invention provide a text segmentation method, device and medium for a large language model to solve the above technical problems.
[0005] In a first aspect, an embodiment of the present invention provides a text segmentation method for a large language model, comprising:
[0006] Get the text to be segmented;
[0007] According to the text type, select a matching segmentation method to segment the text so that each segmented text block does not exceed the input capacity limit of the large language model;
[0008] Each text block is stored in a vector library to provide reference for the large language model to perform specific learning tasks.
[0009] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:
[0010] one or more processors;
[0011] a memory for storing one or more programs,
[0012] When the one or more programs are executed by the one or more processors, the one or more processors implement the text segmentation method for a large language model described in any embodiment.
[0013] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text segmentation method for a large language model described in any one of the embodiments.
[0014] The embodiment of the present invention provides a text segmentation method for a large language model, which performs text segmentation according to the input capacity limit of the large language model and provides matching segmentation methods and priorities for different text types to ensure the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 It is a flowchart of a text segmentation method for a large language model provided by an embodiment of the present invention;
[0017] Figure 2 This is a schematic diagram of a hierarchical relationship recognition error of a directory title provided by an embodiment of the present invention;
[0018] Figure 3 It is a schematic diagram of a page number sequence provided by an embodiment of the present invention;
[0019] Figure 4 is a schematic diagram of a neural network model based on a memory mechanism in a training phase provided by an embodiment of the present invention;
[0020] Figure 5 is a schematic diagram of a neural network model based on a memory mechanism provided by an embodiment of the present invention during a use phase;
[0021] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0023] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0024] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0025] An embodiment of the present invention provides a text segmentation method for a large language model. In order to illustrate the method, the role of knowledge text when the large language model performs a specific learning task is first introduced. In general, the knowledge text can provide prior knowledge for the large language model to perform a specific learning task, and assist the large language model to complete the learning task more accurately based on the prior knowledge. Taking the learning task of intelligent question answering as an example, when a user asks a question through the learning platform, the platform search will retrieve at least one text block that is most relevant to the question, and encapsulate these text blocks together with the question in a prompt, for example: Please answer the following question <specific question> based on <at least one text block that is most relevant to the question>; and input the prompt into the large language model, and the large language model provides the answer.
[0026] Based on the above process, Figure 1 is a flowchart of a text segmentation method for a large language model provided by an embodiment of the present invention. The method is executed by an electronic device, such as Figure 1 As shown, the specific steps include:
[0027] S110: Obtain text to be segmented.
[0028] The text is the knowledge text that can assist the large language model to complete the learning task. In practical applications, the uploading and retrieval of text are performed in units of text blocks. This embodiment will divide the knowledge text into blocks according to the characteristics of the large language model.
[0029] S120 . According to the text type, a matching segmentation method is selected to segment the text so that each segmented text block does not exceed the input capacity limit of the large language model.
[0030] This embodiment ensures that the segmented text block does not exceed the input capacity limit M1 of the large language model, for example, 125K. The input capacity limits of different large language models may be different. Optionally, the maximum length of the user's question can be counted or agreed in advance, and the size of other cohesive texts in the prompt template can be determined. The maximum capacity M2 of the text block can be obtained by subtracting the question capacity and the capacity of the cohesive text from M1. Exemplarily, in the Chinese environment, the maximum capacity of the text block corresponding to a specific large language model can be set to 400 words.
[0031] Based on the maximum capacity of the text block, this embodiment provides multiple text segmentation methods with different priorities. If the method with the highest priority cannot obtain the text block within the maximum capacity, the method with the next priority will be used to continue text segmentation. Optionally, this embodiment provides four text segmentation methods, which are in descending order of priority: segmentation based on the directory structure of the text, segmentation based on custom regularization rules, segmentation based on pages, and segmentation based on natural language analysis technology and large language models.
[0032] In a specific implementation, text segmentation methods with different priorities can be selected according to the text type, which specifically includes the following steps:
[0033] Step 1: determine whether the text is a formatted text, including formatted documents such as word, PDF, etc. If the text is a formatted text, the method with the highest priority is used to segment the text according to the directory structure of the text.
[0034] Optionally, the directory structure includes multiple directory titles and their hierarchical relationships. First, the text between each title can be treated as a text block according to the default directory title field of the formatted text. Exemplarily, the "Title" field in a word document represents a first-level title, and the "Heading" field represents a second-level title. If these fields can be read, the directory titles and hierarchical relationships of the text at all levels can be read; for PDF documents, if a directory bookmark structure is used, the directory titles and their hierarchical relationships at all levels can be directly read from the bookmark code; the corresponding text content can be located according to the titles at all levels, and the text content between adjacent titles can be treated as a text block to more accurately identify the paragraph content.
[0035] If the directory title field does not exist, the directory titles of each level can be identified and the hierarchical relationship can be inferred according to the set title format. Taking PDF documents as an example, titles of specific formats (such as "Chapter 1", "1. Introduction", etc.) can be identified in specific parts of the document (such as the first few pages) or full-text search; then the hierarchy is judged according to the identifiers that lead the directory title. For example, the identifiers 1 / 2 / 3 generally lead to the first-level directory title, and the identifiers 1.1 / 1.2 / 1.3 generally lead to the second-level directory title; if the hierarchy cannot be judged according to the identifiers that lead the directory title, the hierarchical relationship can be tried to be inferred according to the format of the title (such as numbers, font size, number of indent spaces, etc.). After the inference is successful, the text between each title is divided into the same text block.
[0036] If there is no text that meets the title format, you can call a third-party interface to generate directory titles and hierarchical relationships at all levels. The input of this interface is the entire text, and the output is the directory titles and hierarchical relationships of the text. The text between the output titles is divided into the same text block, and text segmentation based on the directory structure can also be achieved. In particular, in actual applications, when there are multiple levels of directory titles, when using a third-party interface to generate a directory structure, there will be confusion in the identification of the relationship between the lower-level directory titles and the upper-level directory titles. Figure 2 As shown in the figure, the parent directory title of the horizontal line "1.1 Level 3 Title 1" should be "1. Level 2 Title 3", but it is mistakenly identified as belonging to the parent title "1. Level 2 Title 1". To avoid this situation, the identified directory titles of each level can be arranged according to the hierarchical relationship, where multiple titles of the same level are arranged in ascending order of title numbers, and each level of title follows the parent title (i.e. Figure 2 Then, locate the starting page number of each level of the directory title and arrange the page numbers in the order of the title, such as Figure 3 As shown, check whether the arranged title sequence meets the relationship of increasing page numbers. If not, move the directory title with page numbers set back to the correct position so that the title sequence after the move meets the relationship of increasing page numbers. Figure 3 For example, if there is a page rollback between page 25 and page 20 in the arranged page number sequence, the directory title "1.1 Level 3 Title 1" with the page rollback is moved between the titles of page 22 and page 30 to ensure the page number increments. The directory after the move is Figure 2 in the correct directory.
[0037] Step 2: If all the methods in step 1 are used and the directory structure of the document still cannot be generated, then the second method is used to segment the text according to the regularization rules. The specific regularization rules can be customized by the platform or the user, such as segmenting according to customized tags, keywords or format symbols. This method is particularly suitable for documents pre-processed by the user, such as when the user segments different paragraph information according to multiple lines of space, hoping that related paragraphs can no longer be segmented. Optionally, this embodiment defines the following regularization rules:
[0038] "\n\n" / / Split when encountering two line breaks
[0039] ".|!|?" / / Split when encountering . or ! or ? (Chinese punctuation)
[0040] "\.\s|\!\s|\?\s" / / Split when encountering . or ! or ? (English punctuation)
[0041] ";|;\s" / / Split when encountering ; or ; (Chinese punctuation or English punctuation)
[0042] "\n" / / Split when encountering a line break
[0043] ",|,\s" / / Split when encountering, or, (Chinese punctuation or English punctuation)
[0044] Similarly, the priorities of the above regularization rules decrease gradually from top to bottom, that is, when the text block segmented according to the first regularization rule is larger than the maximum capacity of the text block, it is segmented again according to the second regularization rule, and so on, until the segmented text blocks are within the maximum capacity, or the last regularization rule is exhausted.
[0045] Step 3: If the regularization rules in step 2 are not satisfied, and the text is of PDF or PPT type, the third method is used to segment the text according to the page, that is, each page of data forms a separate text and is not segmented again.
[0046] Step 4: If the segmentation conditions of the above three methods are not met, or the above three methods cannot obtain a text block that meets the maximum capacity, then the fourth method is used to segment the text using the NLP (Natural Language Processing) model and the large language model. These two methods can understand and parse the semantic structure of the document content and organize and segment the document according to these semantic structures. Specifically, the NLP model can perform paragraph splitting and divide the text into independent paragraphs. The basic principle is to rely on identifying sentence boundaries and paragraph separators in the text, such as periods, question marks, exclamation marks, and blank lines or specific tags between paragraphs (for example, in HTML, useTags), a Chinese-friendly NLP model can be used for processing. For more complex texts, a large language model can be used for segmentation. The large language model treats the text as a series of words or characters and captures the long-distance dependencies in these sequences to understand the differences and boundaries of the text content.
[0047] In another specific implementation, the priority order of the above segmentation method and / or the size of the text block can also be changed according to the specific learning task to improve the segmentation effect. The following three scenarios of general knowledge, learning and examination are used as examples for explanation. Optionally, in the general knowledge scenario, it is usually necessary to ask and answer questions in the platform. This type of data has no clear format. The priority of the customized regularized segmentation method can be adjusted to the highest, and the text is segmented according to some fixed separators in Chinese (such as.!; line breaks). If the length of the segmented text is still greater than the set length, continue to recursively segment until the requirements are met. Most of the knowledge documents in the learning scenario are relatively standard documents. The priority of segmentation according to the directory structure can be maintained. For documents in word format, the documents are split according to the title, directory, chapter and other information, so that relatively cohesive texts are all in a single segmented document; for PDF documents, a third-party interface can be called to analyze the directory hierarchy of the text, so as to split the document more intelligently. In the exam scenario, the text content is relatively fixed, and is generally composed of questions, options, answers, and analysis. For this kind of data with obvious characteristics, you can customize the regularization rules that match the data, and give priority to using this regularization for document segmentation; at the same time, the overall content of the exam questions will not be very large, so the size of the segmented document content can be appropriately reduced to achieve better matching effects in text block searches.
[0048] S130: Storing each text block in a vector library to provide a reference for the large language model to perform a specific learning task.
[0049] Still taking the intelligent question answering learning task as an example, when answering a question using a large language model, first retrieve at least one text block with the strongest relevance to the learning task from the vector library. Optionally, when both the question and the text block are represented by vectors, the similarity between the question vector and each text block vector can be calculated, and the N text blocks with the highest similarity are taken as the N text blocks with the strongest relevance, where N is a natural number. Due to the capacity limit of the text block or the limitation of the segmentation method itself, a substantial original text may be segmented into two text blocks, for example, a long text that was not finished is segmented, or the logical relationship and hierarchical relationship of the original text are changed after segmentation.
[0050] For example, suppose the original text is "The main branches of the artificial intelligence knowledge system include: machine learning, deep learning, natural language processing, computer vision, reinforcement learning, etc. Among them, machine learning is the foundation of artificial intelligence. It is a technology that enables computers to automatically learn and improve performance by training models; deep learning is a type of machine learning, which is a technology that learns data representation through multi-layer neural networks; natural language processing refers to the technology that enables computers to understand and generate human language; computer vision refers to the technology that enables computers to understand and process images and videos; reinforcement learning is a type of machine learning, which is a technology that learns the optimal strategy through the interaction between intelligent agents and the environment." The segmented text may be "The main branches of the artificial intelligence knowledge system include: machine learning, deep learning, natural language processing, computer vision, etc. Perception, reinforcement learning, etc. Among them, machine learning is the foundation of artificial intelligence. It is a technology that enables computers to automatically learn and improve performance through training models; || (segmentation symbol) deep learning is a type of machine learning. It is a technology that learns data representation through multi-layer neural networks; natural language processing refers to the technology that enables computers to understand and generate human language; computer vision refers to the technology that enables computers to understand and process images and videos; reinforcement learning is a type of machine learning. It is a technology that learns the optimal strategy through the interaction between intelligent agents and the environment. "Such a block dismembers the multi-level branch relationship of the artificial intelligence knowledge system, which is to divide the entire text with substantive meaning into two text blocks. When answering the question "What are the first-level branches of the artificial intelligence knowledge system?", it may affect the answer of the large language model.
[0051] In order to compensate for the impact of inappropriate segmentation, this embodiment does not directly provide the above-mentioned N most relevant text blocks to the large language model, but verifies whether inappropriate segmentation has been performed between at least one text block with the second strongest relevance to the learning task and its adjacent text block. For the convenience of distinction and description, this embodiment refers to the text block with the strongest relevance to the learning task as the first text block, and the text block with the second strongest relevance to the learning task as the second text block. Exemplarily, if the N text blocks ranked in the top of the similarity are taken as N first text blocks, then the n text blocks ranked in the similarity range of N+1 to N+n (n is a natural number less than N) are taken as n second text blocks. Optionally, for any second text block, first locate its adjacent text block in the original text, which is called the third text block, combine the second text block and the third text block into one text block according to their context order in the original text, and calculate the relevance of the combined text block to the learning task. If the relevance of the combined text block to the learning task is greater than the relevance of the second text block to the learning task, it means that the second text block and the third text block are two text blocks obtained after inappropriate segmentation, and it is necessary to compare their combined text block with each first text block, and redetermine at least one text block with the strongest relevance to the learning task. In this embodiment, these redetermined text blocks are called fourth text blocks, and these text blocks are input into the large language model as a reference for executing the learning task.
[0052] In a specific implementation, during the comparison process, when the relevance of the combined text block to the learning task is greater than the relevance of any first text block to the learning task, if the combined text block satisfies the requirement of not exceeding the maximum capacity M2, the combined text block is used as a fourth text block; if the combined text block exceeds the maximum capacity M2, a summary generator can be used to generate a summary of the third text block, and the summary is combined with the second text block according to the context order of the third text block and the second text block in the original text to obtain a new combined text block; at the same time, the number of words in the summary is limited when generating the summary, so that the new combined text block can be less than the maximum capacity M2. Then, it is verified whether the relevance of the new combined text block to the learning task is still greater than that of a first text block. If so, the new combined text block is used as a fourth text block.
[0053] Each time the large language model answers a question or performs a specific learning task, the above process is repeated and the fourth text block generated this time is recorded. When a certain number of fourth text blocks are accumulated, the text blocks with combined meanings of the second text blocks and the third text blocks corresponding to each fourth text block can be used to learn the internal relationship between these text blocks using a neural network model based on a memory mechanism. Figure 4 is a schematic diagram of a neural network model based on a memory mechanism in a training phase provided by an embodiment of the present invention, such as Figure 4 As shown, the model includes an encoder, a memory module and a decoder; wherein the encoder and the decoder can adopt the transformer architecture, respectively including multiple layers of multi-head self-attention layers and normalization layers, and the specific number of layers can be determined according to actual conditions; the memory module is connected between the encoder and the decoder of the transformer. During the training process, the second text block and the third text block corresponding to any fourth text block, as well as other text blocks adjacent to the second text block and the third text block in the original text, together constitute a positive sample (i.e., a sample with substantial meaning after the combination of the intermediate text blocks) in the context order. For example, if the third text block is the next text block of the second text block, the positive sample at this time is "the previous text block of the second text block + the second text block + the third text block + the next text block of the third text block"; the positive sample is input into the encoder to extract the deep semantic features including the context information; after the deep semantic features are input into the memory module, the features are reconstructed according to the positive sample features stored in the memory module, and the components related to the representative features of the positive samples in the deep semantic features are strengthened to highlight the difference between the positive sample and the negative sample (i.e., the sample whose intermediate text block does not have significant combination meaning); the reconstructed features are continuously input into the decoder, and the fourth text block is restored through the output of the model. A loss function can be constructed according to the difference between the decoder output and the real fourth text block, and the model parameters are updated by minimizing the loss function, so that after each positive sample is input into the model, the model output continuously approaches each fourth text block.
[0054] Furthermore, the positive sample features in the memory module represent the typical semantic components of the positive samples. After the deep semantic features are input into the memory module, the following operations are performed in the memory module: the deep semantic features are decomposed into K parts, and the similarity between each part and the positive sample features already in the memory module is calculated respectively; for the same positive sample feature, the similarity between the feature and each part is weighted averaged to obtain the similarity between the feature and the input deep semantic features; each positive sample feature F i The similarity a with the input deep semantic feature I i By arranging them in sequence, the reconstruction vector [a1, a2, ..., a M ,], where M represents the number of positive sample features stored in the memory module. At the same time, the positive sample feature with the lowest similarity to the deep semantic feature is used as the target feature, and the weighted sum of each part of the deep semantic feature is performed according to the similarity between the parts and the target feature, and the target feature is updated to the result of the weighted sum. This result is equivalent to reconstructing the features in the memory module according to the parts of the positive sample. Through such feature feedback, the memory module can continuously extract the typical semantic components of the positive sample, thereby distinguishing it from the negative sample.
[0055] Optionally, you can construct the following loss function:
[0056]
[0057] Wherein, P represents the number of elements of the fourth text block. When the text is represented as an embedded vector, P represents the number of elements of each text vector. and P i I represents the i-th element in the restored fourth text block and the real fourth text block respectively; k represents the kth part of the deep semantic feature, F m represents the positive sample feature that is most similar to the deep semantic feature in the memory module, F s represents the second most similar positive sample to the deep semantic feature in the memory module, γ is a hyperparameter for maintaining the basic distance between classes, and α and β represent the loss weights respectively.
[0058] When the loss function is minimized, It is possible to ensure that the difference between the restored fourth text block and the real fourth text block is minimized; It can make the deep semantic features as close as possible to a certain positive sample feature in the memory module, thereby reducing the intra-class differences of the positive sample categories; It can ensure the diversity of positive sample features in the memory module to record different normal modes.
[0059] Furthermore, considering that the present embodiment introduces a summary generator, the training process and the loss function may be adjusted as follows for the fourth text block including the summary: if a fourth text block includes the summary, during the training, the second text block and the third text block corresponding to the fourth text block, as well as other text blocks adjacent to the second text block and the third text block in the original text are arranged in context order as an input sample (e.g., the previous text block of the second text block + the second text block + the third text block + the next text block of the third text block), and at the same time, the fourth text block and the other text blocks are arranged in context order as another Input samples (i.e., the previous text block of the second text block + the second text block + the summary of the third text block + the next text block of the third text block); input the two input samples into the encoder respectively, and extract the deep semantic features of the text blocks respectively; input the two deep semantic features into the memory module respectively, and reconstruct the features according to the positive sample features stored in the memory module respectively; input the two reconstructed features into the decoder respectively, and restore the fourth text block respectively; train the neural network model by constraining the differences between the two restored text blocks and the fourth text block to be minimized respectively, and the differences between the two reconstructed features to be minimized. Optionally, the loss function at this time can be:
[0060]
[0061] in, and represents the i-th element in the fourth text block restored according to the one input sample and the other input sample respectively; J represents the number of elements of the reconstructed feature, S 1j and S 2j represents the jth component in the reconstructed features corresponding to the one input sample and the other input sample respectively; w1 and w2 represent the loss weights respectively. Under such a loss function, the positive sample features stored in the memory module will gradually take into account the positive sample summary part's extraction angle of the original text, ensuring that the summary information in the restored fourth text block does not miss the key information of the intermediate text as much as possible.
[0062] After the model training is completed, the parameters of the encoder and decoder, as well as the positive sample features in the memory module are no longer updated. The trained model can be used as an inappropriate segmentation identifier to automatically identify whether there is a possibility of inappropriate segmentation between text blocks that are adjacent in the original text. Specifically, any two adjacent text blocks in the vector library (two text blocks that are adjacent in the original text) are combined in context order, and the combined text block C and the adjacent text blocks of C in the original text are arranged in context order and input into the trained model together. The model will output a text block that does not exceed the maximum capacity M, such as Figure 5 As shown. If the similarity between the model output and the above-mentioned combined text block (i.e., the two adjacent text blocks A+B) is greater than the first threshold, and the similarity between the deep semantic features generated in the model and any positive sample features in the memory module is greater than the second threshold, then the input model is considered to be a positive sample, and continue to determine whether the combined text block does not exceed the maximum capacity M2 of the text block; if so, add the combined text block to the vector library; otherwise, add the text block output by the model to the vector library. Correspondingly, if the above two threshold conditions are not met, the input model is considered to be a negative sample, and the two adjacent text blocks do not have a combined meaning.
[0063] In a specific embodiment, each of the above two thresholds can be set manually, or can be determined based on the original text B0 composed of the fourth text blocks B4 including the summary and the second text block and the third text block during the training process. Optionally, the similarities of multiple groups of B4 and B0 are calculated, and these similarities are averaged. The average value represents the degree of similarity between the summary and the original text. If it is greater than this value, it can be considered that the two texts are similar enough; a value less than the average value can be taken as the first threshold, that is, the difference between the summary and the original text is allowed. Optionally, for each group of B4 and B0, two input samples are constructed with B4 and B0 of the same group as the intermediate text, and the similarity S of the deep semantic features generated by the two input samples in the model is calculated. x , the similarity S corresponding to multiple groups B4 and B0 x The average is calculated, and the average represents the difference in deep semantic features between the summary and the original text level. A value smaller than the average can be taken as the second threshold, that is, the difference in deep semantic features between the summary and the original text level is allowed.
[0064] In summary, the model can not only learn the contextual relationship between the intermediate text and its adjacent text blocks, but also distinguish positive samples from negative samples through the memory module, so as to quickly identify the combined text with substantial meaning and make up for the impact of inappropriate segmentation. Compared with simply calculating the semantic coherence and semantic similarity between text blocks to determine whether the text blocks need to be combined, the method of this embodiment does not particularly limit the standard for predicting whether the text block has a combined meaning, but selects the fourth text block that does have a combined meaning from practical applications to form a positive sample library, and extracts the internal relationship of the positive sample with the help of the memory mechanism (not limited to semantic coherence and semantic similarity, but also including logical relationships, hierarchical structures, and other implicit relationships, etc.), which will be more accurate in learning the text relevance and judging the combined meaning.
[0065] In summary, this embodiment provides a text segmentation method for a large language model, performs text segmentation according to the input capacity limit of the large language model, and provides matching segmentation methods and priorities for different text types to ensure the segmentation effect. For different application scenarios (or learning tasks), this embodiment provides a flexible priority order and text block size to adapt to the text features in different scenarios. In view of the problem of chaotic hierarchy of the directory structure generated by the third-party interface, this embodiment detects and identifies the misplaced directory title through page number increment, corrects the directory structure, and ensures orderly text segmentation. In view of the problem that text blocks with substantive meaning are inappropriately segmented, this embodiment verifies whether the text blocks with the second strongest correlation have combination meaning in the text block retrieval, and reduces the impact of inappropriate segmentation on the execution of learning tasks by the large language model; when the positive samples with combination meaning accumulate to a certain extent, this embodiment learns the intrinsic association of the combined text through a neural network model based on the memory mechanism, thereby automatically identifying the text blocks with sufficient combination meaning in the vector library, and further compensating for the impact of inappropriate segmentation on the execution of learning tasks.
[0066] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 6 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 6 The example of connecting through bus is taken in the following.
[0067] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the text segmentation method for a large language model in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned text segmentation method for a large language model.
[0068] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0069] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.
[0070] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the text segmentation method for a large language model of any embodiment is implemented.
[0071] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.
[0072] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0073] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0074] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.
Claims
1. A text segmentation method for a large language model, characterized in that: include: Get the text to be segmented; According to the text type, select a matching segmentation method to segment the text so that each segmented text block does not exceed the input capacity limit of the large language model; Each text block is stored in a vector library to provide reference for the large language model to perform specific learning tasks; When the large language model performs a specific learning task, retrieving a first text block that is most relevant to the learning task from the vector library; Combine the second text block having the second strongest relevance to the learning task and the third text block adjacent to the second text block in the original text in context order, and calculate the relevance of the combined text block to the learning task; In the case where the relevance of the combined text block to the learning task is greater than the relevance of the first text block to the learning task, if the combined text block meets the input capacity limit, the combined text block is used as a new fourth text block with the strongest relevance to the learning task; otherwise, the summary of the third text block is combined with the second text block in context order so that the new combined text block meets the input capacity limit, and the fourth text block with the strongest relevance to the learning task is re-determined based on the new combined text block for input into a large language model as a reference for performing the learning task.
2. The method according to claim 1, characterized in that The method of selecting a matching segmentation method according to the text type to segment the text includes: If the text is formatted text, segment the text according to the directory structure of the text; If the directory structure is not generated, the text is segmented according to the regularization rules; If the regularization rule is not satisfied and the text is of PDF type or PPT type, the text is segmented according to the page; If none of the above conditions are met, use natural language analysis technology and large language models to segment text.
3. The method according to claim 2, characterized in that The directory structure includes directory titles at all levels and their hierarchical relationships; If the text is formatted text, segmenting the text according to the directory structure of the text includes: According to the default directory title field of the formatted text, read the directory titles and hierarchical relationships of each level of the text; If the directory title field does not exist, identify the directory titles at each level according to the set title format and infer the hierarchical relationship; If there is no text satisfying the title format, a third-party interface is called to generate directory titles and hierarchical relationships at each level.
4. The method according to claim 3, characterized in that After calling the third-party interface to generate the catalog titles and hierarchical relationships at all levels, the method further includes: Arrange the titles of the catalog at all levels in a hierarchical relationship, where multiple titles of the same level are arranged in ascending order of title numbers, and the titles of each level are immediately following the titles of the upper level to which they belong; Locate the starting page number of each level of directory titles, and check whether the arranged title sequence satisfies the relationship of increasing page numbers; If it is not satisfied, move the table of contents title with page number rolled back to the correct position so that the title sequence after the move satisfies the relationship of increasing page numbers.
5. The method according to claim 1, characterized in that After re-determining the fourth text block that is most relevant to the learning task, the method further includes: After a certain number of fourth text blocks are accumulated, the second text blocks and the third text blocks corresponding to each fourth text block, and other text blocks adjacent to the second text blocks and the third text blocks in the original text together constitute each input sample in context order, and the neural network model based on the memory mechanism is trained so that the model output continuously approaches each fourth text block; Combining any two adjacent text blocks in the vector library in context order; The combined text block and its adjacent text blocks in the original text are arranged in context order and input into the trained model together; if the similarity between the model output and the combined text block is greater than a set threshold, the combined text block or the model output is added to the vector library.
6. The method according to claim 5, characterized in that The neural network model based on the memory mechanism includes an encoder, a memory module and a decoder; The second text block and the third text block corresponding to each fourth text, and other text blocks adjacent to the second text block and the third text block in the original text constitute each input sample in context order, and the neural network model based on the memory mechanism is trained so that the model output continuously approaches each fourth text block, including: If a fourth text block includes an abstract, the second text block and the third text block corresponding to the fourth text block, and other text blocks adjacent to the second text block and the third text block in the original text are arranged in context order as an input sample, and at the same time, the fourth text block and the other adjacent text blocks are arranged in context order as another input sample; Inputting the two input samples into the encoder respectively, and extracting the deep semantic features of the text blocks respectively; Inputting the two deep semantic features into the memory module respectively, and reconstructing the features according to the positive sample features stored in the memory module respectively, wherein the positive sample features are used to characterize the typical semantic components of the input samples with combined meaning of the intermediate text block; Inputting the two reconstructed features into the decoder respectively to restore the fourth text block respectively; The neural network model is trained by constraining the differences between the two restored text blocks and the fourth text block to be minimized respectively, and the difference between the two reconstructed features to be minimized.
7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the text segmentation method for a large language model as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the text segmentation method for a large language model described in any one of claims 1-6 is implemented.
Citation Information
Cited By
Text block dynamic segmentation method and system based on RAG
CN120353880A
RAG-based text block dynamic segmentation method and system
CN120353880B
