Text segmentation method and device in natural language processing

Abstract

The embodiment of the invention provides a text segmentation method and device in natural language processing. The method comprises the steps that a target text segment is segmented into sentences, and sentence embedding vectors of the sentences are obtained; a relation network graph is constructed with sentences as nodes, and connection edges between the nodes are determined according to the similarity between the sentence embedding vectors; performing community detection on the relational network diagram, and dividing nodes into different communities; and according to a community detection result, extracting a plurality of sentences corresponding to nodes divided in the same community, obtaining positions of the plurality of sentences in the target text segment, and forming text blocks based on the plurality of sentences and the positions thereof. The text segmentation accuracy can be improved.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computers, and in particular, to a text segmentation method and apparatus in natural language processing. Background Art

[0002] Text segmentation is a fundamental task in natural language processing (NLP), aiming to divide continuous text into meaningful segments. For example, in retrieval-augmented generation (RAG) systems, text segmentation is a key step in breaking down large-scale documents or knowledge base content into smaller segments suitable for embedding and retrieval. The efficiency and accuracy of text segmentation play a vital role in improving the system's retrieval efficiency and the accuracy of generated answers.

[0003] A common text segmentation method in existing technologies relies on manually designed rules. While simple to implement, it performs poorly when handling complex sentences and long context dependencies. This results in previously coherent text being severed, resulting in poor segmentation accuracy and, in turn, impacting the vector recall performance of the RAG system. Other common text segmentation methods also suffer from poor text segmentation accuracy. Summary of the Invention

[0004] One or more embodiments of this specification describe a text segmentation method and apparatus in natural language processing, which can improve text segmentation accuracy.

[0005] In a first aspect, a text segmentation method in natural language processing is provided, the method comprising:

[0006] Split the target text segment into sentences and obtain the sentence embedding vector of each sentence;

[0007] Build a relationship network graph with sentences as nodes, where the connection edges between nodes are determined by the similarity between sentence embedding vectors;

[0008] Performing community detection on the relationship network graph and dividing the nodes into different communities;

[0009] According to the results of community detection, several sentences corresponding to the nodes classified in the same community are extracted, and the positions of the several sentences in the target text segment are obtained. Based on the several sentences and their positions, text segments are formed.

[0010] In a possible implementation, segmenting the target text segment into sentences includes:

[0011] Searching for preset punctuation marks in the target text segment;

[0012] The target text segment is segmented at the locations of the preset punctuation marks to obtain segmented sentences.

[0013] Furthermore, the preset punctuation marks include:

[0014] A period, question mark, exclamation point, or semicolon.

[0015] In one possible implementation, obtaining a sentence embedding vector for each sentence includes:

[0016] Input the target sentence into the first embedding model to obtain a sentence embedding vector of the target sentence; or,

[0017] The target sentence and the context sentences of the target sentence are input into the second embedding model to obtain a sentence embedding vector of the target sentence.

[0018] In a possible implementation, constructing a relationship network diagram using sentences as nodes includes:

[0019] Determine the similarity between the sentence embedding vectors corresponding to any two sentences and use it as the similarity between the two sentences;

[0020] Each sentence corresponds to a node, and whether there is a connection edge between the corresponding two nodes is determined based on the similarity between the two sentences to construct a relationship network graph.

[0021] Furthermore, determining the similarity between sentence embedding vectors corresponding to any two sentences includes:

[0022] Normalize the sentence embedding vector of any sentence to obtain the normalized vector of the sentence;

[0023] Perform a dot product on the normalized vectors of any two sentences to obtain the similarity between the two sentences.

[0024] Furthermore, determining whether there is a connecting edge between two corresponding nodes based on the similarity between the two sentences includes:

[0025] For the first node, find the K second nodes that are ranked in the top K similarities with it among all other nodes;

[0026] If the similarity between the first node and the second node is greater than a preset threshold, a connecting edge is added between the first node and the second node.

[0027] In a possible implementation, the relationship network graph is an undirected graph; or, the relationship network graph is a directed graph, and the direction of the connecting edge is determined according to the order of the two sentences connected by it in the target text segment.

[0028] In a possible implementation, performing community detection on the relationship network graph to divide nodes into different communities includes:

[0029] Initialize each node as an independent community;

[0030] For each node, try to move it to the community to which its neighboring nodes belong. After each move, calculate the modularity of the relationship network graph. If moving the node to the community of the neighboring node can improve the modularity, then perform this move.

[0031] After the change in modularity becomes stable, the currently identified community is regarded as a new node, a new relationship network diagram is constructed, and community detection continues.

[0032] Furthermore, the modularity is used to evaluate whether the currently divided community structure can maximize the difference between the connection density within the community and the expected random connection density.

[0033] Furthermore, constructing a new relationship network diagram includes:

[0034] Update the edge weight of the connection between the two new nodes to make it represent the connection strength between the communities corresponding to the two new nodes.

[0035] In a second aspect, a text segmentation device for natural language processing is provided, the device comprising:

[0036] The first segmentation unit is used to segment the target text segment into sentences and obtain the sentence embedding vector of each sentence;

[0037] a graph construction unit, configured to construct a relationship network graph using the sentences obtained by the first segmentation unit as nodes, wherein the connection edges between the nodes are determined based on the similarity between the sentence embedding vectors;

[0038] a detection unit, configured to perform community detection on the relationship network graph obtained by the graph construction unit, and divide the nodes into different communities;

[0039] The second segmentation unit is used to extract a number of sentences corresponding to nodes classified into the same community based on the community detection results obtained by the detection unit, obtain the positions of the several sentences in the target text segment, and form text segments based on the several sentences and their positions.

[0040] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0041] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0042] Through the method and device provided by the embodiment of this specification, the target text segment is first divided into sentences, and the sentence embedding vector of each sentence is obtained; then a relationship network diagram is constructed with the sentences as nodes, wherein the connection edges between the nodes are determined according to the similarity between the sentence embedding vectors; then community detection is performed on the relationship network diagram to divide the nodes into different communities; finally, according to the results of the community detection, a number of sentences corresponding to the nodes divided into the same community are extracted, and the positions of the several sentences in the target text segment are obtained, and based on the several sentences and their positions, text segments are formed. As can be seen from the above, the embodiment of this specification describes the relationship between sentences in the target text segment as a relationship network diagram. Since two discontinuous sentences can be associated through the graph structure, it can break through the distance limit between sentences and establish a connection between any two highly semantically related sentences in a long text, and can handle nonlinear and complex text structures. In addition, by introducing the community detection analysis graph structure, it is possible to comprehensively analyze the relationship between all sentences from a global perspective, find substructures with close connections in the graph, and then achieve the purpose of dividing semantically coherent and similar sentences, which can improve the accuracy of text segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0045] Figure 2 A flowchart of a text segmentation method in natural language processing according to one embodiment is shown;

[0046] Figure 3 shows a relationship network diagram according to one embodiment;

[0047] Figure 4 A schematic block diagram of a text segmentation apparatus in natural language processing according to one embodiment is shown. DETAILED DESCRIPTION

[0048] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0049] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves text segmentation in natural language processing. It can be understood that text segmentation is used to divide continuous text into meaningful segments. Figure 1 A text segment, consisting of sentences 1, 2, and 3, is first segmented into two text blocks, namely, text block 1 and text block 2. Then, text block 1 is embedded to obtain vector 1, and text block 2 is embedded to obtain vector 2. Finally, vectors 1 and 2 are added to the vector database. Subsequently, in retrieval-augmented generation (RAG), the target vector can be retrieved from the vector database based on the user's question. The large model is then used to generate the corresponding answer based on the target vector and the user's question.

[0050] The following is a brief explanation of some of the terms mentioned above.

[0051] RAG is a natural language processing model that combines retrieval and generation. It retrieves relevant information from a large number of documents or knowledge bases before generating an answer, improving the accuracy and relevance of the generated content. This approach leverages external knowledge sources and enhances the model's performance in answering complex questions.

[0052] Embedding is a method of converting high-dimensional data (such as words, sentences, or images) into a continuous vector representation in a low-dimensional vector space. This representation retains the semantic and structural information of the original data, making it easy to process and calculate in machine learning models.

[0053] A vector database is a database system specifically designed for storing, managing, and querying high-dimensional vector data. It supports efficient similarity search and vector retrieval and is commonly used in fields such as machine learning, recommendation systems, image retrieval, and natural language processing. By optimizing index structures and search algorithms, vector databases can rapidly process large amounts of vector data and achieve real-time response.

[0054] It should be noted that when different text segmentation methods are used, different text segments will be obtained. Figure 1For example, the segmentation results include text segment 1 consisting of sentence 1, and text segment 2 consisting of sentences 2 and 3. It is understandable that this is not the only segmentation result of text segmentation. For example, the segmentation result may also be that sentence 1 constitutes one text segment, sentence 2 constitutes another text segment, and sentence 3 constitutes another text segment; or, sentence 1 and sentence 2 constitute one text segment, and sentence 3 constitutes another text segment.

[0055] Traditional text segmentation methods are mainly based on manually designed segmentation rules. For example, one of the most common segmentation methods is to divide long text into small text segments according to a fixed length. However, this fixed-length segmentation method often causes sentence segmentation problems, resulting in semantic incoherence of the segmented text segments. In order to make up for the shortcomings of fixed-length segmentation, some schemes propose the concept of overlapping length, that is, to make sure that there is a certain length of overlap between two consecutive text segments, so that the problem of semantic destruction is alleviated to a certain extent. In addition, the punctuation-based segmentation method divides sentences by identifying punctuation marks (such as periods and question marks) at the end of sentences.

[0056] Here are some common text segmentation methods.

[0057] A common text segmentation method is fixed-length segmentation. Fixed-length segmentation is the most basic and commonly used text segmentation method. It divides the text into segments with a fixed number of characters or words. For example, every 500 characters or every 100 words in a long document is considered a segmentation unit. This method is simple to implement and computationally efficient, making it suitable for scenarios with clear requirements for segmentation granularity. Fixed-length segmentation ignores the semantic structure of the text and may segment in the middle of a sentence or paragraph, resulting in the segmented fragments lacking complete semantic information. In addition, in text with multiple topics or complex content, fixed-length segmentation may mix information from different topics in the same segmentation unit, reducing the relevance of retrieval and the coherence of generated content.

[0058] Another common text segmentation method is recursive character segmentation. This segmentation method gradually refines the segmentation granularity, starting at the character level and segmenting until specific conditions are met. Generally speaking, the text is segmented step by step according to a pre-agreed set of punctuation marks. For example, the text is first segmented into different paragraphs by line breaks. If the length of a paragraph exceeds the maximum length limit, it is further segmented within the paragraph by periods, and so on, until the length of the segmented text segments does not exceed the maximum length limit. Because punctuation is used as the segmentation basis, this method can capture the semantic information of the text to a certain extent. Compared with fixed-length segmentation, it can largely avoid the problem of segmenting within a single sentence. Punctuation, as a natural segmentation point of the article, can only serve as a weak semantic segmentation marker. In actual use, it still faces problems such as incorrectly segmenting complete semantic segments and constructing semantic segments that mix multiple topics.

[0059] Another common text segmentation method is semantic segmentation. Semantic segmentation uses an embedding model to model the semantics of individual sentences and the semantic relationships between them. First, a long text is initially segmented according to specific rules (such as fixed length and punctuation). After this initial segmentation, the sentences are embedded using an embedding model to obtain embedding vectors. Embedding vectors, representing natural sentences in a low-dimensional space, capture the semantic information of the sentences. Specifically, semantically similar sentences are closer in the embedding space, while semantically similar sentences are farther apart. Using the embedding vectors as a starting point, semantic segmentation calculates the similarity between consecutive sentences, typically using cosine similarity as a metric. After determining the similarity between consecutive sentences, the points where the similarity changes dramatically are analyzed to identify the locations of semantic transitions, thereby segmenting the text at these points. Compared to previous segmentation methods, semantic segmentation incorporates a deep learning-based embedding model that effectively captures the semantic connections between sentences and dynamically determines the optimal segmentation locations. However, its semantic segmentation strategy only considers the semantic connections between consecutive sentences. For example, if a text consists of three sentences, A, B, and C, the semantic segmentation will calculate the similarity between A and B, and the similarity between B and C, respectively, without considering the connection between A and C. If the semantics of B contain significant noise, sentences A, B, and C, which should have been clustered together, may be prematurely segmented at B. Similarly, if B contains ambiguous semantics that are close to both A and C, A and C, which originally had no semantic connection, may be clustered together through B, resulting in semantic confusion in the segmented segments.

[0060] Another common text segmentation method is topic node segmentation. Topic node segmentation extracts text semantic features based on an embedding model or a large language model. Similar to the semantic segmentation above, the long text is first segmented, and then each sentence after the initial segmentation is traversed one by one. For each sentence, there are two options: merge it with the previous sentence to construct a text block with the same topic, or choose to create a new topic. This step can use the embedding model to calculate similarity, or directly obtain a yes or no answer based on the large language model. For the method using the embedding model, it is necessary to calculate the embedding vector of the existing text of the current topic and the embedding vector of the current sentence respectively. If the similarity between the two is greater than the threshold, the topic is merged. Otherwise, a new topic is segmented and constructed. The method using the large language model uses a specific prompt to guide the large language model to compare the existing topic text and the current sentence to see if they should be merged. The main problem with this method is that it is computationally intensive and time-consuming, which is mainly caused by the way it determines the segmentation points. When using an embedding model to determine the split point, it is necessary to repeatedly calculate the embedding vectors of the text in the existing topic. Taking the three sentences A, B, and C as an example, when determining whether B should be split, it is necessary to calculate the embedding vector of A. When determining whether C should be split, it is necessary to calculate the embedding vector of the combined text of A and B. If the topic is very long, the amount of calculation will increase quadratically. When using a large language model to determine the split point, the large language model itself has a very large amount of calculation and is extremely costly. In addition, due to the quadratic increase in the amount of calculation, a large number of large language models will be called during the splitting process, introducing a computational bottleneck. In addition, the topic node segmentation only considers the relationship between the current sentence and the previous topic node, and cannot capture the overall semantic association from a global perspective, and cannot obtain a globally optimal solution.

[0061] The embodiment of this specification hopes to group semantically coherent and similar sentences into the same text segmentation, thereby improving the accuracy of text segmentation.

[0062] Figure 2 A flowchart of a text segmentation method in natural language processing according to an embodiment is shown. The method can be based on Figure 1 The implementation scenario shown in Figure 2As shown, the text segmentation method in natural language processing in this embodiment includes the following steps: Step 21, segmenting the target text segment into sentences and obtaining the sentence embedding vector of each sentence; Step 22, constructing a relationship network diagram with sentences as nodes, wherein the connection edges between nodes are determined based on the similarity between the sentence embedding vectors; Step 23, performing community detection on the relationship network diagram and dividing the nodes into different communities; Step 24, based on the results of the community detection, extracting several sentences corresponding to the nodes divided into the same community, and obtaining the positions of the several sentences in the target text segment, and forming text segments based on the several sentences and their positions. The specific execution method of each of the above steps is described below.

[0063] First, in step 21, the target text segment is divided into sentences, and sentence embedding vectors of each sentence are obtained. It can be understood that an article usually includes several text segments, and a text segment contains several sentences.

[0064] In one example, segmenting the target text segment into sentences includes:

[0065] Searching for preset punctuation marks in the target text segment;

[0066] The target text segment is segmented at the locations of the preset punctuation marks to obtain segmented sentences.

[0067] Furthermore, the preset punctuation marks include:

[0068] A period, question mark, exclamation point, or semicolon.

[0069] For example, based on a predefined set of punctuation marks, such as period, question mark, exclamation mark, semicolon, etc., the input original text is roughly segmented to obtain a set of sentences.

[0070] In one example, obtaining a sentence embedding vector for each sentence includes:

[0071] Input the target sentence into the first embedding model to obtain a sentence embedding vector of the target sentence; or,

[0072] The target sentence and the context sentences of the target sentence are input into the second embedding model to obtain a sentence embedding vector of the target sentence.

[0073] In this example, a sentence embedding vector for a single sentence can be generated based on the sentence alone. Alternatively, a sliding window or deep learning can be added to organically integrate the context of a single sentence to form a more robust embedding vector representation.

[0074] For example, using the embedding model, calculate the embedding vector of each roughly segmented sentence. Let the embedding model be f, then the embedding vector of the i-th sentence is e i =f(s i Here, you can choose an appropriate embedding model based on your actual business needs. It's worth noting that the quality of the embedding model directly determines the quality of the generated embedding vectors. If the embedding model cannot effectively map the semantics of the sentence into a low-dimensional space, the subsequent mapping and community detection results will be reduced.

[0075] Then, in step 22, a relational network diagram is constructed with the sentences as nodes, wherein the connection edges between the nodes are determined based on the similarity between the sentence embedding vectors. It can be understood that the relational network diagram can establish a connection between two discontinuous sentences.

[0076] In one example, constructing a relationship network graph with sentences as nodes includes:

[0077] Determine the similarity between the sentence embedding vectors corresponding to any two sentences and use it as the similarity between the two sentences;

[0078] Each sentence corresponds to a node, and whether there is a connection edge between the corresponding two nodes is determined based on the similarity between the two sentences to construct a relationship network graph.

[0079] In this example, the relationship between sentences in the text is described through a relationship network diagram, and low-similarity interference sentences are filtered out during the diagram construction process, which can accurately describe the complex nonlinear associations between sentences.

[0080] Furthermore, determining the similarity between sentence embedding vectors corresponding to any two sentences includes:

[0081] Normalize the sentence embedding vector of any sentence to obtain the normalized vector of the sentence;

[0082] Perform a dot product on the normalized vectors of any two sentences to obtain the similarity between the two sentences.

[0083] Furthermore, determining whether there is a connecting edge between two corresponding nodes based on the similarity between the two sentences includes:

[0084] For the first node, find the K second nodes that are ranked in the top K similarities with it among all other nodes;

[0085] If the similarity between the first node and the second node is greater than a preset threshold, a connecting edge is added between the first node and the second node.

[0086] In this example, the k-nearest neighbor algorithm is used to filter out low-similarity interference sentences during the graph construction process, which can more accurately describe the complex nonlinear relationships between sentences.

[0087] In one example, the relationship network graph is an undirected graph; or, the relationship network graph is a directed graph, and the direction of the connecting edge is determined according to the order of the two sentences connected by it in the target text segment.

[0088] In this example, a directed graph can be used instead of an undirected graph to more precisely depict the relationship between sentences.

[0089] Figure 3 FIG. 1 shows a relationship network diagram according to an embodiment. Figure 3 , the relationship network diagram is an undirected graph. In step 21 above, the target text segment is divided into four sentences, which are sentence 1, sentence 2, sentence 3, and sentence 4 in order. Sentence 1 corresponds to node 1, sentence 2 corresponds to node 2, sentence 3 corresponds to node 3, and sentence 4 corresponds to node 4. There is a connecting edge between node 1 and node 2, a connecting edge between node 1 and node 4, and a connecting edge between node 3 and node 4.

[0090] Building a relationship network diagram includes the following processes:

[0091] First, for any two sentences, calculate the cosine similarity between the embedding vectors and obtain the similarity matrix. The choice of similarity calculation method can be based on the matching of the indicators used by the embedding model. This step is completed by vector L2 normalization and matrix multiplication. The embedding vector after L2 normalization is recorded as Then all the embedding vectors constitute Through one matrix multiplication, we can obtain the similarity matrix

[0092] Then, for any sentence, we search for K nearest neighbors, traverse each row of the similarity matrix, sort the similarity of each row from high to low, and select the K elements with the highest similarity as the K nearest neighbors of the sentence corresponding to that row. Specifically: N k (i)=argsort(j∈1,2,...,n\i)(C ij )|top k.

[0093] Finally, each sentence is considered as a node, and an undirected graph is constructed based on the results of the K nearest neighbors. The specific steps are as follows: First, a corresponding node is created for each sentence; then, all sentences are traversed and their K nearest neighbors are determined. When traversing each nearest neighbor, if the similarity between the current sentence and the nearest neighbor exceeds the preset threshold τ, an edge is established between the two, and the weight of the edge is set to the similarity C ij This step is expressed as E=(vi , v j )∣V j ∈N k (i)&C ij >τ.

[0094] Then, in step 23, community detection is performed on the relationship network graph to divide the nodes into different communities. It is understandable that there is no single method for community detection, and any existing community detection algorithm can be used.

[0095] Community detection is a technique in graph theory and network analysis used to identify closely connected groups of nodes in a network, also known as "communities" or "neighborhoods." Nodes within these communities are densely connected, but their connections to other communities are relatively sparse.

[0096] In one example, performing community detection on the relationship network graph and dividing nodes into different communities includes:

[0097] Initialize each node as an independent community;

[0098] For each node, try to move it to the community to which its neighboring nodes belong. After each move, calculate the modularity of the relationship network graph. If moving the node to the community of the neighboring node can improve the modularity, then perform this move.

[0099] After the change in modularity becomes stable, the currently identified community is regarded as a new node, a new relationship network diagram is constructed, and community detection continues.

[0100] Furthermore, the modularity is used to evaluate whether the currently divided community structure can maximize the difference between the connection density within the community and the expected random connection density.

[0101] Furthermore, constructing a new relationship network diagram includes:

[0102] Update the edge weight of the connection between the two new nodes to make it represent the connection strength between the communities corresponding to the two new nodes.

[0103] For example, based on the constructed relationship network graph, the Louvain community detection algorithm is used to divide nodes into different communities. The specific steps include the following stages: initialization, modularity optimization, aggregation, and iterative optimization.

[0104] Among them, the initialization phase: each node is initially regarded as an independent community.

[0105] Modularity optimization phase: For each node in the network graph, we attempt to move it to a different community to which its neighboring nodes belong. After each move, we calculate the modularity of the graph, evaluating whether the current community structure maximizes the difference between the connection density within the community and the expected random connection density. If moving a node to a neighboring community improves the overall modularity, we perform the move.

[0106] Aggregation phase: After modularity stops increasing significantly, the currently identified communities are treated as new supernodes. A new relationship network graph is constructed, where the edge weights between supernodes represent the strength of the connections between the original communities.

[0107] Iterative optimization phase: Repeat the modularity optimization and aggregation phases on the new relationship network diagram until the modularity is no longer significantly improved or the preset number of iterations is reached.

[0108] It should be noted that in addition to the Louvain algorithm, other community detection algorithms can also be used, such as the Clauset-Newman-Moore algorithm, spectral clustering, label propagation, and Infomap. In the Louvain algorithm, each sentence can only be assigned to one community. However, in practice, the content expressed in certain sentences may naturally be associated with multiple communities. Therefore, one-to-many sentence community division can be achieved by selecting or redesigning a community detection algorithm.

[0109] Finally, in step 24, based on the results of the community detection, several sentences corresponding to the nodes classified into the same community are extracted, and the positions of the several sentences in the target text segment are obtained. Based on the several sentences and their positions, text segments are formed. It is understandable that the several sentences may not be consecutive sentences, and the several sentences are combined according to their positions to form text segments. For example, sentences 1, 2, and 4 are classified into the same community, where the numbers in the sentence identifiers represent their positions. Then, combining sentences 1, 2, and 4 in sequence forms a text segment.

[0110] In this embodiment, based on the results of community detection, sentences corresponding to nodes in the same community are extracted and sorted according to the order in the original text to ensure text coherence. The sorted sentences are then connected in sequence to form a complete text segment. Each text segment contains highly semantically related sentences.

[0111] According to the method provided in the embodiment of this specification, the target text segment is first divided into sentences, and the sentence embedding vector of each sentence is obtained; then a relationship network diagram is constructed with the sentences as nodes, wherein the connection edges between the nodes are determined according to the similarity between the sentence embedding vectors; then community detection is performed on the relationship network diagram to divide the nodes into different communities; finally, according to the results of the community detection, a number of sentences corresponding to the nodes divided into the same community are extracted, and the positions of the several sentences in the target text segment are obtained, and based on the several sentences and their positions, text segments are formed. As can be seen from the above, the embodiment of this specification describes the relationship between sentences in the target text segment as a relationship network diagram. Since two discontinuous sentences can be associated through the graph structure, it can break through the distance limit between sentences and establish a connection between any two highly semantically related sentences in a long text, and can handle nonlinear and complex text structures. In addition, by introducing the community detection analysis graph structure, it is possible to comprehensively analyze the relationship between all sentences from a global perspective, find substructures with close connections in the graph, and then achieve the purpose of dividing semantically coherent and similar sentences, which can improve the accuracy of text segmentation.

[0112] In addition, compared with the common text segmentation method, the embodiments of this specification have obvious advantages.

[0113] Among them, compared with the fixed-length segmentation method and the recursive character segmentation method, the embodiment of this specification can adaptively analyze the semantic association between sentences and determine the segmentation points by introducing the embedded model to model the semantics of the text, which greatly reduces the probability of cutting off continuous semantic segments and merging unrelated segments.

[0114] Compared with the semantic segmentation method, the embodiment of this specification uses community detection instead of the method of finding similarity mutation points between consecutive sentences. First, it can establish the relationship between sentences that are not adjacent in the order of writing. Second, in the process of iterative optimization of modularity, community detection will comprehensively consider the connections and weights between all nodes, and find a better segmentation method from a global perspective. It can avoid the problem of erroneously adding segmentation points when interfering sentences such as semantic ambiguity and semantic deviation appear. For example, in the writing of technical documents, in order to supplement some additional information about technology and products, supplementary explanations may be inserted. These explanations may appear in the form of annotations, or they may be directly interspersed in the text, following certain transition words. For example, the technical manual for a network device, when introducing network configuration, states, "This section describes how to optimize device performance. First, log in to the system management interface, navigate to the network settings option, and adjust IP address assignments and subnet masks..." It then adds, "Also, ensure all device firmware is updated to the latest version to ensure optimal device performance and address security vulnerabilities." After this additional explanation, the document returns to the performance tuning section, "Next, we will conduct performance testing. This step uses built-in performance testing tools to measure network latency and packet loss..." In this example, conventional semantic segmentation methods cannot address the noise introduced by supplementary information, resulting in the fragmentation of the complete semantic unit of performance tuning. However, graph-based community detection can handle nonlinear structures and fully aggregate device performance tuning fragments, thereby providing more accurate reference information for RAG during the recall phase.

[0115] Compared to topic-node segmentation, graph-based community detection requires less computation. The core computation lies in obtaining an embedding vector for each sentence. This step eliminates the need to invoke a large language model or repeatedly calculate embedding vectors for text within a single topic. Furthermore, leveraging the parallel processing capabilities of the embedding model, batching multiple sentences into inputs can further accelerate segmentation.

[0116] According to another embodiment, a text segmentation device in natural language processing is also provided, which is used to execute the method provided in the embodiment of this specification. Figure 4 FIG. 1 is a schematic block diagram of a text segmentation device in natural language processing according to an embodiment. Figure 4 As shown, the apparatus 400 includes:

[0117] The first segmentation unit 41 is used to segment the target text segment into sentences and obtain sentence embedding vectors for each sentence;

[0118] A graph construction unit 42 is configured to construct a relationship network graph using the sentences obtained by the first segmentation unit 41 as nodes, wherein the connection edges between the nodes are determined based on the similarity between the sentence embedding vectors;

[0119] A detection unit 43 is configured to perform community detection on the relationship network graph obtained by the graph construction unit 42 and divide the nodes into different communities;

[0120] The second segmentation unit 44 is used to extract a number of sentences corresponding to nodes classified into the same community based on the community detection results obtained by the detection unit 43, and obtain the positions of the several sentences in the target text segment, and form text segments based on the several sentences and their positions.

[0121] Optionally, as an embodiment, the first segmentation unit 41 includes:

[0122] A search subunit, configured to search for preset punctuation marks in the target text segment;

[0123] The segmentation subunit is used to segment the target text segment at the position of the preset punctuation mark obtained by the search subunit to obtain each segmented sentence.

[0124] Furthermore, the preset punctuation marks include:

[0125] A period, question mark, exclamation point, or semicolon.

[0126] Optionally, as an embodiment, the first segmentation unit 41 is specifically used to input the target sentence into a first embedding model to obtain a sentence embedding vector of the target sentence; or, input the target sentence and the context sentence of the target sentence into a second embedding model to obtain a sentence embedding vector of the target sentence.

[0127] Optionally, as an embodiment, the graph construction unit 42 includes:

[0128] Determine a subunit for determining the similarity between the sentence embedding vectors corresponding to any two sentences, and use it as the similarity between the two sentences;

[0129] A sub-unit is constructed to correspond each sentence to a node, and whether there is a connecting edge between the corresponding two nodes is determined based on the similarity between the two sentences obtained by the sub-unit, so as to construct a relationship network diagram.

[0130] Furthermore, the determination subunit is specifically used to normalize the sentence embedding vector of any sentence to obtain a normalized vector of the sentence; and perform a dot product on the normalized vectors of any two sentences to obtain the similarity between the two sentences.

[0131] Furthermore, the construction subunit includes:

[0132] A search subunit is used to search for K second nodes that are ranked in the top K positions in similarity with the first node among other nodes;

[0133] The adding subunit is used to add a connecting edge between the first node and the second node if the similarity between the first node and the second node obtained by the searching subunit is greater than a preset threshold.

[0134] Optionally, as an embodiment, the relationship network graph is an undirected graph; or, the relationship network graph is a directed graph, and the direction of the connecting edge is determined according to the order of the two sentences it connects in the target text segment.

[0135] Optionally, as an embodiment, the detection unit 43 includes:

[0136] Initialization subunit, used to initialize each node into an independent community;

[0137] The moving subunit is used to try to move each node to the community to which its neighboring nodes belong. After each move, the modularity of the relationship network graph is calculated. If moving the node to the community of the neighboring node can improve the modularity, then this move is performed;

[0138] The graph updating subunit is used to treat the currently identified community as a new node after the change in modularity obtained by moving the subunit becomes stable, construct a new relationship network graph, and continue community detection.

[0139] Furthermore, the modularity is used to evaluate whether the currently divided community structure can maximize the difference between the connection density within the community and the expected random connection density.

[0140] Furthermore, the graph updating subunit is specifically used to update the edge weight of the connection edge between two new nodes so that it represents the connection strength between the communities corresponding to the two new nodes.

[0141] Through the device provided by the embodiment of this specification, the first segmentation unit 41 first segments the target text segment into sentences and obtains the sentence embedding vector of each sentence; then the graph construction unit 42 constructs a relationship network graph with sentences as nodes, wherein the connection edges between nodes are determined according to the similarity between the sentence embedding vectors; then the detection unit 43 performs community detection on the relationship network graph and divides the nodes into different communities; finally, the second segmentation unit 44 extracts a number of sentences corresponding to the nodes divided into the same community according to the results of the community detection, and obtains the positions of the several sentences in the target text segment, and forms text blocks based on the several sentences and their positions. As can be seen from the above, the embodiment of this specification describes the relationship between sentences in the target text segment as a relationship network graph. Since two discontinuous sentences can be associated through the graph structure, it can break through the distance limit between sentences and establish a connection between any two highly semantically related sentences in a long text, and can handle nonlinear and complex text structures. In addition, by introducing the community detection analysis graph structure, we can comprehensively analyze the relationship between all sentences from a global perspective, find substructures with close connections in the graph, and then achieve the purpose of dividing semantically coherent and similar sentences, which can improve the accuracy of text segmentation.

[0142] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.

[0143] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 2 The method described.

[0144] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0145] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text segmentation method in natural language processing, the method comprising: Split the target text segment into sentences and obtain the sentence embedding vector of each sentence; Build a relationship network graph with sentences as nodes, where the connection edges between nodes are determined by the similarity between sentence embedding vectors; Performing community detection on the relationship network graph and dividing the nodes into different communities; According to the results of community detection, several sentences corresponding to the nodes classified in the same community are extracted, and the positions of the several sentences in the target text segment are obtained. Based on the several sentences and their positions, text segments are formed.

2. The method according to claim 1, wherein The step of dividing the target text segment into sentences includes: Searching for preset punctuation marks in the target text segment; The target text segment is segmented at the locations of the preset punctuation marks to obtain segmented sentences.

3. The method according to claim 2, wherein: The preset punctuation marks include: A period, question mark, exclamation point, or semicolon.

4. The method according to claim 1, wherein The step of obtaining the sentence embedding vector of each sentence includes: Input the target sentence into the first embedding model to obtain a sentence embedding vector of the target sentence; or, The target sentence and the context sentences of the target sentence are input into the second embedding model to obtain a sentence embedding vector of the target sentence.

5. The method according to claim 1, wherein The step of constructing a relationship network diagram using sentences as nodes includes: Determine the similarity between the sentence embedding vectors corresponding to any two sentences and use it as the similarity between the two sentences; Each sentence corresponds to a node, and whether there is a connection edge between the corresponding two nodes is determined based on the similarity between the two sentences to construct a relationship network graph.

6. The method according to claim 5, wherein: Determining the similarity between sentence embedding vectors corresponding to any two sentences includes: Normalize the sentence embedding vector of any sentence to obtain the normalized vector of the sentence; Perform a dot product on the normalized vectors of any two sentences to obtain the similarity between the two sentences.

7. The method according to claim 5, wherein: The step of determining whether there is a connecting edge between two corresponding nodes according to the similarity between the two sentences includes: For the first node, find the K second nodes that are ranked in the top K similarities with it among all other nodes; If the similarity between the first node and the second node is greater than a preset threshold, a connecting edge is added between the first node and the second node.

8. The method of claim 1, wherein: The relationship network graph is an undirected graph; or, the relationship network graph is a directed graph, and the direction of the connecting edge is determined according to the order of the two sentences connected by it in the target text segment.

9. The method of claim 1, wherein: The performing community detection on the relationship network graph and dividing the nodes into different communities includes: Initialize each node as an independent community; For each node, try to move it to the community to which its neighboring nodes belong. After each move, calculate the modularity of the relationship network graph. If moving the node to the community of the neighboring node can improve the modularity, then perform this move. After the change in modularity becomes stable, the currently identified community is regarded as a new node, a new relationship network diagram is constructed, and community detection continues.

10. The method of claim 9, wherein: The modularity is used to evaluate whether the currently divided community structure can maximize the difference between the connection density within the community and the expected random connection density.

11. The method of claim 9, wherein: The construction of a new relationship network diagram includes: Update the edge weight of the connection between the two new nodes to make it represent the connection strength between the communities corresponding to the two new nodes.

12. A text segmentation device in natural language processing, the device comprising: The first segmentation unit is used to segment the target text segment into sentences and obtain the sentence embedding vector of each sentence; a graph construction unit, configured to construct a relationship network graph using the sentences obtained by the first segmentation unit as nodes, wherein the connection edges between the nodes are determined based on the similarity between the sentence embedding vectors; a detection unit, configured to perform community detection on the relationship network graph obtained by the graph construction unit, and divide the nodes into different communities; The second segmentation unit is used to extract a number of sentences corresponding to nodes classified into the same community based on the community detection results obtained by the detection unit, obtain the positions of the several sentences in the target text segment, and form text segments based on the several sentences and their positions.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 11.

14. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 11 is implemented.

Images