Text segmentation method, device, storage medium and electronic device

Abstract

This specification discloses a text segmentation method, device, storage medium and electronic device, which respectively perform dependency syntactic analysis on each initial text block obtained by segmenting the original knowledge document to obtain the semantic triples of each initial text block. For each initial text block, a semantic network graph is constructed based on the semantic triples of the initial text block. The nodes in the semantic network graph represent sentence components, and the edges represent dependency relationships. The semantic density of the initial text block is determined based on the graph density of the semantic network graph. In this method, the abstract semantic density is converted into a graph structure, and the semantic density is quantified by the graph density of the graph structure, so that text blocks with relatively uniform information content can be obtained according to the semantic density segmentation, thereby improving the effect of large-scale model enhanced retrieval.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a text segmentation method, device, storage medium, and electronic device. Background Art

[0002] Retrieval-Augmented Generation (RAG) technology combines the generation capabilities of an external knowledge base and a large language model (LLM) and is widely used in scenarios such as question-answering and content generation.

[0003] Text chunking, which involves dividing the vast raw knowledge documents in a knowledge base into retrieval-friendly "chunks," is a critical prerequisite for RAG implementation. The appropriate size of the chunks directly impacts retrieval accuracy. Currently, commonly used chunking strategies include fixed-size chunking and document structure chunking. Fixed-size chunking is too rigid, lacks semantics, and can easily result in key information being scattered across different chunks. Document structure chunking, while segmented according to natural sentences or paragraphs, is not suitable for large-scale model processing due to the diverse content of the articles and uneven chunk sizes.

[0004] In order to overcome the defects of the existing block segmentation method, this specification proposes a text block segmentation method. Summary of the Invention

[0005] This specification provides a text segmentation method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a text segmentation method, including:

[0008] Segment the original knowledge document to obtain initial text blocks;

[0009] Performing dependency syntactic analysis on each of the initial text blocks to obtain semantic triples of each of the initial text blocks, wherein the semantic triples represent dependency relationships between sentence components;

[0010] For each initial text block, construct a semantic network graph based on the semantic triples of the initial text block, wherein the nodes in the semantic network graph represent sentence components and the edges represent dependency relationships;

[0011] Determining the semantic density of the initial text block according to the graph density of the semantic network graph;

[0012] According to the semantic density of the initial text block, it is determined whether the initial text block needs to be adjusted and optimized. When adjustment and optimization are not required, the initial text block is determined to be the target text block obtained by the final segmentation of the original knowledge document.

[0013] Optionally, determining the semantic density of the initial text block according to the graph density of the semantic network graph specifically includes:

[0014] Performing entity recognition on the sentence components extracted from the semantic triples to determine the entity type of each sentence component, with corresponding entity weights being set for different entity types;

[0015] Calculating a semantic comprehensive weight according to the entity weight of each node entity type in the semantic network graph;

[0016] The graph density of the semantic network graph is weighted by the semantic comprehensive weight to obtain the semantic density of the initial text block.

[0017] Optionally, determining whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block specifically includes:

[0018] When the semantic density does not meet the specified condition, the initial text block is further segmented to obtain sub-text blocks until the semantic density of the sub-text blocks meets the specified condition, and the sub-text blocks are used as target text blocks obtained by the final segmentation of the original knowledge document;

[0019] When the semantic density meets the specified condition, the initial text block is used as the target text block obtained by the final segmentation of the original knowledge document.

[0020] Optionally, determining whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block specifically includes:

[0021] Calculating a semantic score of the initial text block according to the information entropy of the initial text block and the semantic density of the initial text block;

[0022] According to the semantic score, it is determined whether the initial text block needs to be adjusted and optimized.

[0023] Optionally, the initial text block is a document-level segmentation result;

[0024] The initial text block is further divided into sub-text blocks, specifically including:

[0025] According to the semantics of the paragraphs contained in the initial text block, the initial text block is divided into a plurality of paragraph-level sub-text blocks;

[0026] According to the sentence semantics contained in each paragraph-level sub-text block, the paragraph-level sub-text block is further divided into multiple sentence-level sub-text blocks.

[0027] Optionally, when the semantic density does not meet a specified condition, the initial text block is further segmented to obtain sub-text blocks, specifically including:

[0028] Extracting important nodes from the semantic network graph, where the degree of the important nodes is greater than a specified threshold;

[0029] Determine a subgraph within a preset number of hops centered on the important node;

[0030] According to the dependency relationship between the sentence components represented by the subgraph, the text description content is parsed to obtain the subtext block.

[0031] Optionally, the method further includes:

[0032] For each target text block obtained by finally segmenting the determined original knowledge document, the unclear information in the target text block is completed according to the context of the target text block, and the completed target text block is used for enhanced retrieval.

[0033] This specification provides a text segmentation device, the device comprising:

[0034] The initial segmentation module segments the original knowledge document to obtain initial text blocks;

[0035] A syntactic analysis module performs dependency syntactic analysis on each of the initial text blocks to obtain semantic triples of each of the initial text blocks, wherein the semantic triples represent dependency relationships between sentence components;

[0036] A graph construction module, for each initial text block, constructs a semantic network graph based on the semantic triples of the initial text block, wherein the nodes in the semantic network graph represent sentence components and the edges represent dependency relationships;

[0037] A semantic density calculation module, which determines the semantic density of the initial text block according to the graph density of the semantic network graph;

[0038] The adjustment module determines whether the initial text block needs to be adjusted and optimized according to the semantic density of the initial text block. When adjustment and optimization are not required, the initial text block is determined to be the target text block obtained by the final segmentation of the original knowledge document.

[0039] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned text segmentation method is implemented.

[0040] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned text segmentation method when executing the program.

[0041] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0042] In the text segmentation method provided herein, dependency syntactic analysis is performed on each initial text block obtained by segmenting an original knowledge document to obtain semantic triples for each initial text block. For each initial text block, a semantic network graph is constructed based on the semantic triples of the initial text block. Nodes in the semantic network graph represent sentence components, and edges represent dependency relationships. The semantic density of the initial text block is determined based on the graph density of the semantic network graph.

[0043] In this method, the abstract semantic density is converted into a graph structure, and the semantic density is quantified by the graph density of the graph structure, so that text blocks with relatively uniform information content can be obtained according to the semantic density segmentation, thereby improving the effect of large-scale model enhanced retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0045] Figure 1 This is a flowchart of a text segmentation method in this manual;

[0046] Figure 2 A schematic diagram of a semantic network diagram provided in an embodiment of this specification;

[0047] Figure 3 This is a diagram of an enhanced retrieval process provided in an embodiment of this specification;

[0048] Figure 4 A schematic diagram of a text segmentation device provided in this specification;

[0049] Figure 5 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0051] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0052] Figure 1 The flowchart of a text segmentation method in this specification includes the following steps:

[0053] S100: Segment the original knowledge document to obtain initial text blocks.

[0054] All steps in the text segmentation method provided in this specification can be implemented by any electronic device with computing capabilities, such as a terminal, server, etc. For ease of description, the text segmentation method provided in this specification is described below using only the server as the execution entity.

[0055] In this specification, the original knowledge document is first segmented to obtain initial text blocks. This segmentation is a rough segmentation, which can be segmented according to structures such as documents, paragraphs, sentences, etc., or segmented by fixed length, or segmented based on semantics using a language model, or any other existing segmentation method.

[0056] S102: performing dependency syntactic analysis on each of the initial text blocks to obtain semantic triples of each of the initial text blocks, wherein the semantic triples represent dependency relationships between sentence components.

[0057] After obtaining the initial text block, a dependency syntax analysis tool is used to perform dependency syntax analysis on the initial text block.

[0058] Dependency parsing tools can identify words in an initial text block, label them as sentence components, and identify dependency relationships between different words. These dependency relationships represent the grammatical connection between one word (core word) and another (dependent word). Grammatical connection types include subject-verb relationships, object relationships, modifier relationships, and predicate relationships.

[0059] This dependency relationship is usually expressed as a triple in the format of (dependency word, core word, grammatical connection type). In a triple, the dependency word is a subordinate sentence component that is constrained by the core word, and the core word is a core sentence component that controls the dependency word.

[0060] For example, for the sentence "Deep learning has significantly promoted the development of computer vision, and convolutional neural networks have achieved breakthroughs in image classification tasks," the semantic triples obtained after dependency parsing include: (deep learning, promotion, subject relationship), (development, promotion, object relationship), and (computer vision, onset, modification relationship).

[0061] S104: For each initial text block, a semantic network graph is constructed according to the semantic triples of the initial text block, wherein nodes in the semantic network graph represent sentence components and edges represent dependency relationships.

[0062] The semantic triples of the initial text block represent the semantic structure within the text block. In the triples, the connection between the dependent word and the central word is directional. Based on this directional connection, a semantic network diagram can be constructed using the words in the sentence as nodes. This semantic network diagram can be used to visualize abstract semantics and calculate semantic density.

[0063] Continuing with the above example, for the sentence "Deep learning has significantly promoted the development of computer vision, and convolutional neural networks have made breakthrough progress in image classification tasks", after extracting semantic triples, the semantic network graph constructed is as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of a semantic network diagram provided in an embodiment of this specification.

[0064] exist Figure 2 The graph contains four nodes: "Deep Learning," "Computer Vision," "Development," and "Promotion." The dependent and core words in each triple are transformed into nodes as sentence components, and the edges represent the grammatical connection between the dependent and core words in each triple.

[0065] S106: Determine the semantic density of the initial text block according to the graph density of the semantic network graph.

[0066] The graph density of graph data is the ratio of the actual number of edges in the graph data to the maximum possible number of edges of the graph data. It can measure the density of the connections between nodes in the graph network.

[0067] In this specification, semantic triples are used to transform the semantic relationships of an original text block into the topological structure of a semantic network graph. The more information points a text block contains, the more semantic entity words that can be extracted, and the more complex the relationships between these words, indicating a higher semantic density. Therefore, the graph density of a semantic network graph provides a theoretical basis for characterizing the semantic density of a text block.

[0068] For a semantic network graph, its semantic density The calculation method can be expressed as follows:

[0069]

[0070] in, represents the total number of edges in the semantic network graph, and n represents the total number of nodes in the semantic network graph. It represents the total number of edges contained in a fully connected graph of n nodes, that is, the maximum possible number of edges of the graph data of n nodes.

[0071] S108: Determine whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block. If no adjustment and optimization is needed, determine that the initial text block is the target text block obtained by the final segmentation of the original knowledge document.

[0072] The server may determine whether the initial text block meets a specified condition based on a pre-specified density threshold.

[0073] If the semantic density of the initial text block is less than the density threshold, it indicates that the information in the initial text block is relatively scattered and needs further segmentation. If the semantic density of the initial text block is not less than the density threshold, it indicates that the initial text block meets the specified conditions and is used as the target text block for the final segmentation.

[0074] For initial text blocks that require further segmentation, an iterative segmentation approach can be used to segment the initial text block into sub-text blocks. The semantic density of each sub-text block is then calculated using the semantic density determination method described above. If the semantic density of a sub-text block does not meet the specified criteria, the sub-text block is segmented again until its semantic density meets the specified criteria, at which point the sub-text block is determined as the target text block.

[0075] After the segmentation is completed, the target text blocks divided into the original knowledge document are vectorized and a retrieval library is constructed so that in the text generation task of the large model, the target text blocks related to the input text can be found in the retrieval library through enhanced retrieval.

[0076] Because the text segmentation method in this manual uses semantic density as the measurement standard for text blocks, the information distribution of each target text block obtained by segmentation is relatively uniform. During vector embedding, the embedded vectors of the target text blocks obtained can better represent the information differences between different target text blocks, thereby enhancing the accuracy of vector retrieval in the retrieval library, providing domain knowledge information for the large model, and enhancing the accuracy of the output results of the large model.

[0077] Based on the above Figure 1The text segmentation method performs dependency syntactic analysis on each initial text block obtained by segmenting the original knowledge document to obtain the semantic triples of each initial text block. For each initial text block, a semantic network graph is constructed based on the semantic triples of the initial text block. The nodes in the semantic network graph represent sentence components, and the edges represent dependency relationships. The semantic density of the initial text block is determined based on the graph density of the semantic network graph. In this method, the abstract semantic density is converted into a graph structure, and the semantic density is quantified by the graph density of the graph structure, so that text blocks with relatively uniform information content can be obtained based on the semantic density segmentation, thereby improving the effect of large-scale model enhanced retrieval.

[0078] In one embodiment of the present specification, in the above step S104, the semantic density may also be determined in the following manner.

[0079] Specifically, the server performs entity recognition on the sentence components in the semantic triples to determine the entity type of each sentence component. Different entity types are pre-set with corresponding entity weights to reflect the degree of influence of different entity types on semantics.

[0080] Based on the above Figure 2 In the embodiment, after real-time recognition, it can be determined that the entity type of "computer vision" is "technology", the entity type of "deep learning" is "technology", the entity type of "promotion" is "action", and the entity type of "development" is "abstract".

[0081] Then, the semantic comprehensive weight is calculated according to the entity weight of each node entity type in the semantic network graph.

[0082] The comprehensive semantic weight can be calculated in various ways.

[0083] In one approach, based on the principle of information gain, the synergistic enhancement effect of the entity types of each node in a semantic network diagram can be reflected, and the product of the entity weights of the entity types of each node can be calculated as the semantic comprehensive weight of the semantic network diagram.

[0084] In another way, if the average effect of the entity type of each node in a semantic network diagram is considered, the average of the entity weights of the entity type of each node can be calculated as the semantic comprehensive weight of the semantic network diagram.

[0085] Finally, the graph density of the semantic network graph is weighted by the semantic comprehensive weight to obtain the semantic density of the initial text block.

[0086] In this embodiment, the node entity type can express the content importance of the sentence component represented by the node, and the graph density of the structural network diagram expresses the graph structure complexity of the semantic network diagram. Therefore, the semantic density calculated in this embodiment is a text block measurement indicator that combines structural complexity and content importance. When evaluating the segmentation quality of the initial text block based on the semantic density in this embodiment, a higher quality target text block can be obtained.

[0087] In the above step S108 , the information entropy of each initial text block may also be calculated.

[0088] The semantic score of the initial text block is calculated based on the information entropy of the initial text block and the semantic density of the initial text block. Based on the semantic score, it is determined whether the initial text block needs to be adjusted and optimized.

[0089] In this embodiment, the weighted sum of information entropy and semantic density can be used as the semantic score. Information entropy reflects the uncertainty of information in a text block, while semantic density indicates the degree of information dispersion in the text block. The semantic score combines the semantic characteristics of information entropy and semantic density to measure the quality of text segmentation along these two dimensions: information uncertainty and dispersion.

[0090] In one embodiment of the present specification, the initial text block can be defined as a document-level text block, that is, the initial text block is obtained by segmenting the original knowledge document at the document level. Specifically, a large model can be used to segment the original knowledge document using preset document segmentation prompt words.

[0091] In the above step S108, when the initial text block needs to be further segmented, a hierarchical segmentation method can be adopted.

[0092] First, according to the paragraph semantics contained in the initial text block, the initial text block is divided into multiple paragraph-level sub-text blocks.

[0093] Then, according to the sentence semantics contained in each paragraph-level sub-text block, the paragraph-level sub-text block is further divided into multiple sentence-level sub-text blocks.

[0094] This specification does not limit the implementation of paragraph-level segmentation and sentence-level segmentation. In an example, segmentation can be performed using a large model by setting paragraph-level segmentation prompt words and sentence-level segmentation prompt words.

[0095] In this embodiment, after obtaining short-circuit-level subtext blocks, the semantic density of paragraph-level subtext blocks can be calculated to determine whether to continue segmenting the paragraph-level subtext blocks. Alternatively, the semantic density calculation is not performed for paragraph-level subtext blocks. After obtaining sentence-level subtext blocks, the semantic density is calculated for the sentence-level subtext blocks to determine whether the sentence-level subtext blocks meet specified conditions. Sentence-level subtext blocks that meet the specified conditions are then determined as target text blocks for the final retrieval library construction, thereby constructing a more fine-grained retrieval library.

[0096] Based on this embodiment, the initial text block is a document-level text block, and the initial text block and paragraph-level sub-text blocks contain relatively more information.

[0097] The server can extract the main idea of ​​each text block from the initial text blocks and paragraph-level sub-text blocks obtained during the segmentation process. When constructing the search database, the main idea of ​​each text block is also vectorized and stored as knowledge information to enhance retrieval.

[0098] During enhanced retrieval, because the vector to be retrieved contains embedding vectors at three levels: document level, paragraph level, and sentence level, the query vector of the query information can be used to continue searching at each level and output retrieval results of different granularities.

[0099] The retrieval results of different granularities complement each other semantically, providing more accurate semantic information for the task of generating large models during enhanced retrieval, and assisting the large model in outputting more accurate results.

[0100] Figure 3 This is an enhanced retrieval process diagram provided in the embodiments of this specification, such as Figure 3 As shown, the search vector for an original knowledge document is constructed by concatenating the vectors of document-level text blocks, paragraph-level text blocks, and sentence-level text blocks. In response to an input query, a query vector is determined, and a search is performed against the vector database consisting of the search vectors to obtain result vectors similar to the query vector.

[0101] The result vector can return only the search results of similar sentence-level text blocks, or it can return the search results of three levels: document-level text blocks, paragraph-level text blocks, and sentence-level text blocks.

[0102] In one embodiment, for each target text block finally obtained, the unclear information in the target text block can be supplemented according to the context of the target text block, and the supplemented target text block can be used for enhanced retrieval.

[0103] Because in the process of text block division, according to language habits, the expressions of adjacent target text blocks may be related to each other, then a single target text block may contain unclear information, such as sentences containing pronouns, omitted sentences that depend on the context, etc.

[0104] This target text block needs to be combined with the context to determine the clear meaning. This embodiment completes the unclear information in each sentence based on the context of the sentence, so that each target text block can have a clear meaning.

[0105] In one embodiment of the present specification, in the above step S108, when the semantic density of an initial text block is less than a density threshold, information pruning may be performed on the initial text block to obtain a sub-text block.

[0106] Specifically, in the semantic network graph of the initial text block, nodes whose degrees are greater than a specified threshold are extracted as important nodes.

[0107] Then, a subgraph of the important node is obtained, centered on the important node and within a preset number of hops. The preset number of hops can be selected as needed, for example, 3 hops can be selected. Then, the connected nodes within the 3-hop range of the important node as the center form the subgraph of the important node.

[0108] The semantic network graph represents the entire semantic content of the initial text block, while the subgraph represents a relatively important portion of the semantic content of the initial text block. After obtaining the subgraph, the subgraph is reverse-parsed into text description content to obtain the subtext block.

[0109] The sub-text block of this embodiment includes relatively important information in the initial text block, and the degree of semantic dispersion is inevitably reduced.

[0110] In one embodiment of the present specification, during retrieval, in order to speed up the retrieval, the retrieval may be performed in the order of the embedding vector of the main text of the document block, the embedding vector of the main text of the paragraph block, and the embedding vector of the sentence block.

[0111] Specifically, for the vector to be retrieved in each document block, a search is performed in the vector embedding of the main text of the document block. When it is determined that there is a document-level result vector similar to the query vector of the query information in the document block, the paragraph-level search is continued in the document block. Otherwise, the search is switched to other document blocks.

[0112] When searching in the vector embeddings of each paragraph block corresponding to the document-level result vector, the embedding vector of the main text of each paragraph block is searched. When it is determined that there is a paragraph-level result vector similar to the query vector in the document block, that is, the embedding vector of the main text of the paragraph block, the paragraph-sentence level search is continued in the paragraph block. Otherwise, the search is switched to other paragraph blocks.

[0113] When performing sentence-level retrieval in the document block, the embedding vectors of each sentence block contained in the paragraph block are retrieved, and sentence blocks similar to the query vector are determined as the result vectors at the sentence level.

[0114] In one embodiment of the present specification, in the above step S108, after determining the target text block, the emotion tag of each target text block may be extracted, and the emotion tag may be stored in the retrieval library as the associated information of the target text block.

[0115] During enhanced retrieval, if the embedding vector of a target text block is retrieved as the result vector, the large model reads the sentiment label of the target text block and determines the sentiment polarity of the subsequent text generated based on the sentiment polarity represented by the sentiment label. This guides the large model output to be more controllable, ensuring that the model outputs more accurate results that match the sentiment polarity of the retrieval results.

[0116] For example, in a question-answering system, the user's original input is used as query information, and the target text block "The battery life is 10% shorter than the previous generation" is retrieved from the vector library. The sentiment label of this sentence block is "negative". Then, guided by this sentiment label, the large model is more inclined to generate negative answers, such as "The battery life of this model has decreased by 10%. It is recommended to give priority to alternative options."

[0117] The above is the text segmentation method provided in this specification. Based on the same idea, this specification also provides a corresponding text segmentation device, such as Figure 4 shown.

[0118] Figure 4 A schematic diagram of a text segmentation device provided in this specification specifically includes:

[0119] The initial segmentation module 200 is used to segment the original knowledge document to obtain initial text blocks;

[0120] Syntactic analysis module 202, configured to perform dependency syntactic analysis on each of the initial text blocks to obtain semantic triples of each of the initial text blocks, wherein the semantic triples represent dependency relationships between sentence components;

[0121] A graph construction module 204 is configured to construct a semantic network graph for each initial text block based on the semantic triples of the initial text block, wherein nodes in the semantic network graph represent sentence components and edges represent dependency relationships;

[0122] A semantic density calculation module 206 is configured to determine the semantic density of the initial text block based on the density of the semantic network graph;

[0123] The adjustment module 208 is used to determine whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block. When adjustment and optimization are not required, the initial text block is determined to be the target text block obtained by the final segmentation of the original knowledge document.

[0124] Optionally, the semantic density calculation module 206 is specifically used to perform entity recognition on the sentence components extracted from the semantic triples, determine the entity type of each sentence component, and set corresponding entity weights for different entity types. According to the entity weights of the entity types of each node in the semantic network diagram, the semantic comprehensive weight is calculated. The graph density of the semantic network diagram is weighted by the semantic comprehensive weight to obtain the semantic density of the initial text block.

[0125] Optionally, the adjustment module 208 is specifically used to continue to segment the initial text block into sub-text blocks when the semantic density does not meet the specified conditions, until the semantic density of the sub-text blocks meets the specified conditions, and use the sub-text blocks as the target text blocks obtained by the final segmentation of the original knowledge document; when the semantic density meets the specified conditions, use the initial text block as the target text block obtained by the final segmentation of the original knowledge document.

[0126] Optionally, the adjustment module 208 is specifically configured to calculate a semantic score of the initial text block based on the information entropy of the initial text block and the semantic density of the initial text block, and determine whether the initial text block needs to be adjusted and optimized based on the semantic score.

[0127] Optionally, the initial text block is a document-level segmentation result, and the adjustment module 208 is specifically used to divide the initial text block into a plurality of paragraph-level sub-text blocks according to the paragraph semantics contained in the initial text block, and further divide the paragraph-level sub-text block into a plurality of sentence-level sub-text blocks according to the sentence semantics contained in each paragraph-level sub-text block.

[0128] Optionally, the adjustment module 208 is specifically used to extract important nodes in the semantic network diagram, where the degree of the important nodes is greater than a specified threshold, determine a subgraph within a preset number of hops centered on the important nodes, and parse the text description content to obtain a sub-text block based on the dependency relationship of each sentence component represented by the subgraph.

[0129] Optionally, the adjustment module 208 is specifically used to complete the unclear information in each target text block obtained by the final segmentation of the original knowledge document according to the context of the target text block, and use the completed target text block for enhanced retrieval.

[0130] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a method for chunking text.

[0131] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] Improvements to a technology can be clearly categorized as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can integrate a digital system onto a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using software called a "logic compiler." This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a hardware description language (HDL). There are many types of HDL, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0133] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0134] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0135] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0144] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0146] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0147] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of this application.

Claims

1. A text segmentation method, characterized in that: include: Segment the original knowledge document to obtain initial text blocks; Performing dependency syntactic analysis on each of the initial text blocks to obtain semantic triples of each of the initial text blocks, wherein the semantic triples represent dependency relationships between sentence components; For each initial text block, construct a semantic network graph based on the semantic triples of the initial text block, wherein the nodes in the semantic network graph represent sentence components and the edges represent dependency relationships; Determining the semantic density of the initial text block according to the graph density of the semantic network graph; Determining whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block includes: When the semantic density is less than a pre-specified density threshold, the initial text block is further segmented into sub-text blocks until the semantic density of the sub-text blocks is less than the density threshold, and the sub-text blocks are used as target text blocks obtained by final segmentation of the original knowledge document; When the semantic density is not less than the density threshold, the initial text block is used as the target text block obtained by the final segmentation of the original knowledge document.

2. The method according to claim 1, wherein According to the density of the semantic network graph, the semantic density of the initial text block is determined, specifically including: Performing entity recognition on the sentence components extracted from the semantic triples to determine the entity type of each sentence component, with corresponding entity weights being set for different entity types; Calculating a semantic comprehensive weight according to the entity weight of each node entity type in the semantic network graph; The graph density of the semantic network graph is weighted by the semantic comprehensive weight to obtain the semantic density of the initial text block.

3. The method according to claim 1, wherein Determine whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block, specifically including: Calculating a semantic score of the initial text block according to the information entropy of the initial text block and the semantic density of the initial text block; According to the semantic score, it is determined whether the initial text block needs to be adjusted and optimized.

4. The method according to claim 1, wherein The initial text block is a document-level segmentation result; The initial text block is further divided into sub-text blocks, specifically including: According to the semantics of the paragraphs contained in the initial text block, the initial text block is divided into a plurality of paragraph-level sub-text blocks; According to the sentence semantics contained in each paragraph-level sub-text block, the paragraph-level sub-text block is further divided into multiple sentence-level sub-text blocks.

5. The method according to claim 1, wherein When the semantic density does not meet the specified conditions, the initial text block is further segmented to obtain sub-text blocks, specifically including: Extracting important nodes from the semantic network graph, where the degree of the important nodes is greater than a specified threshold; Determine a subgraph within a preset number of hops centered on the important node; According to the dependency relationship between the sentence components represented by the subgraph, the text description content is parsed to obtain the subtext block.

6. The method according to claim 1, wherein The method further comprises: For each target text block obtained by finally segmenting the determined original knowledge document, the unclear information in the target text block is completed according to the context of the target text block, and the completed target text block is used for enhanced retrieval.

7. A text segmentation device, characterized in that: include: The initial segmentation module segments the original knowledge document to obtain initial text blocks; A syntactic analysis module performs dependency syntactic analysis on each of the initial text blocks to obtain semantic triples of each of the initial text blocks, wherein the semantic triples represent dependency relationships between sentence components; A graph construction module, for each initial text block, constructs a semantic network graph based on the semantic triples of the initial text block, wherein the nodes in the semantic network graph represent sentence components and the edges represent dependency relationships; A semantic density calculation module, which determines the semantic density of the initial text block according to the graph density of the semantic network graph; The adjustment module determines whether the initial text block needs to be adjusted and optimized based on the semantic density of the initial text block, including: When the semantic density is less than a pre-specified density threshold, the initial text block is further segmented into sub-text blocks until the semantic density of the sub-text blocks is less than the density threshold, and the sub-text blocks are used as target text blocks obtained by final segmentation of the original knowledge document; When the semantic density is not less than the density threshold, the initial text block is used as the target text block obtained by the final segmentation of the original knowledge document.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images