Text block processing method, text block processing device, electronic equipment and medium
By performing feature analysis and dynamic matching segmentation tools on the text, the adaptability problem of text segmentation processing is solved, and efficient and reasonable segmentation of different types of text is achieved, ensuring the logical coherence and semantic integrity of the segmentation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-25
- Publication Date
- 2026-04-07
AI Technical Summary
In existing technologies, text segmentation processing cannot adapt to different types of text, resulting in unstable and unreasonable segmentation operations.
By performing feature analysis on the text to be processed, its type features are determined, and target segmentation tools are matched according to preset rules, including category analysis, format analysis, and content analysis. The priority order is text category, format features, and content features. Dynamic matching is performed on recursive segmentation tools, embedded model segmentation tools, or fixed-size segmentation tools for initial segmentation processing and segmentation optimization.
It achieves stable and reasonable segmentation of different types of text, improves segmentation efficiency and effectiveness, and ensures the semantic integrity and logical coherence of text segmentation results.
Smart Images
Figure CN121809447A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of text segmentation, and in particular to a text segmentation processing method, a text segmentation processing apparatus, an electronic device, and a medium. Background Technology
[0002] Text chunking is a key preprocessing technique for building knowledge bases or retrieval-augmented generation (RAG) systems. However, many related technologies employ fixed chunking strategies, which fail to achieve stable and reasonable chunking operations when dealing with different types of text. Summary of the Invention
[0003] This application provides a text segmentation processing method, a text segmentation processing device, an electronic device, and a medium, which match more suitable target segmentation tools for different types of text, thereby achieving stable and reasonable segmentation of different types of text.
[0004] In a first aspect, embodiments of this application provide a text segmentation processing method, which includes: performing feature analysis on the text to be processed to determine the type features of the text to be processed; matching a target segmentation tool according to a preset rule based on the type features of the text to be processed; and performing segmentation processing on the text to be processed using the target segmentation tool.
[0005] According to the aforementioned embodiments of the first aspect of this application, the step of performing feature analysis on the text to be processed to determine the type features of the text to be processed includes: performing category analysis, format analysis, and content analysis on the text to be processed to determine the text category, format features, and content features of the text to be processed.
[0006] According to any of the foregoing embodiments of the first aspect of this application, the step of matching a target segmentation tool according to the type characteristics of the text to be processed and a preset rule includes: matching a target segmentation tool according to the type characteristics of the text to be processed and prioritizing it in the order of text category, format characteristics, and content characteristics.
[0007] According to any of the foregoing embodiments of the first aspect of this application, the content features include at least one of the following: hierarchical features, paragraph length, special format, or keywords; the step of matching a target segmentation tool according to the type features of the text to be processed and a preset rule includes: firstly determining whether the type features include the hierarchical features; if the type features include the hierarchical features, then confirming the recursive segmentation tool as the target segmentation tool; if the type features do not include the hierarchical features, and the paragraph length is higher than a first threshold, then confirming the embedded model segmentation tool as the target segmentation tool; if the type features do not include the hierarchical features, and the paragraph length is lower than a second threshold, then confirming the fixed-size segmentation tool as the target segmentation tool, wherein the second threshold is less than the first threshold.
[0008] According to any of the foregoing embodiments of the first aspect of this application, the step of performing feature analysis on the text to be processed to determine the type features of the text to be processed includes: performing feature analysis on the text to be processed to determine that the text to be processed includes at least two sub-texts with different type features; the step of matching a target segmentation tool according to the type features of the text to be processed and a preset rule includes: matching a target segmentation tool corresponding to each sub-text according to the preset rule based on the type features of each sub-text.
[0009] According to any of the foregoing embodiments of the first aspect of this application, the text segmentation processing method further includes: forming a segmentation strategy by matching the target segmentation tool according to the type characteristics of the text to be processed, and caching and learning multiple segmentation strategies.
[0010] According to any of the foregoing embodiments of the first aspect of this application, the step of segmenting the text to be processed by the target segmentation tool includes: performing initial segmentation processing on the text to be processed by the target segmentation tool to obtain an initial segmentation result; extracting various specific information from the initial segmentation result; and optimizing the initial segmentation result based on the specific information to obtain a text segmentation result.
[0011] According to any of the foregoing embodiments of the first aspect of this application, the initial segmentation result includes a plurality of initial segments arranged in sequence, and the extraction of various specific information from the initial segmentation result includes: extracting the block number and semantic level of each initial segment; extracting the opening sentence and closing sentence of each initial segment; extracting specific terms in each initial segment and the descriptive position of the specific terms; and extracting logical connectors in each initial segment.
[0012] According to any of the foregoing embodiments of the first aspect of this application, the step of optimizing the initial segmentation result based on the specific information to obtain a text segmentation result includes: performing a first segmentation optimization on adjacent initial segments based on the specific information, wherein if there is a logical jump between the opening sentence of the later initial segment and the opening sentence of the earlier initial segment, the adjacent initial segments with the logical jump are merged, or a transition sentence is added to the later initial segment with the logical jump.
[0013] According to any of the foregoing embodiments of the first aspect of this application, the step of optimizing the initial segmentation result based on the specific information to obtain the text segmentation result includes: performing a second segmentation optimization on each adjacent initial segmentation, wherein, in each pair of adjacent initial segments, a summary text is generated based on the content of the previous initial segmentation and added before the opening sentence of the subsequent initial segmentation.
[0014] According to any of the foregoing embodiments of the first aspect of this application, the step of optimizing the initial segmentation result based on the specific information to obtain the text segmentation result includes: performing a third segmentation optimization within each initial segmentation based on the specific information, wherein if it is determined that there is a pronoun in the initial segmentation based on the specific term, then each pronoun is replaced with the actual term it represents.
[0015] According to any of the foregoing embodiments of the first aspect of this application, the step of optimizing the initial segmentation result based on the specific information to obtain a text segmentation result includes: performing a fourth segmentation optimization on large segments and small segments based on the specific information, wherein the large segment is the initial segment with a word count greater than a third threshold, the small segment is the initial segment with a word count less than a fourth threshold, and the fourth threshold is less than the third threshold, wherein if the initial segment is determined to be a multi-level segment based on the semantic level, the specific term, and the logical connector, the multi-level segment is split into at least two text segments according to the level, wherein if the small segment is determined to be a supplementary description of the adjacent preceding initial segment based on the semantic level, the specific term, and the logical connector, the small segment is merged into the adjacent preceding initial segment.
[0016] Secondly, embodiments of this application provide a text segmentation processing apparatus, characterized in that it includes: a text parsing module, used to perform feature parsing on the text to be processed and determine the type features of the text to be processed; a segmentation tool matching module, used to match a target segmentation tool according to a preset rule based on the type features of the text to be processed; and a segmentation processing module, used to perform segmentation processing on the text to be processed using the target segmentation tool.
[0017] According to the foregoing embodiments of the second aspect of this application, the text parsing module is configured to: perform category parsing, format parsing, and content parsing on the text to be processed, and determine the text category, format features, and content features of the text to be processed.
[0018] According to any of the foregoing embodiments of the second aspect of this application, the segmentation tool matching module is configured to: match a target segmentation tool according to the type characteristics of the text to be processed, in the order of priority: text category, format characteristics, and content characteristics.
[0019] According to any of the foregoing embodiments of the second aspect of this application, the content features include at least one of the following: hierarchical features, paragraph length, special format, or keywords; the segmentation tool matching module is configured to: firstly determine whether the type feature includes the hierarchical feature; if the type feature includes the hierarchical feature, then the recursive segmentation tool is identified as the target segmentation tool; if the type feature does not include the hierarchical feature, and the paragraph length is higher than a first threshold, then the embedded model segmentation tool is identified as the target segmentation tool; if the type feature does not include the hierarchical feature, and the paragraph length is lower than a second threshold, then the fixed-size segmentation tool is identified as the target segmentation tool, wherein the second threshold is less than the first threshold.
[0020] According to any of the foregoing embodiments of the second aspect of this application, the text parsing module is configured to: perform feature parsing on the text to be processed to determine that the text to be processed includes at least two sub-texts with different type features; the block matching module is configured to: match the target block matching tool corresponding to each sub-text according to the type feature of each sub-text and the preset rules.
[0021] According to any of the foregoing embodiments of the second aspect of this application, the segmentation tool matching module is configured to: form a segmentation strategy by matching the target segmentation tool according to the type characteristics of the text to be processed, and cache and learn multiple segmentation strategies.
[0022] According to any of the foregoing embodiments of the second aspect of this application, the segmentation processing module includes: an initial segmentation submodule, used to perform initial segmentation processing on the text to be processed by the target segmentation tool to obtain an initial segmentation result; an information extraction submodule, used to extract various specific information from the initial segmentation result; and a segmentation optimization submodule, used to perform segmentation optimization on the initial segmentation result based on the specific information to obtain a text segmentation result.
[0023] According to any of the foregoing embodiments of the second aspect of this application, the information extraction submodule is configured to: extract the block number and semantic level of each initial block; extract the opening sentence and closing sentence of each initial block; extract specific terms and their descriptive positions in each initial block; and extract logical connectors in each initial block.
[0024] According to any of the foregoing embodiments of the second aspect of this application, the block optimization submodule is configured to: perform a first block optimization on adjacent initial blocks based on the specific information, wherein if there is a logical jump between the opening sentence of the later initial block and the opening sentence of the earlier initial block, the adjacent initial blocks with the logical jump are merged, or a transition sentence is added to the later initial block with the logical jump.
[0025] According to any of the foregoing embodiments of the second aspect of this application, the block optimization submodule is configured to: perform a second block optimization on each adjacent initial block, wherein, in each pair of adjacent initial blocks, a summary text is generated based on the content of the previous initial block and added before the opening sentence of the subsequent initial block.
[0026] According to any of the foregoing embodiments of the second aspect of this application, the block optimization submodule is configured to: perform third block optimization within each of the initial blocks based on the specific information, wherein if it is determined that there is a pronoun in the initial block according to the specific term, then each pronoun is replaced with the actual term it represents.
[0027] According to any of the foregoing embodiments of the second aspect of this application, the block optimization submodule is configured to: perform a fourth block optimization on large blocks and small blocks based on the specific information, wherein the large block is the initial block with a word count greater than a third threshold, the small block is the initial block with a word count less than a fourth threshold, and the fourth threshold is less than the third threshold; wherein, if the initial block is determined to be a multi-level block based on the semantic level, the specific term, and the logical connector, the multi-level block is split into at least two text blocks according to the level; wherein, if the small block is determined to be a supplementary description of the adjacent preceding initial block based on the semantic level, the specific term, and the logical connector, the small block is merged into the adjacent preceding initial block.
[0028] Thirdly, embodiments of this application provide an electronic device, which includes a memory and at least one processor. The memory is electrically connected to the at least one processor, and the memory stores instructions. The at least one processor invokes the instructions in the memory to cause the electronic device to execute a text segmentation processing method according to any of the foregoing embodiments of the first aspect of this application.
[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a processor, implement a text segmentation processing method according to any of the foregoing embodiments of the first aspect of this application.
[0030] According to the text segmentation processing method of this application embodiment, feature analysis is performed on the text to be processed to determine the type features of the text to be processed. Based on the type features of the text to be processed, a target segmentation tool is matched according to preset rules. Therefore, more suitable target segmentation tools can be matched for texts with different type features. When the text to be processed has different content types or different hierarchical content, different type features will be determined accordingly. The text segmentation processing method of this application embodiment matches the corresponding target segmentation tool for texts with different type features, which facilitates adaptive processing and efficient segmentation of texts with different type features. After matching the corresponding target segmentation tool, the text to be processed is segmented using the target segmentation tool. While ensuring the efficiency of text segmentation processing, the effect of text segmentation can be further optimized. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0032] Figure 1 This is a flowchart of an embodiment of the text segmentation processing method of this application; Figure 2 This is a flowchart of the steps for segmenting the text to be processed using a target segmentation tool in one embodiment of the text segmentation processing method of this application. Figure 3 This is a schematic diagram of the structure of an embodiment of the text segmentation processing apparatus of this application; Figure 4 This is a schematic diagram of the structure of the block processing module in one embodiment of the text block processing apparatus of this application; Figure 5 This is a flowchart illustrating the text segmentation processing device according to an embodiment of the present application, where the text to be processed is segmented into blocks. Figure 6 This is a schematic diagram of the hardware structure of an embodiment of the electronic device of this application.
[0033] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and the accompanying drawings. Detailed Implementation
[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0035] It should be noted that the descriptions using terms such as "first" and "second" in this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0036] This application provides a text segmentation processing method. Figure 1 This is a flowchart of an embodiment of the text segmentation processing method of this application. The text segmentation processing method includes steps S110 to S140.
[0037] In step S110, feature parsing is performed on the text to be processed to determine the type features of the text to be processed.
[0038] In some embodiments, the step S110 of performing feature parsing on the text to be processed to determine the type features of the text to be processed includes: performing category parsing, format parsing, and content parsing on the text to be processed to determine the text category, format features, and content features of the text to be processed.
[0039] In one example, the text to be processed is parsed to determine its text category, such as product manual, case studies, technical standards, regulations, etc.
[0040] In one example, the text to be processed is parsed to determine its format characteristics, which may include, for example, word, excel, ppt, pdf, markdown, txt, etc.
[0041] In some embodiments, content features include at least one of the following: hierarchical features, paragraph length, special formatting, or keywords.
[0042] In one example, content parsing of the text to be processed includes hierarchical parsing of the text to determine its hierarchical features. Hierarchical features can be numbering, headings, or a combination of numbering and headings, etc. Hierarchical features are illustrated with an example, such as "1. Detailed introduction of technical background", which is a combination of numbering and headings.
[0043] In one example, content parsing of the text to be processed includes parsing the paragraph lengths of the text to determine the paragraph lengths. Based on the paragraph length hierarchy, it can be determined whether the paragraphs in the text to be processed are long narratives or short summaries.
[0044] In one example, content parsing of the text to be processed includes parsing the text for specific formats, thereby determining the specific format of the text. Specific formats include, for example, tables, figure captions, and code blocks.
[0045] In one example, content parsing of the text to be processed includes keyword parsing, which helps to identify the keywords in the text. In one example, by identifying keywords such as "technical solution" and "invention patent" in the text, it is possible to help identify the text category of the text to be processed.
[0046] In step S120, the target segmentation tool is matched according to the type characteristics of the text to be processed and a preset rule.
[0047] In this embodiment, the preset rules include the mapping relationship between combinations of various types of features and various optional segmentation tools. In one example, the preset rules include the mapping relationship between various combinations of text categories, format features, and content features and various optional segmentation tools.
[0048] In one example, optional segmentation tools include fixed-size segmentation tools, recursive segmentation tools, embedded model segmentation tools, etc.
[0049] In some embodiments, step S120, which matches a target segmentation tool according to a preset rule based on the type characteristics of the text to be processed, includes: matching the target segmentation tool according to the type characteristics of the text to be processed, in order of priority: text category, format characteristics, and content characteristics. That is, the segmentation tool can be selected first based on the text category of the text to be processed to obtain at least one relatively matching segmentation tool, then the at least one relatively matching segmentation tool can be further selected based on the format characteristics to obtain at least one more matching segmentation tool, and then the at least one more matching segmentation tool can be further selected based on the content characteristics to obtain the target segmentation tool.
[0050] In some embodiments, content features include at least one of the following: hierarchical features, paragraph length, special format, or keywords. In some embodiments, step S120 may include: first determining whether the type feature includes hierarchical features; if the type feature includes hierarchical features, then the recursive segmentation tool is identified as the target segmentation tool; if the type feature does not include hierarchical features and the paragraph length is higher than a first threshold, then the embedded model segmentation tool is identified as the target segmentation tool; if the type feature does not include hierarchical features and the paragraph length is lower than a second threshold, then the fixed-size segmentation tool is identified as the target segmentation tool, wherein the second threshold is less than the first threshold.
[0051] In the above embodiments, hierarchical features are detected first. If clear hierarchical features are found, the recursive segmentation tool is prioritized as the target segmentation tool. If no clear hierarchical features are found, the paragraph length is determined. If the paragraph length is higher than a first threshold and no clear hierarchical features are found, the embedded model segmentation tool is prioritized as the target segmentation tool. If the parsed content structure is chaotic and the content is short (i.e., the paragraph length is lower than a second threshold and no clear hierarchical features are found), the fixed-size segmentation tool is prioritized as the target segmentation tool. This process reflects a dynamic matching mechanism for identifying the target segmentation tool, dynamically determining the target segmentation tool based on the applicable scenarios of multiple segmentation tools.
[0052] In some optional embodiments, step S110 may include: performing feature analysis on the text to be processed to determine that the text to be processed includes at least two sub-texts with different type features. In this case, step S120 may include: matching a target segmentation tool corresponding to each sub-text according to a preset rule based on the type features of each sub-text. Therefore, when performing feature analysis on the text to be processed to determine that it includes at least two sub-texts with different type features, a corresponding target segmentation tool can be matched for each sub-text, so that sub-texts with different type features can be matched with a more suitable target segmentation tool, which facilitates further improvement in the segmentation efficiency and segmentation effect of the text to be processed.
[0053] In step S130, the text to be processed is segmented using the target segmentation tool.
[0054] In some embodiments, after the target segmentation tool is matched based on step S132, step S130 includes calling the target segmentation tool to segment the text to be processed.
[0055] Figure 2 This is a flowchart illustrating the steps of segmenting the text to be processed using a target segmentation tool in one embodiment of the text segmentation processing method of this application. In some embodiments, step S130 may include steps S131 to S133.
[0056] In step S131, the target segmentation tool performs initial segmentation on the text to be processed to obtain the initial segmentation result.
[0057] In step S132, various specific information is extracted from the initial block division results.
[0058] The initial segmentation result includes multiple initial segments arranged in sequence. In some embodiments, step S132, which extracts various specific information from the initial segmentation result, includes: extracting the segment number and semantic level of each initial segment; extracting the opening sentence and closing sentence of each initial segment; extracting specific terms and their descriptive positions in each initial segment; and extracting logical connectors in each initial segment.
[0059] In step S133, the initial segmentation result is optimized based on specific information to obtain the text segmentation result.
[0060] In some embodiments, a large language model is used to optimize the initial chunking results based on specific information. The large language model possesses contextual understanding and reasoning capabilities, enabling it to analyze text content and logical structure. In some embodiments, the large language model can be used to achieve the following chunking optimization tasks: performing inter-chunk logical integrity analysis and chunking optimization of the initial chunks; performing intra-chunk semantic integrity analysis and chunking optimization of the initial chunks; and performing analysis and chunking optimization on whether large chunks should be further split and whether small chunks should be merged, wherein large chunks are initial chunks with a word count greater than a third threshold, and small chunks are initial chunks with a word count less than a fourth threshold.
[0061] In some embodiments, step S133 includes: performing a first block optimization on adjacent initial blocks based on specific information, wherein if there is a logical jump between the opening sentence of the later initial block and the opening sentence of the earlier initial block, the adjacent initial blocks with the logical jump are merged, or a transition sentence is added to the later initial block with the logical jump.
[0062] By performing the first block optimization, it is possible to achieve logical integrity analysis and block optimization between the initial blocks.
[0063] In one example, initial block A and initial block B are the preceding and following initial blocks in adjacent initial blocks, respectively. For example, the ending sentence of initial block A is "Therefore, the scheme has higher stability...", and the beginning sentence of initial block B is "In addition, regarding the support structure of the scheme...". At this time, the block optimization agent determines that initial block A and initial block B are logically coherent, so initial block A and initial block B do not need further optimization.
[0064] In another example, initial block C and initial block D are the preceding and following initial blocks in an adjacent sequence. For example, the ending sentence of initial block C contains conclusive expressions such as "therefore" or "in conclusion," while the beginning sentence of initial block D begins to describe a new topic, such as "In addition, there is another solution...". In this case, the block optimization agent determines that the logic of initial blocks C and D is inconsistent, i.e., a logical jump. Optionally, initial blocks C and D can be merged to preserve the logical chain; alternatively, initial blocks C and D can remain separate, with a transitional sentence added before the beginning sentence of initial block D.
[0065] In some embodiments, step S133 includes: performing a second block optimization on each adjacent initial block, wherein in each pair of adjacent initial blocks, a summary text is generated based on the content of the previous initial block and added before the opening sentence of the subsequent initial block.
[0066] By performing a second block optimization, context information can be supplemented to the next initial block in each adjacent initial block. Optionally, the context information can be accumulated and passed on.
[0067] It should be noted that the second block optimization for each adjacent initial block can be performed in parallel with other specific block optimization steps. That is, regardless of whether the initial blocks have been merged, split, or kept, a summary text is generated based on the content of the previous initial block and added before the first sentence of the next initial block.
[0068] In some embodiments, step S133 includes: performing third block optimization within each initial block based on specific information, wherein if it is determined that there are pronouns in the initial block according to specific terms, each pronoun is replaced with the actual term it represents.
[0069] By performing third-level block optimization, semantic integrity analysis and block optimization within the initial blocks can be achieved. In the above embodiment, third-level block optimization enables block optimization after keyword incompleteness checking within the initial blocks.
[0070] For example, if a technical keyword appears frequently in the preceding initial block, such as the technical keyword "supporting institution" appearing frequently in the preceding initial block, while the preceding initial block mainly contains pronouns such as "its" or "the institution," then the following initial block is determined to be a missing block. In this case, the pronouns in the missing block are replaced with the actual terms they represent; for example, "the institution" is replaced with "the supporting institution."
[0071] In some embodiments, step S133 includes: performing a fourth block optimization on large blocks and small blocks based on specific information, wherein the large block is an initial block with a word count greater than a third threshold, and the small block is an initial block with a word count less than a fourth threshold, the fourth threshold being less than the third threshold, wherein if the initial block is determined to be a multi-level block based on semantic level, specific terms, and logical connectors, the multi-level block is split into at least two text blocks according to the level, wherein if the small block is determined to be a supplementary description of the adjacent previous initial block based on semantic level, specific terms, and logical connectors, the small block is merged into the adjacent previous initial block.
[0072] By performing fourth-level block optimization, it is possible to analyze and optimize whether large blocks should be further split and whether small blocks should be merged. In some embodiments, when merging small blocks into adjacent previous initial blocks, if the logic at the merge point is not logically consistent, a connecting clause can be added.
[0073] In the above embodiments, after steps S131 to S133, the target segmentation tool performs initial segmentation on the text to be processed, obtaining initial segmentation results. Various specific information is extracted from the initial segmentation results, and segmentation optimization is performed based on this specific information to obtain text segmentation results. This ensures that each text segment of the final text segmentation result can be understood independently, without severing key contextual content and maintaining semantic integrity. In the above embodiments, after step S130, the connection between text segments is optimized, making the content natural and fluent, ensuring smooth logical transitions. In the above embodiments, after step S130, unnecessary fragmented blocks (overly segmented small blocks) can be merged, and lengthy and difficult-to-understand large blocks can be broken down, eliminating fragmentation and repetition.
[0074] In the above embodiment, after step S130, the initial segmentation result is optimized to obtain the text segmentation result. The text segmentation result can be output to a downstream system or stored in a server or database. The downstream system is, for example, a knowledge base or a retrieval-augmented generation (RAG) question answering system.
[0075] According to the text segmentation processing method of this application embodiment, feature analysis is performed on the text to be processed to determine the type features of the text to be processed. Based on the type features of the text to be processed, a target segmentation tool is matched according to preset rules. Therefore, more suitable target segmentation tools can be matched for texts with different type features. When the text to be processed has different content types or different hierarchical content, different type features will be determined accordingly. The text segmentation processing method of this application embodiment matches the corresponding target segmentation tool for texts with different type features, which facilitates adaptive processing and efficient segmentation of texts with different type features. After matching the corresponding target segmentation tool, the text to be processed is segmented using the target segmentation tool. While ensuring the efficiency of text segmentation processing, the effect of text segmentation can be further optimized.
[0076] This application also provides a text segmentation processing device. Figure 3 This is a schematic diagram of the structure of an embodiment of the text segmentation processing device of this application. The text segmentation processing device includes a text parsing module 110, a segmentation tool matching module 120, and a segmentation processing module 130.
[0077] The text parsing module 110 is used to perform feature parsing on the text to be processed and determine the type features of the text to be processed.
[0078] The chunking tool matching module 120 is used to match the target chunking tool according to the type characteristics of the text to be processed and according to preset rules.
[0079] The chunking module 130 is used to perform chunking processing on the text to be processed using the target chunking tool.
[0080] According to the text segmentation processing apparatus of this application embodiment, the text parsing module 110 performs feature parsing on the text to be processed to determine the type features of the text to be processed. The segmentation tool matching module 120 matches the target segmentation tool according to the type features of the text to be processed and a preset rule. Therefore, it can match more suitable target segmentation tools for texts with different type features. When the text to be processed has different content types or different hierarchical content, different type features will be determined accordingly. The text segmentation processing apparatus of this application embodiment matches the corresponding target segmentation tool for texts with different type features, which facilitates adaptive processing and efficient segmentation of texts with different type features. After matching the corresponding target segmentation tool, the segmentation processing module 130 performs segmentation processing on the text to be processed using the target segmentation tool. While ensuring the efficiency of text segmentation processing, it can further optimize the effect of text segmentation.
[0081] In some embodiments, the text parsing module 110 is configured to perform category parsing, format parsing, and content parsing on the text to be processed, and determine the text category, format features, and content features of the text to be processed.
[0082] In some embodiments, the chunking tool matching module 120 is configured to match the target chunking tool according to the type characteristics of the text to be processed, in the order of priority: text category, format characteristics, and content characteristics.
[0083] In some embodiments, content features include at least one of the following: hierarchical features, paragraph length, special format, or keywords. The chunking tool matching module 120 is configured to: first determine whether the type feature includes hierarchical features; if the type feature includes hierarchical features, then the recursive chunking tool is identified as the target chunking tool; if the type feature does not contain hierarchical features and the paragraph length is higher than a first threshold, then the embedded model chunking tool is identified as the target chunking tool; if the type feature does not contain hierarchical features and the paragraph length is lower than a second threshold, then the fixed-size chunking tool is identified as the target chunking tool, wherein the second threshold is less than the first threshold.
[0084] In some embodiments, the text parsing module 110 is configured to: perform feature parsing on the text to be processed to determine that the text to be processed includes at least two sub-texts with different type features. The chunking tool matching module 120 is configured to: match the target chunking tool corresponding to each sub-text according to the type features of each sub-text and according to preset rules.
[0085] In some embodiments, the chunking tool matching module 120 is configured to: form a chunking strategy by matching the target chunking tool according to the type characteristics of the text to be processed, and cache and learn multiple chunking strategies.
[0086] Figure 4 This is a schematic diagram of the structure of a segmentation processing module in one embodiment of the text segmentation processing apparatus of this application. In some embodiments, the segmentation processing module 130 includes an initial segmentation submodule 131, an information extraction submodule 132, and a segmentation optimization submodule 133.
[0087] The initial segmentation submodule 131 is used to perform initial segmentation processing on the text to be processed using the target segmentation tool to obtain the initial segmentation result.
[0088] The information extraction submodule 132 is used to extract various specific information from the initial block results.
[0089] The block optimization submodule 133 is used to optimize the initial block result based on specific information to obtain the text block result.
[0090] In some embodiments, the information extraction submodule 132 is configured to: extract the block number and semantic level of each initial block; extract the opening sentence and closing sentence of each initial block; extract specific terms and their descriptive positions in each initial block; and extract logical connectors in each initial block.
[0091] In some embodiments, the block optimization submodule 133 is configured to perform a first block optimization on adjacent initial blocks based on specific information, wherein if there is a logical jump between the opening sentence of the later initial block and the opening sentence of the earlier initial block, the adjacent initial blocks with the logical jump are merged, or a transition sentence is added to the later initial block with the logical jump.
[0092] In some embodiments, the block optimization submodule 133 is configured to perform a second block optimization on each adjacent initial block, wherein in each pair of adjacent initial blocks, a summary text is generated based on the content of the previous initial block and added before the opening sentence of the next initial block.
[0093] In some embodiments, the block optimization submodule 133 is configured to perform third block optimization within each initial block based on specific information, wherein if it is determined that there are pronouns in the initial block according to specific terms, each pronoun is replaced with the actual term it represents.
[0094] In some embodiments, the block optimization submodule 133 is configured to: perform a fourth block optimization on large blocks and small blocks based on specific information, wherein the large block is an initial block with a word count greater than a third threshold, and the small block is an initial block with a word count less than a fourth threshold, wherein the fourth threshold is less than the third threshold, wherein if the initial block is determined to be a multi-level block based on semantic level, specific terms and logical connectors, the multi-level block is split into at least two text blocks according to the level, wherein if the small block is determined to be a supplementary description of the adjacent previous initial block based on semantic level, specific terms and logical connectors, the small block is merged into the adjacent previous initial block.
[0095] Figure 5 This is a flowchart illustrating the process of segmenting text into blocks according to an embodiment of the text segmentation processing apparatus of this application, as follows: Figure 5The text segmentation processing device includes a text parsing module 110, a segmentation tool matching module 120, and a segmentation processing module. The segmentation processing module includes an initial segmentation submodule 131, an information extraction submodule 132, and a segmentation optimization submodule 133. The text parsing module 110 performs feature analysis on the text to be processed to determine its type characteristics. The segmentation tool matching module 120 matches a target segmentation tool according to the type characteristics of the text to be processed and a preset rule. In one example, the selectable segmentation tools include a fixed-size segmentation tool 810, a recursive segmentation tool 820, an embedded model segmentation tool 830, etc. The segmentation tool matching module 120 can determine one of the multiple selectable segmentation tools as the target segmentation tool for matching, according to the type characteristics of the text to be processed and a preset rule.
[0096] After the target segmentation tool is matched, the initial segmentation submodule 131 performs initial segmentation processing on the text to be processed using the target segmentation tool to obtain the initial segmentation result. The information extraction submodule 132 extracts various specific information from the initial segmentation result. The segmentation optimization submodule 133 optimizes the initial segmentation result based on the specific information to obtain the text segmentation result.
[0097] In the above embodiments, the initial segmentation submodule 131 performs initial segmentation processing on the text to be processed using the target segmentation tool to obtain initial segmentation results. The information extraction submodule 132 extracts various specific information from the initial segmentation results. The segmentation optimization submodule 133 optimizes the initial segmentation results based on the specific information to obtain text segmentation results. This ensures that each text segment of the final text segmentation result can be understood independently, without severing key content in the context, and maintaining semantic integrity. In the above embodiments, after segmentation processing by the segmentation processing module 130, the connection between text segments is optimized, making the content natural and smooth, and ensuring a smooth logical transition. In the above embodiments, after segmentation processing by the segmentation processing module 130, unnecessary fragmented blocks (overly segmented small blocks) can be merged, and long and difficult-to-understand large blocks can be broken down, eliminating fragmentation and repetition.
[0098] In the above embodiment, after the segmentation processing module 130 performs segmentation processing, a text segmentation result is obtained. The text segmentation result can be output to a downstream system or stored in a server or database. The downstream system is, for example, a knowledge base or a retrieval-augmented generation (RAG) question answering system.
[0099] This application also provides an electronic device. Figure 6This is a schematic diagram of the hardware structure of an embodiment of the electronic device of this application. The electronic device includes a memory 910 and at least one processor 920. The memory 910 is electrically connected to the at least one processor 920. The memory 910 stores instructions, and the at least one processor 920 calls the instructions in the memory 910 to cause the electronic device to execute the text segmentation processing method according to any of the foregoing embodiments of this application.
[0100] The text segmentation processing method includes: performing feature analysis on the text to be processed to determine the type features of the text to be processed; matching the target segmentation tool according to the type features of the text to be processed and according to preset rules; and performing segmentation processing on the text to be processed using the target segmentation tool.
[0101] Specifically, the processor 920 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0102] Memory 910 may include a large-capacity memory 910 for data or instructions. For example, and not limitingly, memory 910 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 910 may include removable or non-removable (or fixed) media. Where appropriate, memory 910 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 910 is a non-volatile solid-state memory. In a particular embodiment, memory 910 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.
[0103] In one example, the control device may also include a communication interface 930 and a bus 940. The processor 920, memory 910, and communication interface 930 are connected via the bus 940 and communicate with each other.
[0104] The communication interface 930 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0105] Bus 940 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a Memory 910 bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 940 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0106] Furthermore, in conjunction with the text segmentation processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores instructions that, when executed by a processor, implement any of the text segmentation processing methods in the above embodiments.
[0107] The text segmentation processing method includes: performing feature analysis on the text to be processed to determine the type features of the text to be processed; matching the target segmentation tool according to the type features of the text to be processed and according to preset rules; and performing segmentation processing on the text to be processed using the target segmentation tool.
[0108] This application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0109] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0110] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0111] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A text segmentation processing method, characterized in that, include: The text to be processed is subjected to feature analysis to determine the type features of the text to be processed; Based on the type characteristics of the text to be processed, a target segmentation tool is matched according to preset rules; The target segmentation tool is used to segment the text to be processed into blocks.
2. The text segmentation processing method as described in claim 1, characterized in that, The step of performing feature parsing on the text to be processed to determine the type features of the text to be processed includes: The text to be processed is parsed by category, format, and content to determine its text category, format features, and content features.
3. The text segmentation processing method as described in claim 2, characterized in that, The tool for matching target segments according to preset rules based on the type characteristics of the text to be processed includes: Based on the type characteristics of the text to be processed, the target segmentation tool is matched in the order of priority: text category, format characteristics, and content characteristics.
4. The text segmentation processing method as described in claim 2, characterized in that, The content features include at least one of the following: hierarchical features, paragraph length, special format, or keywords; the tool for matching target segments according to preset rules based on the type features of the text to be processed includes: First, determine whether the type feature includes the hierarchical feature. If the type feature includes the hierarchical feature, then confirm the recursive segmentation tool as the target segmentation tool. If the hierarchical feature is not present in the type feature and the paragraph length is higher than the first threshold, then the embedding model segmentation tool is identified as the target segmentation tool. If the hierarchical feature is not present in the type features and the paragraph length is less than the second threshold, then the fixed-size segmentation tool is identified as the target segmentation tool, wherein the second threshold is less than the first threshold.
5. The text segmentation processing method as described in claim 1, characterized in that, The step of performing feature parsing on the text to be processed to determine the type features of the text to be processed includes: The text to be processed is subjected to feature parsing to determine that the text to be processed includes at least two sub-texts with different types of features; The tool for matching target segments according to preset rules based on the type characteristics of the text to be processed includes: Based on the type characteristics of each sub-text, the target segmentation tool corresponding to each sub-text is matched according to the preset rules.
6. The text segmentation processing method as described in claim 1, characterized in that, Also includes: The process of matching the target segmentation tool based on the type characteristics of the text to be processed forms a segmentation strategy, and multiple segmentation strategies are cached and learned.
7. The text segmentation processing method as described in claim 1, characterized in that, The step of segmenting the text to be processed using the target segmentation tool includes: The target segmentation tool is used to perform initial segmentation on the text to be processed to obtain initial segmentation results. Extract various specific information from the initial block division results; Based on the specific information, the initial segmentation result is optimized to obtain the text segmentation result.
8. The text segmentation processing method as described in claim 7, characterized in that, The initial block division result includes multiple initial blocks arranged in sequence, and the extraction of various specific information from the initial block division result includes: Extract the block number and semantic level of each initial block; Extract the first and last sentences of each initial block; Extract specific terms and their description locations from each initial block; Extract logical connectors from each of the initial blocks.
9. The text segmentation processing method as described in claim 8, characterized in that, The step of optimizing the initial segmentation result based on the specific information to obtain the text segmentation result includes: Based on the specific information, the adjacent initial blocks are optimized in the first block segmentation. If there is a logical jump between the opening sentence of the later initial block and the opening sentence of the earlier initial block, the adjacent initial blocks with the logical jump are merged, or a transition sentence is added to the later initial block with the logical jump.
10. The text segmentation processing method as described in claim 8, characterized in that, The step of optimizing the initial segmentation result based on the specific information to obtain the text segmentation result includes: A second block optimization is performed on each adjacent initial block, wherein in each pair of adjacent initial blocks, a summary text is generated based on the content of the previous initial block and added before the opening sentence of the subsequent initial block.
11. The text segmentation processing method as described in claim 8, characterized in that, The step of optimizing the initial segmentation result based on the specific information to obtain the text segmentation result includes: Based on the specific information, a third block optimization is performed within each of the initial blocks, wherein if a pronoun is determined to exist in the initial block according to the specific terminology, then each pronoun is replaced with the actual term it represents.
12. The text segmentation processing method as described in claim 8, characterized in that, The step of optimizing the initial segmentation result based on the specific information to obtain the text segmentation result includes: Based on the specific information, a fourth segmentation optimization is performed on the large and small segments. The large segment is the initial segment with a word count greater than the third threshold, and the small segment is the initial segment with a word count less than the fourth threshold, where the fourth threshold is less than the third threshold. If the initial segment is determined to be a multi-level segment based on the semantic level, the specific terms, and the logical connectors, then the multi-level segment is split into at least two text segments according to the levels. If the small segment is determined to be a supplementary explanation of the adjacent preceding initial segment based on the semantic level, the specific terms, and the logical connectors, then the small segment is merged into the adjacent preceding initial segment.
13. A text segmentation processing device, characterized in that, include: The text parsing module is used to perform feature parsing on the text to be processed and determine the type features of the text to be processed; The chunking tool matching module is used to match the target chunking tool according to the type characteristics of the text to be processed and according to preset rules. The block processing module is used to block the text to be processed using the target block processing tool.
14. An electronic device, characterized in that, The electronic device includes a memory and at least one processor, the memory being electrically connected to the at least one processor, and the memory storing instructions. The at least one processor invokes the instructions in the memory, causing the electronic device to perform the text segmentation processing method as described in any one of claims 1 to 12.
15. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the text segmentation processing method as described in any one of claims 1 to 12.