Multi-strategy markdown segmentation method and apparatus

Abstract

The application provides a multi-strategy Markdown segmentation method and device, and relates to the technical field of natural language processing. The method comprises the following steps: performing initial segmentation on a to-be-processed document by using the title of the to-be-processed document; for each chapter, performing segmentation on the chapter by using a custom rule to obtain at least one segment, determining a segment with a segment length greater than a preset length limit as a to-be-processed large segment, and determining a segment with a segment length less than or equal to the preset length limit as a to-be-processed small segment; in each chapter, performing first processing on the to-be-processed small segment to obtain a first type of segment, and performing second processing on the to-be-processed large segment to obtain a second type of segment; and merging the segments in each chapter to obtain a final result segment. In this way, by using the differentiated processing strategies for the to-be-processed small segment merging and the to-be-processed large segment segmentation, the semantic integrity and the length rationality of the document segmentation are realized, and the accuracy and efficiency of the text processing are improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a multi-strategy Markdown segmentation method and apparatus. Background Technology

[0002] Markdown is a lightweight markup language widely used for writing and storing text content such as technical documents and knowledge bases. In the field of natural language processing, segmenting Markdown documents is an important prerequisite for tasks such as text analysis and information extraction.

[0003] In related technologies, Markdown document segmentation methods often employ a single strategy, such as segmenting based solely on heading levels or simply by length. This makes it difficult to balance the complexity of the document structure with the semantic integrity of the content. Especially when processing documents containing multi-level headings and custom rule-matched content, existing methods cannot flexibly adjust processing strategies based on segmentation type (e.g., segments matching rules versus segments not matching rules) and segment size (small versus large segments). This results in semantic breaks, incorrect lengths, or missing metadata in the segmentation results, affecting the accuracy and efficiency of subsequent text processing tasks.

[0004] Therefore, there is an urgent need for a segmentation method that can perform differentiated processing based on the differences in segment size and retain the original segment semantic information for its sub-fractions when segmenting large segments. Summary of the Invention

[0005] The purpose of this application is to provide a multi-strategy Markdown segmentation method and apparatus. By adopting differentiated processing strategies for merging small segments and splitting large segments, the method achieves semantic integrity and reasonable length of Markdown document segments, thereby improving the accuracy and efficiency of text processing.

[0006] This application provides a multi-strategy Markdown segmentation method, including: The process involves: acquiring a document to be processed and initially segmenting it using its title to obtain at least one chapter, and generating a chapter summary for each chapter; the document to be processed is a Markdown document; for each chapter, segmenting it using custom rules to obtain at least one segment, and identifying segments with a length greater than a preset length limit as large segments to be processed, and segments with a length less than or equal to the preset length limit as small segments to be processed; within each chapter, performing a first process on the small segments to be processed to obtain a first type of fragment, and performing a second process on the large segments to be processed to obtain a second type of fragment; the second type of fragment includes supplementary metadata generated based on the content of the large segments to be processed; merging the first type of fragment and the second type of fragment in each chapter to obtain the final result fragment; wherein the final result fragment includes metadata, which includes the chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes supplementary metadata.

[0007] Optionally, the step of performing the first processing on the small segment to be processed within each chapter to obtain the first type of fragment includes: within each chapter, using a greedy algorithm to sequentially perform a preset judgment operation on each small segment to be processed; wherein, the preset judgment operation includes at least one of the following: a first sub-operation, a second sub-operation, and a third sub-operation; the first sub-operation includes: if the current small segment to be processed and the next small segment to be processed meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; the second sub-operation includes: if the current small segment to be processed and the next small segment to be processed meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; the second sub-operation includes: if the current small segment to be processed and the next small segment to be processed meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; If the next segment to be processed does not meet the merging conditions, the merging of the current segment ends; the third sub-operation includes: after all segments to be processed within the chapter have been judged, the merged segment and the unmerged segments to be processed are taken as the first type of fragment; the merging conditions include: the current segment to be processed and the next segment to be processed are both matching segments or both unmatched segments; the size of the current segment to be processed is less than the preset length limit; and the sum of the size of the current segment to be processed and the size of the next segment to be processed is less than a preset multiple of the preset length limit.

[0008] Optionally, the second processing of the large segment to be processed to obtain a second type of fragment includes: generating a segment summary of the large segment to be processed as supplementary metadata of the large segment to be processed, and using a semantic model to segment the large segment to be processed to obtain semantic fragments; if the length of the semantic fragment is greater than the preset length limit, then mechanical segmentation processing is performed on the semantic fragment to obtain mechanical fragments, and the mechanical fragments are used as the second type of fragments; otherwise, the semantic fragments are used as the second type of fragments.

[0009] Optionally, generating a segmented summary of the large segment to be processed and segmenting the large segment to be processed using a semantic model to obtain semantic fragments includes: generating a summary of the large segment to be processed using a first model to obtain a segmented summary of the large segment to be processed; and performing semantic segmentation of the large segment to be processed using a second model to obtain the semantic fragments; wherein the first model is a large language model; and the second model is a Sentence-BERT model, or a large language model.

[0010] Optionally, performing mechanical segmentation on the semantic segment to obtain mechanical segments includes: segmenting the semantic segment using preset characters to obtain multiple sub-segments; sequentially concatenating the multiple sub-segments according to the original text order until the length of the concatenated part is greater than the preset length limit, and using the concatenated part as a mechanical segment; wherein, the preset characters include at least one of the following: Chinese period, Chinese exclamation mark.

[0011] Optionally, within each chapter, performing a first process on the unprocessed sub-segments to obtain a first type of fragment includes: determining whether merging unprocessed sub-segments in matching segments is allowed, and whether merging unprocessed sub-segments in unmatched segments is allowed; if merging unprocessed sub-segments in matching segments is allowed, then merging the unprocessed sub-segments in matching segments within the chapter; if merging unprocessed sub-segments in unmatched segments is allowed, then merging the unprocessed sub-segments in unmatched segments within the chapter; and using the merged segment and the unmerged unprocessed sub-segments as the first type of fragment.

[0012] Optionally, before performing the second processing on the large segment to be processed to obtain the second type of fragment, the method further includes: determining whether it is allowed to split the large segment to be processed in the matching segment, and whether it is allowed to split the large segment to be processed in the unmatched segment; determining the large segment to be processed that is allowed to be split as the large segment to be split, and determining the large segment to be processed that is not allowed to be split as the retained large segment; performing the second processing on the large segment to be processed to obtain the second type of fragment includes: performing the second processing on the large segment to be split, and using the fragment obtained after processing, together with the retained large segment, as the second type of fragment.

[0013] This application also provides a multi-strategy Markdown segmentation device, including: The document is divided into three modules: an acquisition module for acquiring a document to be processed; a segmentation module for initially segmenting the document using its title to obtain at least one chapter and generating a chapter summary for each chapter; the document to be processed is a Markdown document; the segmentation module is further configured to segment each chapter using custom rules to obtain at least one segment, and to identify segments with a length greater than a preset length limit as large segments to be processed, and segments with a length less than or equal to the preset length limit as small segments to be processed; the segmentation module is further configured to perform a first processing on the small segments to be processed within each chapter to obtain a first type of fragment, and to perform a second processing on the large segments to be processed to obtain a second type of fragment; the second type of fragment includes supplementary metadata generated based on the content of the large segments to be processed; and a merging module for merging the first type of fragment and the second type of fragment in each chapter to obtain a final result fragment; wherein the final result fragment is accompanied by metadata, the metadata including the chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes supplementary metadata.

[0014] Optionally, the segmentation module is specifically used to perform a preset judgment operation on each segment to be processed sequentially using a greedy algorithm within each chapter; wherein the preset judgment operation includes at least one of the following: a first sub-operation, a second sub-operation, and a third sub-operation; the first sub-operation includes: if the current segment to be processed and the next segment to be processed meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; the second sub-operation includes: if the current segment to be processed and the next segment to be processed do not meet the merging condition... If the merging condition is met, the merging of the current segment ends; the third sub-operation includes: after all unprocessed sub-segments within the chapter have been judged, the merged segment and the unmerged unprocessed sub-segments are taken as the first type of fragment; the merging condition includes: the current unprocessed sub-segment and the next unprocessed sub-segment are both matching segments or both unmatched segments; the size of the current unprocessed sub-segment is less than the preset length limit; and the sum of the size of the current unprocessed sub-segment and the size of the next unprocessed sub-segment is less than a preset multiple of the preset length limit.

[0015] Optionally, the segmentation module is specifically used to generate a segment summary of the large segment to be processed as supplementary metadata of the large segment to be processed, and to segment the large segment to be processed using a semantic model to obtain semantic fragments; the segmentation module is further specifically used to perform mechanical segmentation on the semantic fragment if the length of the semantic fragment is greater than the preset length limit, to obtain mechanical fragments, and to use the mechanical fragments as the second type of fragments, otherwise, to use the semantic fragments as the second type of fragments.

[0016] Optionally, the segmentation module is specifically used to generate a summary of the large segment to be processed using a first model, thereby obtaining a segmented summary of the large segment to be processed; the segmentation module is also specifically used to perform semantic segmentation of the large segment to be processed using a second model, thereby obtaining the semantic fragments; wherein, the first model is a large language model; the second model is a Sentence-BERT model, or a large language model.

[0017] Optionally, the segmentation module is specifically used to segment the semantic segment using preset characters to obtain multiple sub-segments; the segmentation module is also specifically used to sequentially concatenate the multiple sub-segments according to the original text order until the length of the concatenated part is greater than the preset length limit, and the concatenated part is used as a mechanical segment; wherein, the preset characters include at least one of the following: Chinese period, Chinese exclamation mark.

[0018] Optionally, the segmentation module is specifically used to determine whether merging unprocessed sub-segments in matching segments is allowed, and whether merging unprocessed sub-segments in unmatched segments is allowed; the segmentation module is further used to merge unprocessed sub-segments in matching segments within a chapter if merging unprocessed sub-segments in matching segments is allowed, and to merge unprocessed sub-segments in unmatched segments within a chapter if merging unprocessed sub-segments in unmatched segments is allowed; the segmentation module is further used to treat the merged segments and the unmerged unprocessed sub-segments as the first type of fragments.

[0019] Optionally, the device further includes: a judgment module and a determination module; the judgment module is used to determine whether it is allowed to split the large segments to be processed in the matching segments, and whether it is allowed to split the large segments to be processed in the unmatched segments; the determination module is used to determine the large segments to be processed that are allowed to be split as large segments to be split, and to determine the large segments to be processed that are not allowed to be split as large segments to be retained; the splitting module is specifically used to perform a second processing on the large segments to be split, and to take the resulting fragments and the retained large segments as the second type of fragments.

[0020] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described multi-strategy Markdown segmentation methods.

[0021] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described multi-strategy Markdown segmentation methods.

[0022] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described multi-strategy Markdown segmentation methods.

[0023] The multi-strategy Markdown segmentation method and apparatus provided in this application obtain a document to be processed, and perform an initial segmentation of the document to be processed using the title of the document to be processed to obtain at least one chapter, and generate a chapter summary for each chapter; the document to be processed is a Markdown document; for each chapter, the chapter is segmented using a custom rule to obtain at least one segment, and segments with a length greater than a preset length limit are identified as large segments to be processed, and segments with a length less than or equal to the preset length limit are identified as small segments to be processed; within each chapter, a first processing is performed on the small segments to be processed to obtain a first type of segment, and a second processing is performed on the large segments to be processed to obtain a second type of segment; the second type of segment includes supplementary metadata generated based on the content of the large segments to be processed; the first type of segment and the second type of segment of each chapter are merged to obtain a final result segment; wherein, the final result segment is accompanied by metadata, and the metadata includes the chapter summary of the chapter; when the final result segment belongs to the second type of segment, the metadata also includes supplementary metadata. In this way, by adopting differentiated processing strategies for merging small segments and splitting large segments, the semantic integrity and length of Markdown document segments are achieved, thereby improving the accuracy and efficiency of text processing. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts of the multi-strategy Markdown segmentation method provided in this application; Figure 2 This is the second flowchart of the multi-strategy Markdown segmentation method provided in this application; Figure 3 This is a schematic diagram of the structure of the multi-strategy Markdown segmentation device provided in this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. All actions involving the acquisition of signal information or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located and with authorization from the owner of the relevant device.

[0028] Conventional document segmentation methods in related technologies mainly involve segmenting based on punctuation marks and fixed sizes, or using semantic models to perform semantic segmentation of text. However, these methods have the following shortcomings in practical applications: when the segmentation granularity is too large, the semantic information contained in a single segment is mixed, reducing the accuracy of retrieval; when the segmentation granularity is too small, the semantic integrity and contextual information of the segments are insufficient, which also affects the recall effect. Especially when dealing with segments of different lengths generated after rule matching, existing methods struggle to balance the requirements of moderate size and semantic integrity. Furthermore, when a large segment is segmented into multiple smaller segments due to exceeding the length limit, the segmented segments often lose the complete semantic information carried by the original large segment, leading to a decrease in semantic matching ability during the retrieval stage.

[0029] To address the aforementioned technical problems in related technologies, this application provides a multi-strategy Markdown segmentation method. This method segments the document by title, then divides the segments into large and small segments based on a preset length limit and performs differentiated processing. Simultaneously, it attaches supplementary metadata generated based on the original segment content to the sub-fragments generated from the large segments. This achieves the goal of retaining the original hierarchy and semantic information of each segment while effectively controlling the segment size, thus balancing moderate segment size with semantic cohesion and integrity, and improving the knowledge recall accuracy of the retrieval enhancement generation system.

[0030] The multi-strategy Markdown segmentation method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0031] like Figure 1 As shown in the embodiment of this application, a multi-strategy Markdown segmentation method is provided, which may include the following steps 101 to 104: Step 101: Obtain the document to be processed, and use the title of the document to perform an initial segmentation of the document to obtain at least one chapter, and generate a chapter summary for each chapter.

[0032] The document to be processed is a Markdown document.

[0033] For example, the document to be processed is a Markdown document. The system makes full use of the determinism of Markdown headings, parses the heading levels in the document to initially divide the document into independent chapters, and calculates a summary for each chapter using a model.

[0034] For example, a Markdown document might be divided into three sections. Section 1 would be titled "The world contains a wealth of interesting knowledge...", with metadata including "Heading 1: Knowledge" and "Section Summary: Introduction". Section 2 would be titled "1. How many steps does it take to put an elephant in a refrigerator...? 2. What are the important things to do in a day?...", with metadata including "Heading 1: Knowledge", "Heading 2: Fun Q&A", and "Section Summary: Several interesting Q&As". Section 3 would be titled "It is very important to follow the rules of social life. We must follow them. The first rule is to study hard and make progress every day. The second rule is to abide by laws and regulations. Work hard.", with metadata including "Heading 1: Rules" and "Section Summary: Introducing the principles that need to be followed in life and work".

[0035] Step 102: For each chapter, use custom rules to segment the chapter to obtain at least one segment, and determine the segment whose length is greater than the preset length limit as the large segment to be processed, and determine the segment whose length is less than or equal to the preset length limit as the small segment to be processed.

[0036] The at least one segment includes: a matching segment that matches the custom rule, and an unmatched segment that does not match the custom rule.

[0037] Specifically, the custom rule includes at least one regular expression; the regular expression is used to match text patterns in the chapter that represent content hierarchy or entry division.

[0038] Exemplarily, the preset custom rule allows the user to capture the part that hits the content requiring special attention. For the aforementioned Chapter 3, if the user configures to use a regular expression as: " \s 第[一二三四五六七八九十百1234567890]{1,4}(?:章|节|条|部分|篇)\s". The parameters in this regular expression are explained as follows: represents a line break character; \s represents matching any number of whitespace characters; "第" represents matching the character "第"; [一二三四五六七八九十百1234567890]{1,4} represents matching 1 to 4 Chinese numerals or Arabic numerals; (?:章|节|条|部分|篇) represents a non-capturing group, matching any one of "章", "节", "条", "部分" or "篇"; \s represents matching one whitespace character.

[0039] Through this regular expression, Chapter 3 will be split into four segments: the first segment is "It is very important to abide by the rules of social life.", marked as an unmatched segment; the second segment is "We must abide by them.", marked as an unmatched segment; the third segment is "Article 1 Study hard and make progress every day.", marked as a matched segment; the fourth segment is "Article 2 Abide by laws and regulations XXXX. Work hard XXXX.", marked as a matched segment. The system determines a segment with a length greater than the preset length limit as a large segment to be processed, and determines a segment with a length less than or equal to the preset length limit as a small segment to be processed, so as to filter out small segments that need further merging and large segments that need further splitting.

[0040] Step 103: Within each chapter, perform a first processing on the small segments to be processed to obtain first-class fragments, and perform a second processing on the large segments to be processed to obtain second-class fragments.

[0041] Wherein, the second-class fragments include supplementary metadata generated based on the content of the large segments to be processed. The aforementioned first processing is a processing performed on the small segments to be processed, which obtains the first-class fragments by sequentially performing a preset judgment operation on each small segment to be processed using a greedy algorithm, or judging whether to allow merging the small segments to be processed in the matched segments and whether to allow merging the small segments to be processed in the unmatched segments; the aforementioned second processing is a processing performed on the large segments to be processed, which takes the segment summary of the large segments to be processed as supplementary metadata, splits the large segments to be processed by using a semantic model to obtain semantic fragments, performs mechanical splitting processing to obtain mechanical fragments if the length of the semantic fragment is greater than the preset length limit, or judges whether to allow splitting the large segments to be processed in the matched segments and whether to allow splitting the large segments to be processed in the unmatched segments, so as to obtain the second-class fragments containing the supplementary metadata.

[0042] It should be noted that in this embodiment, each chapter does not necessarily contain both type I and type II fragments simultaneously. A chapter may contain only type I fragments, only type II fragments, or both. This depends entirely on the actual situation of the segments obtained after the chapter is segmented and its size is determined by the custom rules. When the length of all segments within a chapter is less than or equal to the preset length limit, the chapter only has small segments to be processed and no large segments to be processed; when the length of all segments within a chapter is greater than the preset length limit, the chapter only has large segments to be processed and no small segments to be processed.

[0043] Specifically, in each of the above chapters, the step of performing the first processing on the small segments to be processed to obtain the first type of fragments may further include the following steps 103a1 to 103a3 regarding how to perform the merging: Step 103a1: Determine whether merging unprocessed sub-segments in matching segments is allowed, and whether merging unprocessed sub-segments in unmatched segments is allowed.

[0044] Step 103a2: If merging unprocessed sub-segments in a matching segment is allowed, then merge the unprocessed sub-segments in the matching segment within the chapter. If merging unprocessed sub-segments in an unmatched segment is allowed, then merge the unprocessed sub-segments in the unmatched segment within the chapter.

[0045] Step 103a3: The merged segments and the unmerged small segments to be processed are taken as the first type of fragments.

[0046] For example, the system determines whether to merge matching or unmatched segments by setting a switch, thereby flexibly adapting to the needs of entry integrity and length control in different scenarios, and directly using the segments that cannot be merged as the output results.

[0047] Furthermore, in the aforementioned section, the step of performing the first processing on the small segments to be processed to obtain the first type of fragments, the step of selecting the execution object (i.e., selecting which small segments to merge) may further include the following step 103b: Step 103b: Within each chapter, a greedy algorithm is used to perform preset judgment operations on each sub-segment to be processed in sequence.

[0048] The preset judgment operation includes at least one of the following: a first sub-operation, a second sub-operation, and a third sub-operation; the first sub-operation includes: if the current segment to be processed and the next segment to be processed meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; the second sub-operation includes: if the current segment to be processed and the next segment to be processed do not meet the merging condition, then end the merging of the current segment; the third sub-operation includes: after all segments to be processed in the chapter have been judged, use the merged segment and the unmerged segments to be processed as the first type of fragment; the merging condition includes: the current segment to be processed and the next segment to be processed are both matching segments or both unmatched segments; the size of the current segment to be processed is less than the preset length limit; and the sum of the size of the current segment to be processed and the size of the next segment to be processed is less than a preset multiple of the preset length limit.

[0049] For example, the merging logic maintains the continuity of the original text and uses a greedy algorithm to merge sequentially. For instance, to determine if segment one can be merged with segment two, since both are unmatched segments, the current segment size is less than the limit size, and the sum of the current segment size and the next segment size is still less than 1.2 times the limit size (i.e., the default value of the preset multiple), segment one can be merged with segment two. The merged segment is "It is very important to abide by the rules of social life. We must abide by them.", and is output as a first-type segment. If the current segment and the next segment cannot meet the merging conditions, such as being of different types or exceeding the limit length, then the merging of the current segment ends.

[0050] It should be noted that in this embodiment of the application, the switching mechanism of steps 103a1 to 103a3 can be used to determine which types of small segments to merge, and then the types that are allowed to be merged can be merged in step 103b.

[0051] In another embodiment, before performing the second processing on the large segment to be processed in step 103 to obtain the second type of fragment, the method further includes the following steps 103c1 and 103c2: Step 103c1: Determine whether it is allowed to split the large unprocessed segments in the matching segments, and whether it is allowed to split the large unprocessed segments in the unmatched segments.

[0052] Step 103c2: Determine the large segments to be processed that are allowed to be split as the large segments to be split, and determine the large segments to be processed that are not allowed to be split as the large segments to be retained.

[0053] Specifically, based on steps 103c1 and 103c2 above, the second processing of the large segment to be processed in step 103 above to obtain the second type of fragment may further include the following step 103d: Step 103d: Perform a second processing on the large segment to be segmented, and use the processed fragments and the retained large segment as the second type of fragments.

[0054] For example, the purpose of distinguishing between matched and unmatched entries is that users may request certain fixed-format entries without length restrictions. If content like "First: ... Second: ..." is encountered, no matter how long each entry is, it should not be split. In this case, the system will directly output it as a large segment to ensure the integrity of key rule entries.

[0055] Specifically, the second processing of the large segment to be processed in step 103 above to obtain the second type of fragment may also include the following steps 103e and 103f: Step 103e: Generate a segment summary of the large segment to be processed as supplementary metadata of the large segment to be processed, and use a semantic model to segment the large segment to be processed to obtain semantic fragments.

[0056] Step 103f: If the length of the semantic segment is greater than the preset length limit, then perform mechanical segmentation on the semantic segment to obtain a mechanical segment, and use the mechanical segment as the second type of segment; otherwise, use the semantic segment as the second type of segment.

[0057] For example, if the above segment four, "Article Two: Abide by Laws and Regulations XXXX. Work Hard XXXX.", is identified as a large segment to be processed due to its excessive length, the system will first calculate a summary of this large segment to strengthen the metadata of the smaller segments after segmentation. Subsequently, semantic model is used for segmentation. If the semantic fragments are still too large, mechanical segmentation is triggered as a fallback strategy to ensure that the final output fragments all meet the length limit.

[0058] Furthermore, in step 103e above, the steps of generating segmented summaries of the large segment to be processed and using a semantic model to segment the large segment to be processed to obtain semantic fragments may also include the following steps 103e1 and 103e2: Step 103e1: Use the first model to generate a summary of the large segment to be processed, and obtain the segment summary of the large segment to be processed.

[0059] Step 103e2: Use the second model to perform semantic segmentation on the large segment to be processed, and obtain the semantic fragment.

[0060] The first model is a large language model; the second model is a Sentence-BERT model, or a large language model.

[0061] For example, at this stage, the amount of content sent to the model for processing is relatively small, making it suitable in terms of processing speed, resource consumption, and accuracy. Different types of models can be used for summarization and semantic segmentation; either a lightweight Sentence-BERT model can be used to improve processing efficiency, or a large language model can be used to ensure segmentation quality.

[0062] Furthermore, in step 103f above, the step of performing mechanical segmentation on the semantic fragment to obtain mechanical fragments may also include the following steps 103f1 and 103f2: Step 103f1: Use preset characters to segment the semantic segments to obtain multiple sub-segments.

[0063] Step 103f2: Sequentially splice the multiple sub-segments according to the original text order until the length of the spliced ​​part is greater than the preset length limit, and use the spliced ​​part as a mechanical segment.

[0064] The preset characters include at least one of the following: Chinese period, Chinese exclamation mark.

[0065] For example, punctuation marks are used as cutting points to divide the excessively long semantic segments into multiple sub-segments, which are then spliced ​​together in sequence until they approach the length limit, thereby maintaining the integrity of the sentence as much as possible while adhering to the size limit.

[0066] It should be noted that if after using the preset character segmentation (to obtain the above mechanical segmentation), there are still segments that exceed the preset segmentation length limit, then no longer length processing will be performed, that is, no further segmentation will be possible, the excessive length will be retained, and the length limit will be invalid.

[0067] Step 104: Merge the first type of fragments and the second type of fragments of each chapter to obtain the final result fragments.

[0068] The final result fragment is accompanied by metadata, which includes a chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes the supplementary metadata.

[0069] It should be noted that if a chapter contains only type 1 fragments, then only type 1 fragments within that chapter will be merged. Similarly, if a chapter contains only type 2 fragments, then only type 2 fragments within that chapter will be merged.

[0070] Furthermore, the metadata also includes: the title corresponding to the chapter; the title includes at least one of first-level headings, second-level headings, and multi-level headings.

[0071] For example, the system integrates the first type of fragments obtained through the first processing with the second type of fragments obtained through the second processing to form all the final result fragments corresponding to the chapter. For the final result fragments generated from the aforementioned segmented large segments to be processed, their metadata includes a segment summary. For example, fragment one, generated after segmenting the above four segments, contains the content "Article Two: Comply with Laws and Regulations XXXX.", and its metadata includes "First-level heading: Rules," "Chapter summary: Introduces principles to be followed in life and work," and "Paragraph summary: Work-related principles." Fragment two contains the content "Work Hard XXXX.", and its metadata also includes the aforementioned first-level heading, chapter summary, and paragraph summary. Due to the presence of paragraph summaries, although the completeness of the fragment decreases, it still contains general paragraph information, increasing the information content of the fragment.

[0072] For example, such as Figure 2 The flowchart shown is a multi-strategy Markdown segmentation method provided in an embodiment of this application. The main steps of the flowchart are as follows: S1, Title Segmentation, takes a Markdown document as input and outputs sections and summary snippets. The summary snippets are generated by a Large Language Model (LLM).

[0073] S2. Section Content Rule Matching: For each section, custom rules are used for matching, dividing the content into matched segments (Segmenty) and unmatched segments (Segmentn). Segment Size Judgment: The size of the matched segments is judged, classifying them as small segments, appropriate segments, and large segments.

[0074] S3, Small Segment Merging, merges small segments to generate S3 result chunks. Metadata includes headers and SummarySn.

[0075] S4. Large Segment Processing: This involves processing large segments, including: Segment Summary; Large Segment Model Segmentation, using a Large Language Model (LLM) to segment data into chunks; Length Judgment, if a chunk is still too long, proceed to the next step; Predefined Symbol Segmentation & Splicing, using predefined characters (such as period, question mark, exclamation mark) for segmentation and splicing to obtain S4 result chunks. Metadata includes headers, Summarysn, and Summaryst.

[0076] S5. Merge Results: Merge the fragments from S3 and S4 to obtain the final result. The processing flow of steps S3 and S4 is independently controlled by a switch parameter that allows merging / splitting, targeting smaller and larger segments respectively, to ensure the semantic integrity and reasonable length of the segmented results.

[0077] The multi-strategy Markdown segmentation method provided in this application embodiment obtains a document to be processed and performs an initial segmentation of the document using its title to obtain at least one chapter, and generates a chapter summary for each chapter; the document to be processed is a Markdown document; for each chapter, a custom rule is used to segment the chapter to obtain at least one segment, and segments with a length greater than a preset length limit are identified as large segments to be processed, and segments with a length less than or equal to the preset length limit are identified as small segments to be processed; within each chapter, a first processing is performed on the small segments to be processed to obtain a first type of segment, and a second processing is performed on the large segments to be processed to obtain a second type of segment; the second type of segment includes supplementary metadata generated based on the content of the large segments to be processed; the first type of segment and the second type of segment of each chapter are merged to obtain a final result segment; wherein, the final result segment is accompanied by metadata, and the metadata includes the chapter summary of the chapter; when the final result segment belongs to the second type of segment, the metadata also includes supplementary metadata. In this way, by adopting differentiated processing strategies for merging small segments and splitting large segments, the semantic integrity and length of Markdown document segments are achieved, thereby improving the accuracy and efficiency of text processing.

[0078] It should be noted that the multi-strategy Markdown segmentation method provided in this application embodiment can be executed by a multi-strategy Markdown segmentation device or a control module within that device for executing the multi-strategy Markdown segmentation method. This application embodiment uses the execution of the multi-strategy Markdown segmentation method by a multi-strategy Markdown segmentation device as an example to illustrate the multi-strategy Markdown segmentation device provided in this application embodiment.

[0079] It should be noted that, in the embodiments of this application, the multi-strategy Markdown segmentation methods shown in the accompanying drawings are all illustrated by way of example with reference to one of the accompanying drawings in the embodiments of this application. In specific implementation, the multi-strategy Markdown segmentation methods shown in the accompanying drawings can also be implemented in conjunction with any other accompanying drawings shown in the above embodiments, which will not be elaborated here.

[0080] The multi-strategy Markdown segmentation apparatus provided in this application is described below. The multi-strategy Markdown segmentation apparatus described below can be referred to in correspondence with the multi-strategy Markdown segmentation method described above.

[0081] Figure 3 A schematic diagram of the structure of the multi-strategy Markdown segmentation device provided in the embodiments of this application is shown below. Figure 3 As shown, it specifically includes: The document acquisition module 301 is used to acquire the document to be processed; the segmentation module 302 is used to perform an initial segmentation of the document to be processed using the title of the document to be processed, to obtain at least one chapter, and to generate a chapter summary for each chapter; the document to be processed is a Markdown document; the segmentation module 302 is also used to segment each chapter using a custom rule, to obtain at least one segment, and to determine segments with a length greater than a preset length limit as large segments to be processed, and segments with a length less than or equal to the preset length limit as small segments to be processed. The segmentation module 302 is further configured to perform a first processing on the small segments to be processed within each chapter to obtain a first type of fragment, and to perform a second processing on the large segments to be processed to obtain a second type of fragment; the second type of fragment includes supplementary metadata generated based on the content of the large segments to be processed; the merging module 303 is configured to merge the first type of fragment and the second type of fragment in each chapter to obtain a final result fragment; wherein, the final result fragment is accompanied by metadata, the metadata including a chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes supplementary metadata.

[0082] Optionally, the segmentation module 302 is specifically used to perform a preset judgment operation on each unprocessed sub-segment sequentially using a greedy algorithm within each chapter; wherein the preset judgment operation includes at least one of the following: a first sub-operation, a second sub-operation, and a third sub-operation; the first sub-operation includes: if the current unprocessed sub-segment and the next unprocessed sub-segment meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; the second sub-operation includes: if the current unprocessed sub-segment and the next unprocessed sub-segment do not ... If the merging conditions are met, the merging of the current segment ends; the third sub-operation includes: after all unprocessed sub-segments within the chapter have been judged, the merged segment and the unmerged unprocessed sub-segments are taken as the first type of fragment; the merging conditions include: the current unprocessed sub-segment and the next unprocessed sub-segment are both matching segments or both unmatched segments; the size of the current unprocessed sub-segment is less than the preset length limit; and the sum of the size of the current unprocessed sub-segment and the size of the next unprocessed sub-segment is less than a preset multiple of the preset length limit.

[0083] Optionally, the segmentation module 302 is specifically used to generate a segment summary of the large segment to be processed as supplementary metadata of the large segment to be processed, and to segment the large segment to be processed using a semantic model to obtain semantic fragments; the segmentation module 302 is also specifically used to perform mechanical segmentation processing on the semantic fragment if the length of the semantic fragment is greater than the preset length limit, to obtain mechanical fragments, and to use the mechanical fragments as the second type of fragments, otherwise, to use the semantic fragments as the second type of fragments.

[0084] Optionally, the segmentation module 302 is specifically used to generate a summary of the large segment to be processed using a first model to obtain a segmented summary of the large segment to be processed; the segmentation module 302 is also specifically used to perform semantic segmentation of the large segment to be processed using a second model to obtain the semantic fragments; wherein, the first model is a large language model; the second model is a Sentence-BERT model, or a large language model.

[0085] Optionally, the segmentation module 302 is specifically used to segment the semantic segment using preset characters to obtain multiple sub-segments; the segmentation module 302 is also specifically used to sequentially splice the multiple sub-segments according to the original text order until the length of the spliced ​​part is greater than the preset length limit, and use the spliced ​​part as a mechanical segment; wherein, the preset characters include at least one of the following: Chinese period, Chinese exclamation mark.

[0086] Optionally, the segmentation module 302 is specifically used to determine whether merging unprocessed sub-segments in a matching segment is allowed, and whether merging unprocessed sub-segments in an unmatched segment is allowed; the segmentation module 302 is further used to merge unprocessed sub-segments in a matching segment within a chapter if merging unprocessed sub-segments in a matching segment is allowed, and to merge unprocessed sub-segments in an unmatched segment within a chapter if merging unprocessed sub-segments in an unmatched segment is allowed; the segmentation module 302 is further used to treat the merged segment and the unmerged unprocessed sub-segments as the first type of fragment.

[0087] Optionally, the device further includes: a judgment module and a determination module; the judgment module is used to determine whether it is allowed to split the large segments to be processed in the matching segments, and whether it is allowed to split the large segments to be processed in the unmatched segments; the determination module is used to determine the large segments to be processed that are allowed to be split as large segments to be split, and to determine the large segments to be processed that are not allowed to be split as large segments to be retained; the splitting module 302 is specifically used to perform a second processing on the large segments to be split, and to take the processed fragments and the retained large segments as the second type of fragments.

[0088] The multi-strategy Markdown segmentation device provided in this application acquires a document to be processed and performs an initial segmentation of the document using its title to obtain at least one chapter, and generates a chapter summary for each chapter; the document to be processed is a Markdown document; for each chapter, a custom rule is used to segment the chapter to obtain at least one segment, and segments with a length greater than a preset length limit are identified as large segments to be processed, and segments with a length less than or equal to the preset length limit are identified as small segments to be processed; within each chapter, a first processing is performed on the small segments to be processed to obtain a first type of fragment, and a second processing is performed on the large segments to be processed to obtain a second type of fragment; the second type of fragment includes supplementary metadata generated based on the content of the large segments to be processed; the first type of fragment and the second type of fragment in each chapter are merged to obtain a final result fragment; wherein, the final result fragment is accompanied by metadata, and the metadata includes the chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes supplementary metadata. In this way, by adopting differentiated processing strategies for merging small segments and splitting large segments, the semantic integrity and length of Markdown document segments are achieved, thereby improving the accuracy and efficiency of text processing.

[0089] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logical instructions in the memory 430 to execute a multi-strategy Markdown segmentation method. This method includes: acquiring a document to be processed, and performing an initial segmentation of the document using its title to obtain at least one chapter, and generating a chapter summary for each chapter; the document to be processed is a Markdown document; for each chapter, segmenting it using custom rules to obtain at least one segment, and identifying segments with a length greater than a preset length limit as large segments to be processed, and segments with a length less than or equal to the preset length limit as small segments to be processed; within each chapter, performing a first processing on the small segments to be processed to obtain a first type of fragment, and performing a second processing on the large segments to be processed to obtain a second type of fragment; the second type of fragment includes supplementary metadata generated based on the content of the large segments to be processed; merging the first type of fragment and the second type of fragment in each chapter to obtain a final result fragment; wherein the final result fragment includes metadata, which includes the chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes supplementary metadata. In this way, by adopting differentiated processing strategies for merging small segments and splitting large segments, the semantic integrity and length of Markdown document segments are achieved, thereby improving the accuracy and efficiency of text processing.

[0090] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the multi-strategy Markdown segmentation method provided by the above methods. The method includes: obtaining a document to be processed, and performing an initial segmentation of the document to be processed using the title of the document to be processed to obtain at least one chapter, and generating a chapter summary for each chapter; the document to be processed is a Markdown document; for each chapter, segmenting the chapter using custom rules to obtain at least one segment, and dividing the segment... Segments with a length greater than a preset length limit are identified as large segments to be processed, while segments with a length less than or equal to the preset length limit are identified as small segments to be processed. Within each chapter, a first processing step is performed on the small segments to be processed to obtain first-type fragments, and a second processing step is performed on the large segments to be processed to obtain second-type fragments. The second-type fragments contain supplementary metadata generated based on the content of the large segments to be processed. The first-type fragments and second-type fragments of each chapter are merged to obtain the final result fragments. The final result fragments are accompanied by metadata, which includes a chapter summary of the chapter. When the final result fragment belongs to the second-type fragments, the metadata also includes supplementary metadata. Thus, by adopting differentiated processing strategies for merging small segments to be processed and splitting large segments to be processed, the semantic integrity and length rationality of Markdown document segments are achieved, improving the accuracy and efficiency of text processing.

[0092] Furthermore, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned multi-strategy Markdown segmentation methods. The method includes: acquiring a document to be processed; performing an initial segmentation of the document to be processed using its title to obtain at least one chapter; and generating a chapter summary for each chapter; wherein the document to be processed is a Markdown document; and for each chapter, segmenting the chapter using custom rules to obtain at least one segment, and identifying segments with a length greater than a preset length limit as segments to be processed. The system employs a segmentation method, identifying segments with a length less than or equal to a preset length limit as small segments to be processed. Within each chapter, a first processing step is performed on the small segments to be processed, resulting in first-type fragments, and a second processing step is performed on the large segments to be processed, resulting in second-type fragments. These second-type fragments include supplementary metadata generated based on the content of the large segments to be processed. The first-type and second-type fragments of each chapter are merged to obtain the final result fragments. These final result fragments include metadata, including a chapter summary. When the final result fragment belongs to the second-type fragment, the metadata also includes supplementary metadata. Thus, by employing differentiated processing strategies for merging small segments and splitting large segments to be processed, the semantic integrity and length rationality of Markdown document segmentation are achieved, improving the accuracy and efficiency of text processing.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-strategy Markdown segmentation method, characterized in that, include: Obtain the document to be processed, and use the title of the document to perform an initial segmentation to obtain at least one chapter, and generate a chapter summary for each chapter; the document to be processed is a Markdown document; For each chapter, the chapter is divided using custom rules to obtain at least one segment. Segments with a length greater than a preset length limit are identified as large segments to be processed, and segments with a length less than or equal to the preset length limit are identified as small segments to be processed. Within each chapter, the first processing is performed on the small segments to be processed to obtain the first type of fragments, and the second processing is performed on the large segments to be processed to obtain the second type of fragments; The second type of fragmentation includes a segmented summary generated based on the content of the large segment to be processed; The segment summary serves as supplementary metadata for the large segment to be processed. The segment summary is used to ensure that the sub-segments obtained after the large segment to be processed is segmented retain the semantic information of the large segment to be processed. Merge the first-type and second-type fragments of each chapter to obtain the final result fragments; The final result fragment is accompanied by metadata, which includes a chapter summary of the chapter. When the final result shard belongs to the second type of shard, the metadata also includes supplementary metadata.

2. The method according to claim 1, characterized in that, The at least one segment includes: a matching segment that matches the custom rule, and an unmatched segment that does not match the custom rule; the custom rule includes at least one regular expression; the regular expression is used to match text patterns in the chapter that represent content hierarchy or item division.

3. The method according to claim 1, characterized in that, Within each chapter, the first processing step is performed on the sub-segment to be processed, resulting in a first type of fragment, including: Within each chapter, a greedy algorithm is used to perform preset judgment operations on each sub-segment to be processed in sequence; The preset judgment operation includes at least one of the following: a first sub-operation, a second sub-operation, and a third sub-operation; the first sub-operation includes: if the current segment to be processed and the next segment to be processed meet the merging condition, then merge the two, and use the merged segment as the new current segment, and repeat the first sub-operation; the second sub-operation includes: if the current segment to be processed and the next segment to be processed do not meet the merging condition, then end the merging of the current segment; the third sub-operation includes: after all segments to be processed in the chapter have been judged, use the merged segment and the unmerged segments to be processed as the first type of fragment; the merging condition includes: the current segment to be processed and the next segment to be processed are both matching segments or both unmatched segments; the size of the current segment to be processed is less than the preset length limit; and the sum of the size of the current segment to be processed and the size of the next segment to be processed is less than a preset multiple of the preset length limit.

4. The method according to claim 1, characterized in that, The second processing is performed on the large segment to be processed to obtain the second type of fragment, including: Generate segment summaries of the large segments to be processed as supplementary metadata for the large segments to be processed, and use a semantic model to segment the large segments to be processed to obtain semantic fragments. If the length of the semantic segment is greater than the preset length limit, then the semantic segment is mechanically segmented to obtain a mechanical segment, and the mechanical segment is used as the second type of segment; otherwise, the semantic segment is used as the second type of segment.

5. The method according to claim 4, characterized in that, The process of generating segment summaries of the large segments to be processed and segmenting the large segments to be processed using a semantic model to obtain semantic fragments includes: The first model is used to generate a summary of the large segment to be processed, resulting in a segmented summary of the large segment to be processed. The second model is used to perform semantic segmentation on the large segment to be processed, resulting in the semantic fragments; The first model is a large language model; the second model is a Sentence-BERT model, or a large language model.

6. The method according to claim 4, characterized in that, The step of performing mechanical segmentation on the semantic fragment to obtain mechanical fragments includes: The semantic segments are divided using preset characters to obtain multiple sub-segments; The multiple sub-segments are sequentially spliced ​​together according to the original text order until the length of the spliced ​​part is greater than the preset length limit. The spliced ​​part is then used as a mechanical segment. The preset characters include at least one of the following: Chinese period, Chinese exclamation mark.

7. The method according to claim 1, characterized in that, The metadata also includes: the title corresponding to the chapter; the title includes at least one of first-level headings, second-level headings, and multi-level headings.

8. The method according to claim 1, characterized in that, Within each chapter, the first processing step is performed on the sub-segment to be processed, resulting in a first type of fragment, including: Determine whether merging unprocessed sub-segments within a matching segment is allowed, and whether merging unprocessed sub-segments within an unmatched segment is allowed; If merging unprocessed sub-segments within a matching segment is allowed, then the unprocessed sub-segments within the matching segment are merged within the chapter. If merging unprocessed sub-segments within an unmatched segment is allowed, then the unprocessed sub-segments within the unmatched segment are merged within the chapter. The merged segments, along with the unmerged smaller segments to be processed, are designated as the first type of fragments.

9. The method according to claim 1, characterized in that, Before performing the second processing on the large segment to be processed to obtain the second type of fragment, the method further includes: Determine whether it is allowed to split large unprocessed segments within matched segments, and whether it is allowed to split large unprocessed segments within unmatched segments; Large segments that are allowed to be split are identified as segments to be split, and large segments that are not allowed to be split are identified as segments to be retained. The second processing is performed on the large segment to be processed to obtain the second type of fragment, including: The large segment to be segmented is subjected to a second processing, and the resulting fragments, along with the retained large segment, are used together as the second type of fragments.

10. A multi-strategy Markdown segmentation device, characterized in that, The device includes: The acquisition module is used to acquire documents to be processed. The segmentation module is used to perform an initial segmentation of the document to be processed using the title of the document to be processed, to obtain at least one chapter, and to generate a chapter summary for each chapter; the document to be processed is a Markdown document; The segmentation module is also used to segment each chapter using custom rules to obtain at least one segment, and to determine the segments whose length is greater than a preset length limit as large segments to be processed, and the segments whose length is less than or equal to the preset length limit as small segments to be processed. The segmentation module is further configured to perform a first process on the small segments to be processed within each chapter to obtain a first type of fragment, and to perform a second process on the large segments to be processed to obtain a second type of fragment; the second type of fragment contains a segment summary generated based on the content of the large segments to be processed; the segment summary serves as supplementary metadata for the large segments to be processed, and the segment summary is used to ensure that the sub-fragments obtained after the large segments to be processed are segmented retain the semantic information of the large segments to be processed; The merge module is used to merge the first-type and second-type fragments of each chapter to obtain the final result fragment; The final result fragment is accompanied by metadata, which includes a chapter summary of the chapter; when the final result fragment belongs to the second type of fragment, the metadata also includes supplementary metadata.

Images