Document segmentation method and related method and device, electronic equipment and storage medium

Abstract

The invention discloses a document segmentation method, a related method, a related device, electronic equipment and a storage medium, and the document segmentation method comprises the following steps: segmenting a to-be-segmented document marked with a paragraph identifier based on each document paragraph to obtain a slice set containing a plurality of document slices; determining the slice type of the document slice based on the paragraph identifier in the document slice; performing semantic similarity measurement on the document slice based on a semantic measurement range matched with the slice type of the document slice to obtain a first reward score of the document slice; based on the first reward score of each document slice in the slice set, obtaining a total reward score of the slice set; and based on whether the total reward score of the slice set meets a screening condition related to a total reward threshold, determining whether to reserve the slice set as a target set of the to-be-segmented document. According to the scheme, the semantic integrity of the document slices can be ensured as much as possible, and the processing speed of document segmentation is increased.

Description

Document segmentation methods, related methods, apparatus, electronic devices and storage media Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a document segmentation method and related methods, apparatus, electronic devices and storage media. Background Technology

[0002] With the rapid development of artificial intelligence technology, document segmentation technology has gradually attracted attention. Efficient and intelligent document segmentation technology is also of certain significance for artificial intelligence technologies such as Retrieval-Augmented Generation (RAG).

[0003] Currently, some document segmentation technologies are based on fixed-size segments, but because they do not consider semantic boundaries, semantic information is fragmented. Others identify semantic breakpoints by calculating the semantic similarity between adjacent sentences, but this consumes significant computational resources, resulting in slow processing speeds. Therefore, ensuring the semantic integrity of document segments as much as possible while improving the processing speed of document segmentation has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a document segmentation method and related methods, apparatus, electronic devices, and storage media that can ensure the semantic integrity of document slices as much as possible and improve the processing speed of document segmentation.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a document segmentation method, comprising: segmenting a document to be segmented based on each document segment being marked with a paragraph identifier, to obtain a slice set containing several document slices; determining the slice type of a document slice based on the paragraph identifiers in the document slices; wherein the slice type of a document slice belongs to either a merged block or a split block; performing a semantic similarity measurement on the document slices based on a semantic measurement range matching the slice type of the document slices, to obtain a first reward score for the document slices; obtaining a total reward score for the slice set based on the first reward scores of each document slice in the slice set; and determining whether to retain the slice set as the target set of documents to be segmented based on whether the total reward score of the slice set meets a filtering condition related to a total reward threshold.

[0006] To address the aforementioned technical problems, the first aspect of this application provides a knowledge base construction method, comprising: selecting each document in a document library to be segmented as a document to be segmented; segmenting the document to be segmented to obtain a target set of the document to be segmented; wherein the target set includes several document slices of the document to be segmented, and the target set is obtained by segmenting the document using the document segmentation method in the first aspect; and constructing a knowledge base based on the respective target sets of each document in the document library to be segmented.

[0007] To address the aforementioned technical problems, a third aspect of this application provides a document segmentation apparatus, comprising: an initial segmentation module, a type determination module, a similarity measurement module, a reward determination module, and a set acquisition module. The initial segmentation module is used to segment documents to be segmented based on paragraph identifiers, resulting in a slice set containing several document slices. The type determination module is used to determine the slice type of a document slice based on the paragraph identifiers within the document slices; wherein the slice type of a document slice belongs to either a merge block or a split block. The similarity measurement module is used to perform semantic similarity measurement on the document slices based on a semantic measurement range matching the slice type of the document slices, obtaining a first reward score for each document slice. The reward determination module is used to obtain a total reward score for the slice set based on the first reward scores of each document slice in the slice set. The set acquisition module is used to determine whether to retain the slice set as the target set of documents to be segmented based on whether the total reward score of the slice set meets a filtering condition related to a total reward threshold.

[0008] To address the aforementioned technical problems, a third aspect of this application provides a knowledge base construction apparatus, comprising: a document selection module, a document segmentation module, and a knowledge base construction module. The document selection module is used to select each document in a document library to be segmented as a document to be segmented. The document segmentation module is used to segment the document to be segmented to obtain a target set of the document to be segmented. The target set includes several document slices of the document to be segmented, and the target set is obtained by the document segmentation apparatus described in the third aspect. The knowledge base construction module is used to construct a knowledge base based on the target sets of each document in the document library to be segmented.

[0009] To address the aforementioned technical problems, the fifth aspect of this application provides an electronic device, comprising at least a memory and a processor coupled to each other. The memory stores at least program instructions, and the processor executes the program instructions to implement the document segmentation method in the first aspect or the knowledge base construction method in the second aspect.

[0010] To address the aforementioned technical problems, the sixth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor. These program instructions are used to implement the document segmentation method of the first aspect or the knowledge base construction method of the second aspect.

[0011] The above scheme segments the document to be segmented based on paragraphs marked with paragraph identifiers, resulting in a slice set containing several document slices. Based on the paragraph identifiers within each document slice, the slice type is determined, and the slice type is either a merged block or a split block. Then, a semantic similarity measure is performed on the document slices based on a semantic metric range matching the slice type, yielding a first reward score for each document slice. Based on the first reward scores of each document slice in the slice set, the total reward score of the slice set is obtained. Finally, based on whether the total reward score of the slice set meets a selection criterion related to the total reward threshold, it is determined whether to retain the slice set as the document to be segmented. The target set is determined by several factors. First, by combining paragraph identifiers for segmentation and then performing semantic similarity measurement to evaluate the reward score of document slices, the decision on whether to retain the slice set as the target set for document segmentation is made. Compared to using a fixed-size segmentation, this approach ensures the semantic integrity of document slices as much as possible. Second, by first determining the slice type (whether it's a merged block or a split block) based on paragraph identifiers before semantic similarity measurement, and then determining the semantic measurement range accordingly, the computational cost of semantic similarity measurement is reduced compared to directly measuring the semantic similarity of adjacent sentences, thus improving the processing speed of document segmentation. Therefore, this approach ensures the semantic integrity of document slices as much as possible and improves the processing speed of document segmentation. Attached Figure Description

[0012] Figure 1 is a flowchart illustrating an embodiment of the document segmentation method of this application; Figure 2 is a flowchart illustrating an embodiment of the knowledge base construction method of this application; Figure 3 is a schematic diagram illustrating the framework of an embodiment of the document segmentation device of this application; Figure 4 is a schematic diagram illustrating the framework of an embodiment of the knowledge base construction device of this application; Figure 5 is a schematic diagram illustrating the framework of an embodiment of the electronic device of this application; Figure 6 is a schematic diagram illustrating the framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0014] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0015] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0016] Please refer to Figure 1, which is a flowchart illustrating an embodiment of the document segmentation method of this application. It should be noted that the process steps in this embodiment can be executed by an electronic device with computing capabilities or related equipment containing such electronic devices, such as, but not limited to, smartphones, tablets, microcomputers, servers, etc. The specific type of device is not limited here. Specifically, this embodiment may include the following steps: Step S11: Segment the document to be segmented based on each document segment marked with a segment identifier, to obtain a slice set containing several document slices.

[0017] In one implementation scenario, after obtaining the document to be segmented, the number of paragraphs and the total number of tokens in the document can be counted first. Then, a paragraph identifier is marked for each document paragraph. It should be noted that the paragraph identifier can include a beginning paragraph identifier and an end paragraph identifier. The beginning paragraph identifier can be located at the beginning of a document paragraph, and the end paragraph identifier can be located at the end of a document paragraph. For example, the beginning paragraph identifier of the i-th document paragraph can be denoted as Bi, and the end paragraph identifier of the i-th document paragraph can be denoted as Ei. Of course, the above example is only one possible example of paragraph identifiers in practical applications; other possible scenarios are not limited here, nor will they be listed in detail.

[0018] In one implementation scenario, as a possible approach, to segment a document, a segmenter agent can segment the document marked with paragraph identifiers to obtain a set of slices containing several document slices. It should be noted that the segmenter agent can segment based on document structure (e.g., directly segmenting by paragraph, chapter, etc.), document type (e.g., segmenting based on clauses if the document is a legal provision), etc. These are not limited here, nor will they be listed individually.

[0019] In another implementation scenario, distinct from the aforementioned implementation, as another possible approach, the segmenter agent can further include multiple document segmenters of different types to segment the document to be segmented, and each document segmenter can have several delimiter combinations built-in. Based on this, each document segmenter can be selected as the current segmenter, and each delimiter combination within the current segmenter can be selected as the current combination. Then, the document to be segmented can be segmented based on the current segmenter and its current combinations to obtain a set of slices. That is, when the segmenter agent contains M document segmenters, if these M segmenters contain a total of N delimiter combinations, the document to be segmented can be segmented N times to obtain N slice sets. This method, by configuring multiple different types of document segmenters in the segmenter agent, and with each segmenter having several built-in delimiter combinations, allows for the use of different segmentation strategies as much as possible with the document to be segmented, helping to explore the optimal segmentation strategy for the document to be segmented.

[0020] In a specific implementation scenario, different types of document splitters can be applied to different splitting situations. For example, some document splitters are suitable for splitting based on a specific document structure (such as document splitters that split by chapters, document splitters that split by paragraphs, etc.), while others are suitable for splitting based on a specific document type (such as document splitters that split by legal clauses, etc.).

[0021] In a specific implementation scenario, each document splitter can have at least one built-in delimiter combination. For example, some document splitters can have one built-in delimiter combination, while others can have multiple built-in delimiter combinations. The specific number of built-in delimiter combinations for each document splitter is not limited here, nor will examples be provided. It should be noted that the delimiter combination can be considered as the specific logic by which the document splitter implements document segmentation. For example, taking the delimiter combination {\n,\n\n} as an example, the newline character "\n" combined with the blank line "\n\n" can distinguish different paragraphs, and thus the document can be segmented according to paragraphs. Or, for another example, taking the delimiter combination { / <h1>-< / h1> <h6> / , <section>},in, Define paragraph-level breaks in an HTML (Hypertext Markup Language) document. <h1>-< / h1> <h6>Define heading hierarchy separators in an HTML document. Define list item separators in the HTML document. <section>Define semantic block separators in an HTML document to segment it. It should be noted that the examples above are merely a few possible combinations of separators in practical applications; other possible combinations of separators are not limited here, nor will they be listed in detail.

[0022] It should be noted that the two implementation methods for segmenting the document to be segmented described above are merely two possible implementation methods in practical applications. Other possible implementation methods are not limited here, nor will they be listed one by one. In addition, in the above segmentation process, the document to be segmented can be segmented first by combining the document segmenter and its delimiter combination to obtain several initial slices. Then, based on the conditional constraint of the word threshold, it can be determined whether adjacent initial slices need to be merged to obtain a slice set containing each document slice. For example, by combining a document segmenter and its delimiter combination, after segmenting the document to be segmented, initial slices can be obtained: slice1, slice2, slice3, ..., sliceL. If the total number of tokens in the initial slice1 is not less than the token threshold, or if the total number of tokens in the initial slice1 is less than the token threshold but exceeds the token threshold after adding the total number of tokens in the initial slice2, then the initial slice1 can be treated as a document slice, and the above judgment process can be repeated starting from the initial slice2. Conversely, if the total number of tokens in the initial slice1 is less than the token threshold and the total number of tokens in the initial slice2 still does not exceed the token threshold, then the initial sliceL can be added. If the total number of tokens in slice3 exceeds the token threshold, the initial slices slice1 and slice2 can be merged into a single document slice. The above judgment process is then repeated starting from slice3. If the total number of tokens in slice4 is still below the token threshold, the total number of tokens in slice4 can be added. If the total number of tokens in slice4 exceeds the token threshold, the initial slices slice1, slice2, and slice3 can be merged into a single document slice. The above judgment process is then repeated starting from slice4. If the total number of tokens in slice5 is still below the token threshold, the total number of tokens in slice5 can be added. This process is repeated until a slice set containing several document slices is obtained.

[0023] Step S12: Determine the slice type of the document slice based on the paragraph identifier in the document slice.

[0024] In this embodiment of the disclosure, the document slice type can be either a merged block or a split block. It should be noted that a merged block represents a document block formed by combining at least a portion of the content from different paragraphs, while a split block represents a document block formed by separately splitting a portion of the content from a single paragraph.

[0025] In one implementation scenario, as a possible approach, to distinguish the slice type of a document slice based on paragraph identifiers, the total number of each type of identifier in the document slice can be counted. For example, as mentioned earlier, identifier types can include paragraph start identifiers and paragraph end identifiers, so the total number of paragraph start identifiers Bi and paragraph end identifiers Ei in the document slice can be counted. Based on this, in response to more than one paragraph start identifier, or more than one paragraph end identifier, or both paragraph start and paragraph end identifiers being one, the slice type of the document slice can be determined to be a merged block; in response to only one paragraph start identifier and no paragraph end identifier, or only one paragraph end identifier and no paragraph start identifier, the slice type of the document slice can be determined to be a split block. The above method, by counting the total number of each type of identifier in the document slice and then distinguishing different situations to determine the slice type of the document slice, can improve the accuracy and convenience of determining the slice type.

[0026] In another implementation scenario, distinct from the aforementioned implementation, as another possible approach, to distinguish the slice type of a document slice based on paragraph identifiers within the slice, a large model instruction can be constructed based on the definition text of merged blocks and split blocks (e.g., the aforementioned definition text for merged blocks, "A merged block represents a document block formed by combining at least some content from different paragraphs," and the definition text for split blocks, "A split block represents a document block formed by separately splitting some content from a single paragraph"), and the document slice. This large model instruction instructs the large language model to determine whether a document slice belongs to a merged block or a split block based on the definition texts of the merged and split blocks. Based on this, the output content of the large language model in response to the large model instruction can be obtained to determine the slice type of the document slice. In other words, the general understanding capability of the large language model can be directly utilized, referring to the definition texts of the merged and split blocks, to determine the slice type of the document slice, thus improving the efficiency of slice type determination.

[0027] Step S13: Perform semantic similarity measurement on the document slice based on the semantic measurement range that matches the slice type of the document slice, and obtain the first reward score of the document slice.

[0028] It should be noted that, as mentioned earlier, for any document slice, its slice type can be either a merged block or a split block. Based on this, when a document slice is a merged block, the semantic measurement range matching the slice type is within the slice. In other words, in this case, semantic similarity measurement can be directly performed based on the document slice within its slice to obtain the first reward score for the document slice. Conversely, when a document slice is a split block, the semantic measurement range matching the slice type is between slices. In other words, in this case, semantic similarity measurement can be performed based on the document slice and its adjacent document slices to obtain the first reward score for the document slice.

[0029] In one implementation scenario, when the document slice type is a merged block, the first semantic similarity between the first and last sentences within the document slice can be obtained. Based on this, in response to the first semantic similarity satisfying a first judgment condition related to a first similarity threshold, a first preset score can be assigned to the document slice as its first reward score; conversely, in response to the first semantic similarity not satisfying the first judgment condition related to the first similarity threshold, a second preset score can be assigned to the document slice as its first reward score. It should be noted that the first preset score can be higher than the second preset score. This method, by setting the first preset score (when the first semantic similarity satisfies the first judgment condition) higher than the second preset score (when the first semantic similarity does not satisfy the first judgment condition), enables a higher reward for high semantic similarity between the first and last sentences within a document slice and a lower reward for low semantic similarity between the first and last sentences within a document slice, when the document slice type is a merged block.

[0030] In a specific implementation scenario, pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) can be used to extract the semantics of the first and last sentences of a sliced ​​document, obtaining the semantic features of each sentence. Based on this, the first semantic similarity between the first and last sentences can be obtained through metrics such as cosine similarity.

[0031] In a specific implementation scenario, the first judgment condition related to the first similarity threshold can be set as: the first semantic similarity is higher than the first similarity threshold, or the first semantic similarity is not lower than the first similarity threshold, without any limitation here.

[0032] In a specific implementation scenario, the first preset score and the second preset score can be set to opposite numbers, for example, the first preset score can be set to K2 (K2 can be a positive number), and the second preset score can be set to -K2. Of course, the above example is only one possible way to set the first preset score and the second preset score. Other possible settings are not limited here, nor will they be listed one by one.

[0033] In one implementation scenario, when the document slice type is a split block, a sentence pair can be selected based on the identifier type of the paragraph identifier within the document slice. This pair can be either the last statement in the document slice and the first statement in the next document slice, or the first statement in the document slice and the last statement in the previous document slice. Based on this, a second semantic similarity between the two statements in the sentence pair can be obtained. Then, in response to the second semantic similarity satisfying a second judgment condition related to a second similarity threshold, a third preset score can be assigned to the document slice as its first reward score. Conversely, in response to the second semantic similarity not satisfying the second judgment condition related to the second similarity threshold, a fourth preset score can be assigned to the document slice as its first reward score. It should be noted that the fourth preset score is higher than the third preset score. The above method, by setting a third preset score when the second semantic similarity meets the second judgment condition to be lower than a fourth preset score when the second semantic similarity does not meet the second judgment condition, can give low rewards when there is a high semantic similarity between the first and last sentences of document slices and high rewards when there is a low semantic similarity between the first and last sentences of document slices, when the slice type of document slice is split block.

[0034] In a specific implementation scenario, to select statement pairs for a document slice based on the identifier type of the paragraph identifier within the document slice, in response to the paragraph identifier type being a paragraph beginning identifier, the statement at the end of the document slice and the statement at the beginning of the next document slice can be selected as the statement pair. Conversely, in response to the paragraph identifier type being a paragraph end identifier, the statement at the beginning of the document slice and the statement at the end of the previous document slice can be selected as the statement pair. That is, when the document slice is a split block, if the document slice contains a paragraph identifier Bi, it means that the first half of the i-th document paragraph has been split into a separate document slice. In this case, to check whether the split is reasonable, the statement at the end of this document slice and the statement at the beginning of the next document slice (both of which were originally in the i-th document paragraph) can be selected as the statement pair. Similarly, when a document slice is a split block, if the document slice contains a paragraph identifier Ei, it means that the latter half of the i-th document paragraph is split into a separate document slice. In this case, in order to check whether the split is reasonable, the first statement of this document slice and the last statement of the previous document slice (both of which were originally in the i-th document paragraph) can be selected as a statement pair.

[0035] In a specific implementation scenario, after selecting the sentence pair, the second semantic similarity between the two sentences in the sentence pair can be obtained. Specifically, referring to the aforementioned method for measuring the first semantic similarity, the semantics of the two sentences in the sentence pair can be extracted separately using a pre-trained language model such as BERT (Bidirectional Encoder Representations from Transformers) to obtain the semantic features of each sentence in the sentence pair. Then, the second semantic similarity between the two sentences in the sentence pair can be obtained using a measurement method such as cosine similarity.

[0036] In a specific implementation scenario, the second judgment condition related to the second similarity threshold can be set as: the second semantic similarity is higher than the second similarity threshold, or the second semantic similarity is not lower than the second similarity threshold, without any limitation here.

[0037] In a specific implementation scenario, the third and fourth preset scores can be set to opposite numbers, for example, the fourth preset score can be set to K3 (K3 can be a positive number), then the third preset score can be set to -K2. Of course, the above example is only one possible way to set the third and fourth preset scores, and other possible settings are not limited here, nor will they be listed one by one.

[0038] Step S14: Based on the first reward score of each document slice in the slice set, obtain the total reward score of the slice set.

[0039] In one implementation scenario, as a possible approach, the total reward score of the slice set can be obtained by summing the first reward scores of each document slice in the slice set.

[0040] In another implementation scenario, unlike the aforementioned implementation, as another possible approach, before calculating the total reward score, a weighted average can be performed based on the first number of document paragraphs in the document to be segmented and the second number of fixed blocks determined by the fixed block word count in the document to be segmented, to obtain the expected number of slices. Then, based on the numerical difference between the actual number of slices in the slice set and the expected number of slices, a second reward score is obtained, and the second reward score is negatively correlated with the numerical difference. The total reward score can then be obtained based on the first and second reward scores of each document slice in the slice set. This method, before calculating the total reward score, first combines the first number of document paragraphs in the document to be segmented and the second number of fixed blocks determined by the fixed block word count in the document to be segmented to obtain the expected number of slices. Then, based on the numerical difference between this and the actual number of slices in the slice set, a second reward score negatively correlated with the numerical difference is obtained. Finally, combining the first and second reward scores, the total reward score is obtained. This approach combines the second reward score from a global dimension and the first reward score from a local dimension to calculate the total reward score, which helps improve the comprehensiveness of the total reward score.

[0041] In a specific implementation scenario, the second number of fixed blocks in the document to be segmented, determined by the fixed block character count, can be obtained by dividing the total number of tokens in the document by the fixed block character count (i.e., the fixed block token count). Based on this, the expected number of slices, Onum, can be obtained by weighting the first number of document paragraphs and the second number of fixed blocks: Onum = Lamda1 * First Number + Lamda2 * Second Number. In the above formula, Lamda1 and Lamda2 represent the weighting factors for the first and second numbers, respectively. Therefore, by weighting the first and second numbers, the expected number of slices can be made to fall between the pure paragraph segmentation score and the fixed block size segmentation score.

[0042] In a specific implementation scenario, after obtaining the expected number of slices, the numerical difference between the actual number of slices in the slice set and the expected number of slices (i.e., the absolute value of the difference) can be obtained. A second reward score, negatively correlated with this numerical difference, can then be derived. For example, the second reward score can be obtained by multiplying the numerical difference using a discount factor and taking the opposite number. For instance, the second reward score can be expressed as: -1 * |expected number of slices - actual number of slices| * K1, where K1 represents the discount factor.

[0043] It should be noted that the two different methods for calculating the total reward score described above are merely possible examples in practical applications. Other possible calculation methods are not limited here, nor will they be awarded individually. Furthermore, in practical applications, as mentioned earlier, it is also possible to segment the document based on different document splitters and their built-in delimiter combinations, resulting in multiple slice sets. In this case, for each slice set, the corresponding total reward score can be calculated separately using the aforementioned process steps. The specific process can be found in the aforementioned descriptions and will not be repeated here.

[0044] Step S15: Based on whether the total reward score of the slice set meets the filtering conditions related to the total reward threshold, determine whether to retain the slice set as the target set of documents to be segmented.

[0045] In one implementation scenario, the filtering conditions related to the total reward threshold can be set as: the total reward score is higher than the total reward threshold, or the total reward score is not lower than the total reward threshold. There are no restrictions on the filtering conditions here.

[0046] In one implementation scenario, as mentioned earlier, in practical applications, document splitting can be performed separately based on different document splitters and their built-in delimiter combinations to obtain multiple slice sets. For each slice set, the corresponding total reward score can be calculated separately through the aforementioned process steps. In this case, the slice set with the highest total reward score can be selected, and then, based on whether the total reward score of this slice set meets the filtering conditions related to the total reward threshold, it can be determined whether to retain this slice set as the target set of the document to be split. That is, slice sets obtained by using other splitting methods for the document to be split, whose total reward score is lower than the aforementioned highest total reward score, can be directly ignored. Of course, if the document to be split only adopts one splitting method, then only the slice sets obtained by using this splitting method are subjected to the step of determining whether to retain the slice set as the target set of the document to be split based on whether the total reward score of the slice set meets the filtering conditions related to the total reward threshold.

[0047] In one implementation scenario, if the total reward score of the slice set meets the filtering criteria, the slice set can be retained as the target set of documents to be segmented; conversely, if the total reward score of the slice set does not meet the filtering criteria, the slice set can be discarded.

[0048] In one implementation scenario, as mentioned earlier, the slice set is obtained by the segmenter agent from the document to be segmented. If the total reward score of the slice set does not meet the filtering conditions related to the total reward threshold, the reward parameters used to calculate the total reward score (e.g., the aforementioned discount coefficient K1, the absolute values ​​of the first and second preset scores K2, the absolute values ​​of the third and fourth preset scores K3, the first number of weight factors Lamda1, and the second number of weight factors Lamda2) and the segmenter agent's lexical threshold can be adjusted. It should be noted that the lexical threshold is used to impose lexical constraints when merging the initial slices into document slices after the segmenter agent has obtained the initial slices. For details on how the lexical threshold is used to impose lexical constraints on merging the initial slices into document slices, please refer to the aforementioned description, which will not be repeated here. Based on this, the steps described above—segmenting the document to be segmented based on paragraph identifiers for each document segment to obtain a slice set containing several document slices—can be returned and iterated until the total reward score of the latest slice set meets the filtering conditions related to the total reward threshold. It should be noted that the next document to be segmented can utilize the adjusted reward parameters and lexical thresholds obtained when the filtering criteria are met during document segmentation. This method, by combining the total reward score with adjustments to the reward parameters and lexical thresholds and iterating accordingly, can select the most optimal segmentation strategy based on reinforcement learning.

[0049] In one implementation scenario, as a possible example in practical application, in order to achieve document segmentation, the following steps can be performed in sequence: (1) Count the first number of document paragraphs and the total number of words in the document to be segmented. Each document paragraph can be marked with a paragraph identifier, such as the beginning of the i-th document paragraph can be marked with the paragraph beginning identifier Bi and the end of the paragraph can be marked with the paragraph end identifier Ei; (2) Based on the first number of document paragraphs and the total number of words, estimate an expected number of slices (the specific calculation method can be referred to the relevant description above, and will not be repeated here); (3) Design a segmenter agent. The segmenter agent can define multiple document segmenters, such as based on fixed block size, based on document structure, based on document type, etc., and each document segmenter can have different delimiter combinations built in; (4) Randomly select a document segmenter to start the segmentation task; (5) Sequentially extract a delimiter combination and segment the document to be segmented according to this strategy to obtain a slice set; (6) Calculate the total reward score of the slice set, and the total reward The reward score = the first reward score of each document slice in the slice set + the second reward score of the slice set, where: (6.1) The second reward score of the slice set = -1 * |expected number of slices - actual number of slices| * K1, that is, the closer the actual number of slices is to the expected number of slices, the less the deduction; (6.2) For each document slice in the slice set, if there is more than one paragraph beginning identifier, or more than one paragraph ending identifier, or both paragraph beginning identifier and paragraph ending identifier are one, then the slice type of the document slice can be determined as a merged block, and then the first semantic similarity between the first and last sentences within the document slice can be obtained. In response to the first semantic similarity satisfying the first judgment condition related to the first similarity threshold, the document slice can be assigned a first preset score K2 as the first reward score of the document slice. In response to the first semantic similarity not satisfying the first judgment condition related to the first similarity threshold, the document slice can be assigned a second preset score -K2 as the first reward score of the document slice; (6.3) For each document slice in the slice set, if it has only one paragraph beginning identifier and no paragraph ending identifier, or only one paragraph ending identifier and no paragraph beginning identifier, then the slice type of the document slice can be determined as a split block. Therefore, when the identifier type of the paragraph identifier within the document slice is a paragraph beginning identifier, the last statement in the document slice and the first statement in the next document slice are selected as a statement pair. Conversely, when the identifier type of the paragraph identifier within the document slice is a paragraph ending identifier, the first statement in the document slice and the last statement in the previous document slice are selected as a statement pair. This allows the acquisition of the second semantic similarity between the two statements in the statement pair. If the second semantic similarity satisfies the second judgment condition related to the second similarity threshold, a third preset score of -k3 can be assigned to the document slice as its first reward score. If the second semantic similarity does not satisfy the second judgment condition related to the second similarity threshold, the document slice can be... Assign the fourth preset score K3 as the first reward score for the document slice; (7) Continuously change the delimiter combination. If the delimiter combination has been changed, the document slicer can be changed, and the learning process (5) to (7) above can be repeated; (8) After all document slicers and their delimiter combinations have participated in the above process, the document slicer and its delimiter combination with the highest total reward score can be selected. If its total reward score T is not lower than (or higher than) the total reward threshold F, the slice set corresponding to this highest total reward score can be retained; (9) If the total reward score T is lower than (or not higher than) the total reward threshold F, the word threshold and Lamda1, Lamda2 in (1) and K1, K2, K3 in (6) can be adjusted to form a new score calculation strategy, and (4) to (8) can be repeated until the latest total reward score T is not lower than (or higher than) the total reward threshold F. The reinforcement learning ends, and the score calculation strategy is recorded as the initial score calculation strategy for the next document to be sliced.

[0050] The above scheme segments the document to be segmented based on paragraphs marked with paragraph identifiers, resulting in a slice set containing several document slices. Based on the paragraph identifiers within each document slice, the slice type is determined, and the slice type is either a merged block or a split block. Then, a semantic similarity measure is performed on the document slices based on a semantic metric range matching the slice type, yielding a first reward score for each document slice. Based on the first reward scores of each document slice in the slice set, the total reward score of the slice set is obtained. Finally, based on whether the total reward score of the slice set meets a selection criterion related to the total reward threshold, it is determined whether to retain the slice set as the document to be segmented. The target set is determined by several factors. First, by combining paragraph identifiers for segmentation and then performing semantic similarity measurement to evaluate the reward score of document slices, the decision on whether to retain the slice set as the target set for document segmentation is made. Compared to using a fixed-size segmentation, this approach ensures the semantic integrity of document slices as much as possible. Second, by first determining the slice type (whether it's a merged block or a split block) based on paragraph identifiers before semantic similarity measurement, and then determining the semantic measurement range accordingly, the computational cost of semantic similarity measurement is reduced compared to directly measuring the semantic similarity of adjacent sentences, thus improving the processing speed of document segmentation. Therefore, this approach ensures the semantic integrity of document slices as much as possible and improves the processing speed of document segmentation.

[0051] Please refer to Figure 2, which is a flowchart illustrating an embodiment of the knowledge base construction method of this application. It should be noted that the process steps in this embodiment can be executed by an electronic device with computing capabilities or related equipment containing such devices, such as, but not limited to, smartphones, tablets, microcomputers, servers, etc. The specific type of device is not limited here. Specifically, this embodiment may include the following steps: Step S21: Select each document in the document library to be segmented as the document to be segmented.

[0052] Specifically, when building a knowledge base, one can first acquire a document library to be segmented. For example, taking a legal knowledge base as an example, one can acquire legal provisions, judicial interpretations, typical cases, and other documents to form a document library to be segmented. Or, taking an office knowledge base as an example, one can acquire company rules, departmental regulations, work procedures, reward and punishment regulations, and other documents to form a document library to be segmented. Of course, the above examples are just a few possible examples of acquiring a document library to be segmented when building a legal knowledge base and an office knowledge base. Other possible scenarios are not limited here, nor will they be listed one by one. Based on this, each document in the document library to be segmented can be selected as the document to be segmented.

[0053] Step S22: Segment the document to be segmented to obtain the target set of the document to be segmented.

[0054] In this embodiment of the disclosure, the target set includes several document slices of the document to be segmented, and the target set is obtained by segmenting through the process steps in the above-described document segmentation method embodiment. For details, please refer to the aforementioned document segmentation method embodiment, which will not be repeated here.

[0055] Step S23: Construct a knowledge base based on the target set of each document in the document library to be segmented.

[0056] Specifically, the target sets obtained after segmenting each document to be segmented can be included in the knowledge base. For example, document "A" in the document library to be segmented can be segmented to obtain target set "A", document "B" in the document library to be segmented can be segmented to obtain target set "B", and so on. Target sets "A", "B", etc., can then be included in the knowledge base together. Of course, the above example is only one possible case in practical application, and other possible scenarios are not limited here, nor will they be listed here.

[0057] The above scheme selects each document in the document library to be segmented as the document to be segmented, and segments it to obtain a target set of documents to be segmented. The target set includes several document slices of the document to be segmented, and the target set is obtained by segmentation through the process steps in the document segmentation method embodiment described above. Therefore, it can ensure the semantic integrity of the document slices as much as possible and improve the processing speed of document segmentation. Then, based on the target set of each document in the document library to be segmented, a knowledge base is constructed. Therefore, it can improve the semantic integrity of the document slices in the knowledge base and improve the construction efficiency of the knowledge base.

[0058] Please refer to Figure 3, which is a schematic diagram of the framework of an embodiment of the document segmentation device of this application. The document segmentation device 30 includes: an initial segmentation module 31, a type determination module 32, a similarity measurement module 33, a reward determination module 34, and a set acquisition module 35. The initial segmentation module 31 is used to segment the document to be segmented based on each document segment being marked with a paragraph identifier, to obtain a slice set containing several document slices; the type determination module 32 is used to determine the slice type of the document slice based on the paragraph identifier in the document slice; wherein, the slice type of the document slice belongs to either a merge block or a split block; the similarity measurement module 33 is used to perform semantic similarity measurement on the document slice based on the semantic measurement range that matches the slice type of the document slice, to obtain a first reward score for the document slice; the reward determination module 34 is used to obtain a total reward score for the slice set based on the first reward score of each document slice in the slice set; the set acquisition module 35 is used to determine whether to retain the slice set as the target set of documents to be segmented based on whether the total reward score of the slice set meets the screening conditions related to the total reward threshold.

[0059] In the above scheme, the document segmentation device 30 segments the document to be segmented based on each document segment being marked with a paragraph identifier, obtaining a slice set containing several document slices. Based on the paragraph identifiers in the document slices, the slice type of each document slice is determined, and the slice type belongs to either a merge block or a split block. Then, based on the semantic similarity measurement range matching the slice type, a semantic similarity measurement is performed on the document slices to obtain a first reward score for each document slice. Based on the first reward scores of each document slice in the slice set, the total reward score of the slice set is obtained. Finally, based on whether the total reward score of the slice set meets the screening conditions related to the total reward threshold, it is determined whether to retain the slice set as a candidate. The target set for document segmentation is determined by several factors. First, segmentation is performed using paragraph identifiers, followed by semantic similarity measurement to evaluate the reward score of each segment. This determines whether to retain the segment set as the target set for document segmentation. Compared to segmentation with a fixed size, this approach ensures greater semantic integrity of the document segments. Second, before semantic similarity measurement, the paragraph identifiers within the document segments are used to determine whether the segment type is a merged block or a split block. Based on this segment type, the semantic measurement range is determined. Compared to directly measuring the semantic similarity of adjacent sentences, this reduces the computational cost of semantic similarity measurement, thus improving document segmentation processing speed. Therefore, this approach maximizes the semantic integrity of document segments and improves document segmentation processing speed.

[0060] In some disclosed embodiments, when the document slice type is a merged block, the semantic measurement range matching the document slice type is within the slice. The similarity measurement module 33 includes a first acquisition submodule for acquiring the first semantic similarity between the first and last sentences within the document slice; the similarity measurement module 33 includes a first response submodule for assigning a first preset score as a first reward score to the document slice in response to the first semantic similarity satisfying a first judgment condition related to the first similarity threshold; the similarity measurement module 33 includes a second response submodule for assigning a second preset score as a first reward score to the document slice in response to the first semantic similarity not satisfying the first judgment condition related to the first similarity threshold; wherein, the first preset score is higher than the second preset score.

[0061] In some disclosed embodiments, when the document slice type is splitting, the semantic measurement range matching the document slice type is between slices. The similarity measurement module 33 includes a statement selection submodule, used to select the last statement in the document slice and the first statement in the next document slice as a statement pair based on the identifier type of the paragraph identifier within the document slice, or to select the first statement in the document slice and the last statement in the previous document slice as a statement pair. The similarity measurement module 33 includes a second acquisition submodule, used to acquire the second semantic similarity between the two statements in the statement pair. The similarity measurement module 33 includes a third response submodule, used to assign a third preset score as the first reward score of the document slice in response to the second semantic similarity satisfying a second judgment condition related to the second similarity threshold. The similarity measurement module 33 includes a fourth response submodule, used to assign a fourth preset score as the first reward score of the document slice in response to the second semantic similarity not satisfying the second judgment condition related to the second similarity threshold. The fourth preset score is higher than the third preset score.

[0062] In some disclosed embodiments, the statement selection submodule includes a first selection unit, configured to select the last statement in the document slice and the first statement in the next document slice as a statement pair in response to the identifier type of the paragraph identifier in the document slice being a paragraph beginning identifier; the statement selection submodule includes a second selection unit, configured to select the first statement in the document slice and the last statement in the previous document slice as a statement pair in response to the identifier type of the paragraph identifier in the document slice being a paragraph ending identifier.

[0063] In some disclosed embodiments, the identifier type of the paragraph identifier is either a paragraph start identifier or a paragraph end identifier. The type determination module 32 includes a quantity statistics submodule, which is used to count the total number of each identifier type in the document slice. The type determination module 32 includes a first determination submodule, which is used to determine the slice type of the document slice as a merged block in response to the fact that there is more than one paragraph start identifier, or more than one paragraph end identifier, or both a paragraph start identifier and a paragraph end identifier are one. The type determination module 32 includes a second determination submodule, which is used to determine the slice type of the document slice as a split block in response to the fact that there is only one paragraph start identifier and no paragraph end identifier, or only one paragraph end identifier and no paragraph start identifier.

[0064] In some disclosed embodiments, the document segmentation device 30 includes an expected calculation module, used to calculate the expected number of slices by weighting the first number of document paragraphs in the document to be segmented and the second number of fixed blocks in the document to be segmented according to the fixed number of words in the fixed blocks; the document segmentation device 30 includes a global reward module, used to calculate a second reward score based on the numerical difference between the actual number of slices in the slice set and the expected number of slices; wherein the second reward score is negatively correlated with the numerical difference; the reward determination module 34 is specifically used to calculate the total reward score based on the first reward score and the second reward score of each document slice in the slice set.

[0065] In some disclosed embodiments, the slice set is obtained by the segmenter agent segmenting the document to be segmented. The document segmentation device 30 includes a parameter adjustment module, which is used to adjust the reward parameter used to calculate the total reward score and the word threshold of the segmenter agent when the total reward score of the slice set does not meet the screening conditions related to the total reward threshold. The word threshold is used to impose word constraints when the initial slices are merged into document slices after the segmenter agent has segmented the initial slices. The document segmentation device 30 includes a loop iteration module, which is used to return the step of segmenting the document to be segmented based on each document paragraph marked with a paragraph identifier to obtain a slice set containing several document slices and iterates until the total reward score of the latest slice set meets the screening conditions related to the total reward threshold. The next document to be segmented is subject to the reward parameter and word threshold adjusted when the screening conditions are met.

[0066] In some disclosed embodiments, the slice set is obtained by a segmenter agent segmenting the document to be segmented. The segmenter agent contains multiple document segmenters of different types. Each document segmenter has a number of delimiter combinations built in. The initial segmentation module 31 includes a tool selection submodule, which selects each document segmenter as the current segmenter and selects each delimiter combination in the current segmenter as the current combination. The initial segmentation module 31 also includes a document segmentation submodule, which is used to segment the document to be segmented based on the current segmenter and the current combination in the current segmenter to obtain the slice set.

[0067] Please refer to Figure 4, which is a schematic diagram of the framework of an embodiment of the knowledge base construction device of this application. The knowledge base construction device 40 includes: a document selection module 41, a document segmentation module 42, and a knowledge base construction module 43. The document selection module 41 is used to select each document in the document library to be segmented as the document to be segmented. The document segmentation module 42 is used to segment the document to be segmented to obtain a target set of the document to be segmented. The target set includes several document slices of the document to be segmented, and the target set is obtained by the document segmentation device in the third aspect described above. The knowledge base construction module 43 is used to construct a knowledge base based on the target set of each document in the document library to be segmented.

[0068] In the above scheme, the knowledge base construction device 40 selects each document in the document library to be segmented as the document to be segmented, and segments the document to be segmented to obtain a target set of the document to be segmented. The target set includes several document slices of the document to be segmented, and the target set is obtained by the document segmentation device in the above-described document segmentation device embodiment. Therefore, it can ensure the semantic integrity of the document slices as much as possible and improve the processing speed of document segmentation. Then, based on the target set of each document in the document library to be segmented, the knowledge base is constructed. Therefore, it can improve the semantic integrity of the document slices in the knowledge base and improve the construction efficiency of the knowledge base.

[0069] Please refer to Figure 5, which is a schematic diagram of the framework of an embodiment of the electronic device of this application. The electronic device 50 includes at least a memory 51 and a processor 52 coupled to each other. The memory 51 stores at least program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above-described document segmentation method embodiments, or to implement the steps in any of the above-described knowledge base construction method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here.

[0070] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described document segmentation method embodiments, or the steps in any of the above-described knowledge base construction method embodiments. See the foregoing disclosed embodiments for details. Processor 52 can also be called a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0071] In the above scheme, the electronic device 50 segments the document to be segmented based on each document segment marked with a paragraph identifier, obtaining a slice set containing several document slices. Based on the paragraph identifiers in the document slices, the slice type of each document slice is determined, and the slice type belongs to either a merged block or a split block. Then, based on the semantic similarity measurement range matching the slice type, a semantic similarity measurement is performed on the document slices to obtain a first reward score for each document slice. Based on the first reward scores of each document slice in the slice set, the total reward score of the slice set is obtained. Finally, based on whether the total reward score of the slice set meets the screening conditions related to the total reward threshold, it is determined whether to retain the slice set as a target for segmentation. The target set for document segmentation, by combining paragraph identifiers for segmentation and then further evaluating the reward score of document slices using semantic similarity measurement, determines whether to retain the slice set as the target set for document segmentation. Compared to using fixed-size segmentation, this approach ensures the semantic integrity of document slices as much as possible. Furthermore, before semantic similarity measurement, the paragraph identifiers within the document slices are used to determine whether the slice type is a merged block or a split block, and the semantic measurement range is determined based on this slice type. Compared to directly performing semantic similarity measurement on adjacent sentences, this reduces the computational load required for semantic similarity measurement, thus improving document segmentation processing speed. Therefore, this approach ensures the semantic integrity of document slices as much as possible and improves document segmentation processing speed. Furthermore, each document in the document library to be segmented is selected as the document to be segmented, and segmentation is performed based on the document to be segmented to obtain a target set of documents to be segmented. The target set includes several document slices of the document to be segmented, and the target set is obtained by segmentation through the process steps in the above document segmentation method embodiment. Therefore, it can ensure the semantic integrity of the document slices as much as possible and improve the processing speed of document segmentation. Then, based on the target set of each document in the document library to be segmented, a knowledge base is constructed. Therefore, it can improve the semantic integrity of the document slices in the knowledge base and improve the construction efficiency of the knowledge base.

[0072] Please refer to Figure 6, which is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described document segmentation method embodiments, or to implement the steps in the above-described knowledge base construction method embodiments.

[0073] In the above scheme, the computer-readable storage medium 60 segments the document to be segmented based on each document segment being marked with a paragraph identifier, resulting in a slice set containing several document slices. Based on the paragraph identifiers in the document slices, the slice type of each document slice is determined, and the slice type belongs to either a merged block or a split block. Then, based on a semantic similarity measurement range matching the slice type, the document slices are subjected to semantic similarity measurement to obtain a first reward score for each document slice. Based on the first reward scores of each document slice in the slice set, the total reward score of the slice set is obtained. Finally, based on whether the total reward score of the slice set meets the screening conditions related to the total reward threshold, it is determined whether to retain the entire slice set. The target set of documents to be segmented is used. Firstly, by combining paragraph identifiers for segmentation and then further performing semantic similarity measurement to evaluate the reward score of document slices, the decision is made on whether to retain the slice set as the target set of documents to be segmented. Compared to segmenting with a fixed size, this approach ensures the semantic integrity of document slices as much as possible. Secondly, before semantic similarity measurement, the paragraph identifiers in the document slices are used to determine whether the slice type is a merged block or a split block, and then the semantic measurement range is determined based on this slice type. Compared to directly performing semantic similarity measurement on adjacent sentences, this reduces the computational load required for semantic similarity measurement, thus improving the processing speed of document segmentation. Therefore, this approach ensures the semantic integrity of document slices as much as possible and improves the processing speed of document segmentation. Furthermore, each document in the document library to be segmented is selected as the document to be segmented, and segmentation is performed based on the document to be segmented to obtain a target set of documents to be segmented. The target set includes several document slices of the document to be segmented, and the target set is obtained by segmentation through the process steps in the above document segmentation method embodiment. Therefore, it can ensure the semantic integrity of the document slices as much as possible and improve the processing speed of document segmentation. Then, based on the target set of each document in the document library to be segmented, a knowledge base is constructed. Therefore, it can improve the semantic integrity of the document slices in the knowledge base and improve the construction efficiency of the knowledge base.

[0074] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0075] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, the personal information processing rules are clearly informed through signs / information, and authorization is obtained through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.< / section> < / h6> < / section> < / h6>

Claims

1. A document segmentation method, characterized in that, include: The document to be segmented is segmented based on each document segment being marked with a segment identifier, resulting in a slice set containing several document slices; Based on the paragraph identifiers in the document slice, the slice type of the document slice is determined; wherein, the slice type of the document slice belongs to either a merged block or a split block; based on the semantic metric range that matches the slice type of the document slice, a semantic similarity metric is performed on the document slice to obtain a first reward score for the document slice; based on the first reward scores of each document slice in the slice set, the total reward score of the slice set is obtained; based on whether the total reward score of the slice set meets the filtering conditions related to the total reward threshold, it is determined whether to retain the slice set as the target set of the documents to be segmented.

2. The method according to claim 1, characterized in that, When the document slice's slice type is the merged block, the semantic measurement range matching the document slice's slice type is within the slice. The step of performing semantic similarity measurement on the document slice based on the semantic measurement range matching the document slice's slice type to obtain a first reward score for the document slice includes: obtaining a first semantic similarity between the first and last sentences within the document slice; in response to the first semantic similarity satisfying a first judgment condition related to a first similarity threshold, assigning a first preset score as the document slice's first reward score; in response to the first semantic similarity not satisfying the first judgment condition related to the first similarity threshold, assigning a second preset score as the document slice's first reward score; wherein the first preset score is higher than the second preset score.

3. The method according to claim 1, characterized in that, When the document slice's slice type is the split, the semantic measurement range matching the document slice's slice type is the inter-slice range. The step of performing semantic similarity measurement on the document slice based on the semantic measurement range matching the document slice's slice type to obtain the document slice's first reward score includes: selecting, based on the identifier type of the paragraph identifier within the document slice, a statement pair between the last statement in the document slice and the first statement in the next document slice, or a statement pair between the first statement in the document slice and the last statement in the previous document slice; obtaining a second semantic similarity between the two statements in the statement pair; in response to the second semantic similarity satisfying a second judgment condition related to a second similarity threshold, assigning a third preset score as the document slice's first reward score; in response to the second semantic similarity not satisfying the second judgment condition related to the second similarity threshold, assigning a fourth preset score as the document slice's first reward score; wherein the fourth preset score is higher than the third preset score.

4. The method according to claim 3, characterized in that, The identifier type is either a paragraph beginning identifier or a paragraph ending identifier. The step of selecting the last statement in the document slice and the first statement in the next document slice as a statement pair, or selecting the first statement in the document slice and the last statement in the previous document slice as a statement pair, based on the identifier type of the paragraph identifier within the document slice, includes: responding to the paragraph beginning identifier being the identifier type of the paragraph identifier within the document slice, selecting the last statement in the document slice and the first statement in the next document slice as the statement pair; responding to the paragraph ending identifier being the identifier type of the paragraph identifier within the document slice, selecting the first statement in the document slice and the last statement in the previous document slice as the statement pair.

5. The method according to claim 1, characterized in that, The paragraph identifier can be either a paragraph start identifier or a paragraph end identifier. Determining the slice type of the document slice based on the paragraph identifiers in the document slice includes: counting the total number of each of the different identifier types in the document slice; determining the slice type of the document slice as a merged block in response to the following: if there is more than one paragraph start identifier, or more than one paragraph end identifier, or both the paragraph start identifier and the paragraph end identifier are the same; and determining the slice type of the document slice as a split block in response to the following: if there is only one paragraph start identifier and no paragraph end identifier, or only one paragraph end identifier and no paragraph start identifier.

6. The method according to claim 1, characterized in that, Before obtaining the total reward score of the slice set based on the first reward score of each document slice in the slice set, the method further includes: weighting the expected number of slices based on the first number of document paragraphs in the document to be segmented and the second number of fixed blocks determined by the fixed block word count in the document to be segmented; obtaining a second reward score based on the numerical difference between the actual number of slices in the slice set and the expected number of slices; wherein the second reward score is negatively correlated with the numerical difference; obtaining the total reward score of the slice set based on the first reward score of each document slice in the slice set includes: obtaining the total reward score based on the first reward score and the second reward score of each document slice in the slice set.

7. The method according to claim 1, characterized in that, The slice set is obtained by the segmenter agent segmenting the document to be segmented. If the total reward score of the slice set does not meet the filtering conditions related to the total reward threshold, the method further includes: adjusting the reward parameter used to calculate the total reward score and the lexical threshold of the segmenter agent; wherein the lexical threshold is used to: constrain lexical elements when the initial slices are merged into document slices after the segmenter agent has segmented the document; returning to the step of segmenting the document to be segmented based on each document paragraph marked with a paragraph identifier to obtain a slice set containing several document slices and iterating until the total reward score of the latest slice set meets the filtering conditions related to the total reward threshold; wherein the next document to be segmented applies the reward parameter and lexical threshold adjusted when the filtering conditions are met during document segmentation.

8. The method according to claim 1, characterized in that, The slice set is obtained by a segmenter agent segmenting the document to be segmented. The segmenter agent contains multiple document segmenters of different types, and each document segmenter has a number of delimiter combinations built in. The process of segmenting the document to be segmented based on each document paragraph marked with a paragraph identifier to obtain a slice set containing multiple document slices includes: selecting each of the document segmenters as the current segmenter, and selecting each of the delimiter combinations in the current segmenter as the current combination; and segmenting the document to be segmented based on the current segmenter and the current combination in the current segmenter to obtain the slice set.

9. A method for constructing a knowledge base, characterized in that, include: Select each document in the document library to be split as the document to be split; Based on the document to be segmented, a target set of the document to be segmented is obtained; wherein, the target set includes several document slices of the document to be segmented, and the target set is obtained by segmenting by the document segmentation method according to any one of claims 1 to 8; a knowledge base is constructed based on the target set of each document in the document library to be segmented.

10. A document segmentation device, characterized in that, include: The initial segmentation module is used to segment the document to be segmented based on each document segment being marked with a segment identifier, and to obtain a slice set containing several document slices; A type determination module is used to determine the slice type of the document slice based on the paragraph identifier in the document slice; wherein the slice type of the document slice belongs to either a merged block or a split block; a similarity measurement module is used to perform semantic similarity measurement on the document slice based on the semantic measurement range that matches the slice type of the document slice, and obtain a first reward score for the document slice; a reward determination module is used to obtain a total reward score for the slice set based on the first reward scores of each document slice in the slice set; a set acquisition module is used to determine whether to retain the slice set as the target set of the documents to be segmented based on whether the total reward score of the slice set meets the filtering conditions related to the total reward threshold.

11. A knowledge base construction apparatus, characterized in that, include: The document selection module is used to select each document in the document library to be split as the document to be split. A document segmentation module is used to segment the document to be segmented to obtain a target set of the document to be segmented; wherein, the target set includes a plurality of document slices of the document to be segmented, and the target set is obtained by the document segmentation device of claim 10; The knowledge base construction module is used to construct a knowledge base based on the target sets of each document in the document library to be segmented.

12. An electronic device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the document segmentation method according to any one of claims 1 to 8, or the knowledge base construction method according to claim 9.

13. A computer-readable storage medium, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the document segmentation method according to any one of claims 1 to 8, or the knowledge base construction method according to claim 9.

Images