Text chunking method, apparatus, storage medium, and computer device

Abstract

The application discloses a text chunking method and device, a storage medium and a computer device. The method comprises the following steps: obtaining an initial text, and performing segmentation processing on the initial text to obtain a plurality of target texts; performing division processing on the target texts by using identifiers to obtain a plurality of initial sentences; performing splicing processing on the plurality of initial sentences to obtain a plurality of target sentences, and determining a semantic threshold of the target sentences based on the plurality of target sentences, wherein the semantic threshold is used for representing the semantic similarity of different target sentences; and performing chunking processing on the target texts according to a text chunking strategy based on the semantic threshold to obtain a plurality of target chunks, wherein the text chunking strategy is used for representing the chunking rule of the target texts, so that the technical problem of low text retrieval accuracy is solved, and the technical effect of improving the text retrieval accuracy is achieved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a text segmentation method, apparatus, storage medium, and computer device. Background Technology

[0002] Currently, large-scale models can solve most general questions in question-answering operations. However, for certain specific domains, to achieve a better question-answering experience, the mainstream approach is to fine-tune large-scale models and build knowledge bases. The main role of the knowledge base is to act as an external brain for the large-scale model. After clarifying the user's query intent and rewriting the question, it retrieves relevant knowledge blocks from the knowledge base, constructs prompt words based on these prompts along with the user's question, and then submits them to the large-scale model for answering.

[0003] Furthermore, the granularity of text segmentation in the knowledge base directly aligns with the granularity of search results within the knowledge base, thus affecting whether the large model's understanding of the input is biased, ultimately impacting the user experience. In existing technologies, text is typically segmented using hard rules such as fixed length, punctuation, and document structure. However, the resulting segments are often quite long, leading to information redundancy within each segment and resulting in low text retrieval accuracy. Summary of the Invention

[0004] In view of this, embodiments of this application provide a text segmentation method, apparatus, storage medium, and computer device, the main purpose of which is to solve the technical problem of low text retrieval accuracy.

[0005] According to one aspect of this application, a text segmentation method is provided, the method comprising: obtaining initial text and segmenting the initial text to obtain multiple target text segments; dividing the target text using identifiers to obtain multiple initial statements; concatenating the multiple initial statements to obtain multiple target statements, and determining a semantic threshold for the target statements based on the multiple target statements, wherein the semantic threshold is used to characterize the degree of semantic similarity between different target statements; and segmenting the target text according to a text segmentation strategy based on the semantic threshold to obtain multiple target blocks, wherein the text segmentation strategy is used to characterize the segmentation rules of the target text.

[0006] Optionally, determining the semantic threshold of a target statement based on multiple target statements includes: transforming multiple target statements using an embedding model to obtain multiple initial vectors, wherein the embedding model is constructed through a neural network model; determining multiple similarities using a cosine distance function based on the multiple initial vectors; sorting the multiple similarities to obtain a sorting result, and determining the semantic threshold of the target statement based on the sorting result.

[0007] Optionally, based on a semantic threshold, the target text is segmented according to a text segmentation strategy to obtain multiple target segments, including: segmenting the target text using a semantic threshold to obtain multiple initial segments; and updating the multiple initial segments based on the semantic threshold to obtain multiple target segments.

[0008] Optionally, based on a semantic threshold, multiple initial blocks are updated to obtain multiple target blocks, including: obtaining the block length of the initial blocks; determining the function threshold of the initial blocks based on the block length and the blocking function of the initial blocks; and updating multiple initial blocks based on the function threshold and the semantic threshold to obtain multiple target blocks.

[0009] Optionally, multiple initial blocks are updated based on a function threshold and a semantic threshold to obtain multiple target blocks, including: determining a dynamic threshold for the initial blocks based on a function threshold and a semantic threshold, wherein the dynamic threshold is the maximum value between the function threshold and the semantic threshold; and updating multiple initial blocks based on the dynamic threshold to obtain multiple target blocks.

[0010] Optionally, based on a dynamic threshold, multiple initial blocks are updated to obtain multiple target blocks, including: comparing multiple similarities corresponding to multiple target statements with the dynamic threshold to obtain a first comparison result; and based on the first comparison result, multiple initial blocks are updated to obtain multiple target blocks.

[0011] Optionally, before obtaining the initial text, the method further includes: obtaining the text to be segmented and filtering the text to be segmented to obtain the initial text, wherein the text data quality of the initial text is higher than that of the text data of the text to be segmented.

[0012] According to another aspect of this application, a text segmentation apparatus is provided, comprising: a first acquisition unit for acquiring initial text and segmenting the initial text to obtain multiple target text segments; a segmentation unit for segmenting the target text using identifiers to obtain multiple initial statements; a determination unit for concatenating the multiple initial statements to obtain multiple target statements, and determining a semantic threshold of the target statements based on the multiple target statements, wherein the semantic threshold is used to characterize the degree of semantic similarity between different target statements; and a second acquisition unit for segmenting the target text according to a text segmentation strategy based on the semantic threshold to obtain multiple target blocks, wherein the text segmentation strategy is used to characterize the segmentation rules of the target text.

[0013] According to another aspect of this application, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described text segmentation method.

[0014] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described text segmentation method.

[0015] By employing the above technical solutions, embodiments of this application provide a text segmentation method, apparatus, storage medium, and computer device. The method first obtains initial text, segments the initial text to obtain multiple target text segments, uses identifiers to divide the target text into multiple initial statements, then concatenates these initial statements to obtain multiple concatenated target statements, and then determines a semantic threshold based on these multiple target statements to characterize the semantic similarity between different target statements. Finally, based on the obtained semantic threshold, the target text is segmented according to the segmentation rules of the target text to achieve the purpose of obtaining multiple target segments. Considering that the text segmentation strategy based on the obtained semantic threshold can effectively segment the target text, reduce information redundancy within segments, shorten segment length, solve the technical problem of low text retrieval accuracy, and achieve the technical effect of improving text retrieval accuracy.

[0016] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a text segmentation method provided in an embodiment of this application is shown. Figure 2 A schematic diagram of a dynamic threshold curve provided in an embodiment of this application is shown; Figure 3 The illustration shows a flowchart of another text segmentation method provided in an embodiment of this application; Figure 4 A flowchart illustrating another text segmentation method provided in an embodiment of this application is shown; Figure 5 This illustration shows a structural schematic diagram of a text segmentation device provided in an embodiment of this application; Figure 6 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0018] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0019] In this embodiment, Figure 1 The following is a flowchart illustrating a text segmentation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: Step S101: Obtain the initial text and segment it to obtain multiple target text segments.

[0020] In the technical solution provided by step S101 of this application, an initial text can be obtained first, and then the initial text can be segmented to obtain multiple target texts. The initial text can be a whole text downloaded from the Internet.

[0021] For example, first obtain the entire text, and then segment the text according to its structure to obtain multiple segments, i.e., the target text, which facilitates subsequent processing steps.

[0022] Step S102: Use identifiers to segment the target text to obtain multiple initial statements.

[0023] In the technical solution provided by step S102 of this application, the target text obtained in the above steps can be divided using an identifier to obtain multiple initial statements. The identifier can be a delimiter, and the target text can be referred to as intra-segment text.

[0024] For example, the text within a paragraph can be divided into individual sentences based on delimiters, meaning the text within the paragraph is ultimately divided into 101 sentences. It should be noted that this is merely an example of dividing the target text into multiple initial sentences, and does not specify any particular method for dividing the target text into multiple initial sentences.

[0025] Step S103: Concatenate multiple initial statements to obtain multiple target statements, and determine the semantic threshold of the target statements based on the multiple target statements. The semantic threshold is used to characterize the degree of semantic similarity between different target statements.

[0026] In the technical solution provided in step S103 of this application, after obtaining multiple initial statements, the multiple initial statements are concatenated to obtain multiple target statements. Based on the multiple target statements, the semantic threshold of the target statements can be determined. The semantic threshold can be a preset threshold, which can be called a static semantic threshold or a semantic segmentation threshold.

[0027] Optionally, multiple initial statements can be concatenated, that is, context can be added to each statement. The resulting multiple target statements are statements with context. For example, after dividing the text within a paragraph into multiple individual sentences, context can be added to each sentence. That is, "sentence 1" becomes "sentence 1 sentence 2", "sentence 2" becomes "sentence 1 sentence 2 sentence 3", and so on.

[0028] Step S104: Based on the semantic threshold, the target text is segmented according to the text segmentation strategy to obtain multiple target segments, wherein the text segmentation strategy is used to characterize the segmentation rules of the target text.

[0029] In the technical solution provided in step S104 of this application, based on the obtained semantic threshold, the target text can be segmented according to the text segmentation strategy to obtain multiple target segments, thereby achieving the purpose of effectively segmenting the target text, reducing information redundancy within the segment, and shortening the segment length.

[0030] For example, based on a preset semantic threshold, the text within a segment is segmented according to a text segmentation strategy, dividing 101 sentences into at least 95 segments to achieve the final target segmentation.

[0031] It should be noted that this is only a preferred implementation method for dividing the target text into blocks according to the text segmentation strategy based on semantic thresholds to obtain multiple target blocks. The specific methods and processes for obtaining multiple target blocks based on semantic thresholds and the text segmentation strategy are not limited.

[0032] In the technical solution provided by steps S101 to S104 of this application, initial text can be obtained first, and then segmented to obtain multiple target text segments. Identifiers are used to divide the target text, resulting in multiple initial statements. These initial statements are then concatenated to obtain multiple concatenated target statements. Based on these target statements, a semantic threshold is determined to characterize the semantic similarity between different target statements. Finally, based on the obtained semantic threshold, the target text is segmented according to the segmentation rules to obtain multiple target segments. Considering that the obtained semantic threshold allows for effective segmentation of the target text using a text segmentation strategy, reducing information redundancy within segments, shortening segment length, and solving the technical problem of low text retrieval accuracy, this approach achieves the technical effect of improving text retrieval accuracy.

[0033] The method described in this embodiment will be further described below.

[0034] As an optional implementation method, determining the semantic threshold of a target statement based on multiple target statements includes: using an embedding model to transform multiple target statements to obtain multiple initial vectors, wherein the embedding model is constructed through a neural network model; using a cosine distance function to determine multiple similarities based on the multiple initial vectors; sorting the multiple similarities to obtain a sorting result, and determining the semantic threshold of the target statement based on the sorting result.

[0035] In this embodiment, multiple target statements are first transformed using an embedding model to obtain multiple initial vectors. Then, based on the multiple initial vectors obtained above, multiple similarities are determined using a cosine distance function. Finally, the multiple similarities are sorted to obtain a sorting result. Based on the sorting result, the semantic threshold of the target statement is determined.

[0036] Optionally, the embedding model can be called an encoder model (Bi-encoder model). The target statement can be called a combined statement, or simply a combined sentence. The cosine distance function is expressed using the cosine distance formula. The similarity can be called the similarity value.

[0037] Optionally, multiple similarities can be sorted in ascending order to obtain a ranking result. Based on the ranking result, the 95th similarity value is selected as the static semantic threshold. It should be noted that this is only a preferred implementation method for determining the semantic threshold based on the ranking result, and the process and method of determining the semantic threshold based on the ranking result are not specifically limited.

[0038] For example, first divide the paragraph content into sections according to the delimiters. The sentences are then processed by connecting the preceding and following content to add contextual information to each sentence. An embedding model is then used to convert these contextually linked sentences into vectors. The cosine distance formula is used to record the distance between adjacent sentence vectors. The similarity values ​​of the group will be sorted in ascending order, and the first similarity value in the group will be selected. The value is used as a semantic segmentation threshold.

[0039] As an optional implementation method, the target text is segmented according to a text segmentation strategy based on a semantic threshold to obtain multiple target segments, including: segmenting the target text using a semantic threshold to obtain multiple initial segments; and updating the multiple initial segments based on the semantic threshold to obtain multiple target segments.

[0040] In this embodiment, based on the semantic threshold obtained in the above steps, the target text can be segmented to obtain multiple initial segments. Then, based on the semantic threshold, the multiple initial segments are updated to obtain multiple target segments.

[0041] For example, after ranking multiple similarities, the first one... Using a similarity score as the semantic segmentation threshold, the number of blocks into which the entire paragraph content is segmented is then calculated. This means there are multiple initial blocks. Because the above method will cause... If the similarity of the blocks is less than the semantic segmentation threshold, the block content may become too long. Therefore, it is necessary to update multiple initial blocks according to the aforementioned semantic segmentation threshold in order to obtain multiple final target blocks.

[0042] Optionally, before segmenting the target text using semantic thresholds to obtain multiple initial segments, it is necessary to determine the conditional probability of the next sentence belonging to the current segment, i.e., it needs to satisfy... ,in, Used to characterize the first in the text content sentence, Used to indicate from arrive The blocks.

[0043] As an optional implementation method, multiple initial blocks are updated based on semantic thresholds to obtain multiple target blocks, including: obtaining the block length of the initial blocks; determining the function threshold of the initial blocks based on the block length and the blocking function of the initial blocks; and updating multiple initial blocks based on the function threshold and semantic thresholds to obtain multiple target blocks.

[0044] In this embodiment, the block length of the initial block can be obtained, and then the function threshold can be determined based on the block length and the blocking function of the initial block. Then, based on the function threshold and the semantic threshold, multiple target blocks can be obtained.

[0045] Optionally, the initial block can be called the current block, and the block length can be determined by... To represent, and The blocking function can dynamically change the connection threshold of the next sentence based on the block length. It can be a monotonically increasing function, which can be expressed by the following formula:

[0046] in, The rate of increase of the entire function is denoted as . The larger the value, the higher the threshold. The obtained block length is input into the above blocking function to obtain the corresponding function value, and this function value is determined as the function threshold.

[0047] As an optional implementation, multiple initial blocks are updated based on a function threshold and a semantic threshold to obtain multiple target blocks. This includes: determining a dynamic threshold for the initial blocks based on the function threshold and the semantic threshold, wherein the dynamic threshold is the maximum value between the function threshold and the semantic threshold; and updating multiple initial blocks based on the dynamic threshold to obtain multiple target blocks.

[0048] In this embodiment, based on the obtained function threshold and semantic threshold, a dynamic threshold for the initial block can be determined. Then, based on this dynamic threshold, multiple initial blocks can be updated to obtain multiple target blocks. The semantic threshold can be represented by t. The dynamic threshold can be represented by the following formula:

[0049] For example, Figure 2 This application provides a schematic diagram of a dynamic threshold curve, as illustrated in an embodiment. Figure 2 As shown, semantic segmentation is used when the number of sentences within a block is small. As a semantic threshold, the length of the segments is also incorporated into subsequent segmentation thresholds as segments accumulate. This involves taking into account various factors to achieve the goal of dynamically adjusting the segmentation threshold, blocking excessively long sentences within a segment, shortening the granularity of text segments, and thus improving the accuracy of subsequent text retrieval.

[0050] It should be noted that this is only a preferred embodiment for determining the dynamic threshold of the initial block, and the process and method for determining the dynamic threshold of the initial block are not specifically limited. As long as the process and method for determining the dynamic threshold of the initial block are based on the function threshold and the semantic threshold, they are all within the protection scope of this application, and will not be listed here.

[0051] As an optional implementation method, multiple initial blocks are updated based on a dynamic threshold to obtain multiple target blocks, including: comparing multiple similarities corresponding to multiple target statements with the dynamic threshold to obtain a first comparison result; and updating multiple initial blocks based on the first comparison result to obtain multiple target blocks.

[0052] In this embodiment, multiple similarities corresponding to multiple target statements can be obtained according to the above steps. The multiple similarities corresponding to multiple target statements are compared with dynamic thresholds to obtain a first comparison result. Then, based on the first comparison result, multiple initial blocks are updated to obtain multiple target blocks.

[0053] Optionally, in response to a similarity greater than a dynamic threshold, the target statement is merged with the previous block; in response to a similarity less than or equal to the dynamic threshold, a new block is created for the target statement.

[0054] For example, the similarity of each sentence is compared sequentially to see if it meets a dynamic threshold. If it does, the sentences are merged; otherwise, a new chunk is created.

[0055] As an optional embodiment, before obtaining the initial text, the method further includes: obtaining the text to be segmented and filtering the text to be segmented to obtain the initial text, wherein the text data quality of the initial text is higher than the text data quality of the text to be segmented.

[0056] In this embodiment, the text to be segmented is obtained and filtered to obtain the initial text after filtering. The filtering process includes replacing or deleting erroneous symbols in the text to be segmented.

[0057] For example, after obtaining the entire text, you can perform character filtering, replace or delete some incorrect symbols, remove some messy characters, and divide the text into paragraphs according to the chapter format of the recursive text.

[0058] It should be noted that this is only a preferred implementation method for filtering the text to be segmented to obtain the initial text, and does not specifically limit the filtering method of the text to be segmented.

[0059] In this embodiment, initial text is first obtained and then segmented to obtain multiple target text segments. Identifiers are used to further divide the target text into multiple initial statements. These initial statements are then concatenated to obtain multiple concatenated target statements. Based on these target statements, a semantic threshold is determined to characterize the semantic similarity between the different target statements. Finally, based on the obtained semantic threshold, the target text is segmented according to the target text segmentation rules to obtain multiple target blocks. Since the text segmentation strategy based on the obtained semantic threshold can effectively segment the target text, reduce information redundancy within blocks, shorten block length, and solve the technical problem of low text retrieval accuracy, it achieves the technical effect of improving text retrieval accuracy.

[0060] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, in order to fully explain the specific implementation process of this embodiment, Figure 3 This application provides a flowchart illustrating another text segmentation method according to an embodiment, such as... Figure 3 As shown, the method includes: Step S301: Filter the entire text.

[0061] In this embodiment, character filtering can be performed on the entire text, such as replacing or deleting some incorrect symbols in the entire text.

[0062] Step S302: Divide the entire text into segments according to its structure.

[0063] Step S303: Divide the text within the segment into individual sentences based on the delimiters.

[0064] In this embodiment, the text within a segment can be divided into separate sentences based on delimiters, for example, 101 sentences.

[0065] Step S304: Concatenate the context for each sentence.

[0066] In this embodiment, context can be spliced ​​together for each sentence, that is, "sentence 1" is spliced ​​together to become "sentence 1 sentence 2", "sentence 2" is spliced ​​together to become "sentence 1 sentence 2 sentence 3", and so on.

[0067] Step S305: Calculate the similarity between adjacent sentences.

[0068] In this embodiment, the similarity between two adjacent sentences after the previous concatenation is calculated sequentially, and a total of 100 similarity values ​​can be obtained.

[0069] Step S306: Sort the multiple similarities to determine the static semantic threshold.

[0070] In this embodiment, the similarity groups obtained in the previous step are sorted in ascending order. If the current environment has strong retrieval capabilities, the existence of some fragments is allowed to meet the effect of simplified retrieval. The 95th similarity can be selected as the static semantic threshold.

[0071] Step S307: Determine the function threshold.

[0072] In this embodiment, the blocking function parameters are configured, with the default increment rate set to 1. The dynamic threshold for the next sentence is dynamically obtained based on the number of sentences in the block, i.e., as shown below. Figure 2 As shown, in the early stage of block combination, a static semantic threshold is used. Since the increase in block length is not conducive to the model's understanding of the text, the dynamic threshold will actively cut off the low information density sentences after the block and compress the granularity.

[0073] Step S308: Determine whether the similarity of the current statement is greater than the dynamic threshold.

[0074] In this embodiment, it is necessary to determine whether the similarity of the current statement is greater than the dynamic threshold. If the similarity of the current statement is greater than the dynamic threshold, then step S309 is executed. If the similarity of the current statement is less than or equal to the dynamic threshold, then step S310 is executed.

[0075] Step S309: Merge sentences.

[0076] Step S310: Start a new block.

[0077] In this embodiment, furthermore, Figure 4 This illustration shows a flowchart of another text segmentation method provided in an embodiment of this application, namely as follows: Figure 4 As shown, firstly, the text is divided into a number of segments according to the delimiters. The sentences are categorized into independent sentence 1, ..., independent sentence n-2, independent sentence n-1, independent sentence n, independent sentence n+1, etc. Contextual information can then be added to each individual sentence. An embedding model is then used to convert these combined sentences with contextual information into vectors. The cosine distance formula is used to record the distance between adjacent sentence vectors. The similarity value is used to determine whether the similarity of sentence n meets the dynamic threshold. If it does, sentence n is added to the original block and merged with other sentences. If it does not meet the threshold, a new block is created.

[0078] Step S311: Complete the segmentation of the current segment.

[0079] In this embodiment, the text segmentation method described above can divide 101 sentences into at least 95 segments.

[0080] In this embodiment, initial text is first obtained, then segmented to obtain multiple target text segments. Identifiers are used to further divide the target text into multiple initial statements. These initial statements are then concatenated to obtain multiple concatenated target statements. A semantic threshold is then determined based on these target statements to characterize the semantic similarity between them. Finally, based on the obtained semantic threshold, the target text is segmented according to the segmentation rules to obtain multiple target segments. In other words, in this embodiment, by incorporating a regularized blocking function and dynamically increasing the merging threshold as the segment length increases, the segment length is shortened, thus solving the technical problem of low text retrieval accuracy and improving the overall text retrieval accuracy.

[0081] Furthermore, as Figure 1 The specific implementation of the method, Figure 5 This paper shows a schematic diagram of the structure of a text segmentation device provided in an embodiment of this application, as shown below. Figure 5 As shown, the text segmentation device 500 includes: a first acquisition unit 501, a segmentation unit 502, a determination unit 503, and a second acquisition unit 504.

[0082] The first acquisition unit 501 is used to acquire the initial text and segment the initial text to obtain multiple target text segments.

[0083] The segmentation unit 502 is used to segment the target text using identifiers to obtain multiple initial statements.

[0084] The determining unit 503 is used to concatenate multiple initial statements to obtain multiple target statements, and based on the multiple target statements, to determine the semantic threshold of the target statements, wherein the semantic threshold is used to characterize the degree of semantic similarity between different target statements.

[0085] The second acquisition unit 504 is used to segment the target text into blocks based on a semantic threshold and a text segmentation strategy to obtain multiple target blocks, wherein the text segmentation strategy is used to characterize the segmentation rules of the target text.

[0086] Optionally, the device is also used to: transform multiple target sentences using an embedding model to obtain multiple initial vectors, wherein the embedding model is constructed through a neural network model; determine multiple similarities using a cosine distance function based on the multiple initial vectors; sort the multiple similarities to obtain a sorting result, and determine the semantic threshold of the target sentences based on the sorting result.

[0087] Optionally, the device is also used to: segment the target text using a semantic threshold to obtain multiple initial segments; and update the multiple initial segments based on the semantic threshold to obtain multiple target segments.

[0088] Optionally, the device is also used to: obtain the block length of the initial block; determine the function threshold of the initial block based on the block length and the blocking function of the initial block; and update multiple initial blocks based on the function threshold and the semantic threshold to obtain multiple target blocks.

[0089] Optionally, the device is also used to: determine a dynamic threshold for the initial block based on a function threshold and a semantic threshold, wherein the dynamic threshold is the maximum value between the function threshold and the semantic threshold; and update multiple initial blocks based on the dynamic threshold to obtain multiple target blocks.

[0090] Optionally, the device is further configured to: compare multiple similarities corresponding to multiple target statements with dynamic thresholds respectively to obtain a first comparison result; and update multiple initial blocks based on the first comparison result to obtain multiple target blocks.

[0091] Optionally, the device is also used to: acquire the text to be segmented, and filter the text to be segmented to obtain the initial text, wherein the text data quality of the initial text is higher than the text data quality of the text to be segmented.

[0092] In this embodiment, an initial text is acquired by a first acquisition unit and segmented to obtain multiple target text segments. A segmentation unit then segments the target text using identifiers to obtain multiple initial statements. A determination unit concatenates the multiple initial statements to obtain multiple target statements, and based on these target statements, a semantic threshold is determined, where the semantic threshold characterizes the semantic similarity between different target statements. A second acquisition unit then segments the target text according to a text segmentation strategy based on the semantic threshold to obtain multiple target blocks. This text segmentation strategy characterizes the segmentation rules of the target text, thereby solving the technical problem of low text retrieval accuracy and achieving the technical effect of improving text retrieval accuracy.

[0093] It should be noted that other corresponding descriptions of the functional units involved in the text segmentation device provided in this application embodiment can be found by referring to... Figure 1 The corresponding descriptions in the method will not be repeated here.

[0094] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 6 This application provides a schematic diagram of the device structure of a computer device according to an embodiment of the present application. Figure 6 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0095] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0096] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0097] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0098] It should be noted that the user personal information involved in the embodiments of this application is all authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals. It should be noted that if any software tools or components other than those of this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.

[0099] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A text segmentation method, characterized in that, The method includes: Obtain the initial text and segment it to obtain multiple target text segments; The target text is segmented using identifiers to obtain multiple initial statements; Multiple initial statements are concatenated to obtain multiple target statements, and a semantic threshold for each target statement is determined based on the multiple target statements, wherein the semantic threshold is used to characterize the degree of semantic similarity between different target statements; Based on the semantic threshold, the target text is segmented according to the text segmentation strategy to obtain multiple target segments, wherein the text segmentation strategy is used to characterize the segmentation rules of the target text.

2. The method according to claim 1, characterized in that, Based on multiple target statements, the semantic threshold of the target statements is determined, including: Multiple target statements are transformed using an embedding model to obtain multiple initial vectors, wherein the embedding model is constructed using a neural network model; Based on multiple initial vectors, multiple similarities are determined using the cosine distance function; The similarity scores are sorted to obtain a sorting result, and the semantic threshold of the target statement is determined based on the sorting result.

3. The method according to claim 1, characterized in that, Based on the semantic threshold, the target text is segmented according to a text segmentation strategy to obtain multiple target segments, including: The target text is segmented using the semantic threshold to obtain multiple initial segments; Based on the semantic threshold, the initial blocks are updated to obtain the target blocks.

4. The method according to claim 3, characterized in that, Based on the semantic threshold, multiple initial blocks are updated to obtain multiple target blocks, including: Obtain the block length of the initial block; Based on the block length and the blocking function of the initial block, determine the function threshold of the initial block; Based on the function threshold and the semantic threshold, the initial blocks are updated to obtain the target blocks.

5. The method according to claim 4, characterized in that, Based on the function threshold and the semantic threshold, multiple initial blocks are updated to obtain multiple target blocks, including: Based on the function threshold and the semantic threshold, a dynamic threshold for the initial block is determined, wherein the dynamic threshold is the maximum value between the function threshold and the semantic threshold; Based on the dynamic threshold, the initial blocks are updated to obtain the target blocks.

6. The method according to claim 5, characterized in that, Based on the dynamic threshold, multiple initial blocks are updated to obtain multiple target blocks, including: The similarity scores corresponding to the multiple target statements are compared with the dynamic threshold to obtain a first comparison result; Based on the first comparison result, the initial blocks are updated to obtain the target blocks.

7. The method according to claim 1, characterized in that, Before obtaining the initial text, the method further includes: Obtain the text to be segmented and filter it to obtain the initial text, wherein the text data quality of the initial text is higher than that of the text to be segmented.

8. A text segmentation device, characterized in that, The device includes: The first acquisition unit is used to acquire initial text and segment the initial text to obtain multiple target text segments; A segmentation unit is used to segment the target text using identifiers to obtain multiple initial statements; A determining unit is configured to concatenate multiple initial statements to obtain multiple target statements, and based on the multiple target statements, determine a semantic threshold of the target statements, wherein the semantic threshold is used to characterize the degree of semantic similarity between different target statements; The second acquisition unit is used to perform block processing on the target text according to the semantic threshold and the text block strategy to obtain multiple target blocks, wherein the text block strategy is used to characterize the block rules of the target text.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images