Text segmentation method and device, storage medium and electronic equipment

Abstract

The invention discloses a text segmentation method and device, a storage medium and electronic equipment. The method relates to the technical field of artificial intelligence and the field of medical health. Based on the text type of the to-be-segmented text, determining an initial slicing mode for performing a segmentation operation on the to-be-segmented text; performing data segmentation processing on the to-be-segmented text by adopting an initial slicing mode to obtain each initial data block corresponding to the to-be-segmented text; and performing data segmentation processing on each initial data block by at least adopting a method of dynamically adjusting the length of the segmentation block to obtain each target data block corresponding to the to-be-segmented text. For different text types, the method for dynamically adjusting the length of the segmentation block or the method for dynamically adjusting the length of the segmentation block and the method for dynamically adjusting the length of the repeated block are adopted to perform data segmentation processing on each initial data block, so that the content of each block of the original initial data block can be ensured to be complete; and the dicing result is not easy to ambiguate.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and medical health, and in particular to a text segmentation method, device, storage medium and electronic device. Background Art

[0002] Retrieval-augmented generation (RAG) is one of the applications of big models and the main way to implement big models. The basic paradigm of traditional RAG is that the files uploaded by users are parsed, then divided into blocks and stored in the vector database. When a user asks a question, the question is matched with the text in the vector database to obtain the relevant context, and then the context is combined with the user's question to form a prompt, which is input to the big model. The big model generates a response based on the relevant context and question. In this process, when the capabilities of the big models are the same, the quality of the context completely determines the accuracy of the big model's response, so block segmentation is the top priority of the entire RAG link. The method of segmenting text can support functions such as disease auxiliary diagnosis, health management, and remote consultation. The existing hard-cutting method mainly hard-cuts the text to be segmented given a predetermined segmentation size and a predetermined repetition block size, but this violent cutting method easily destroys the coherence of the sentence and causes ambiguity. Summary of the invention

[0003] In view of this, the present invention provides a text segmentation method, device, storage medium and electronic device, the main purpose of which is to solve the problem that the current traditional method of text segmentation easily causes sentence coherence to be destroyed and ambiguity to occur.

[0004] To solve the above problems, the present application provides a text segmentation method, comprising:

[0005] Get the text to be segmented;

[0006] Determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0007] Using the initial segmentation method to perform data segmentation processing on the text to be segmented, and obtaining initial data blocks corresponding to the text to be segmented;

[0008] At least one method of dynamically adjusting the length of the segmented blocks is used to perform data segmentation processing on each of the initial data blocks to obtain each target data block corresponding to the text to be segmented.

[0009] Optionally, the determining of an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented specifically includes:

[0010] When the text type of the to-be-segmented text is the first text type, a segmentation method using a preset paragraph extraction method and a preset identifier is determined as an initial segmentation method;

[0011] When the text type of the to-be-segmented text is the second text type, determining the line-by-line segmentation mode as the initial segmentation mode;

[0012] The first text type includes word type and txt type;

[0013] The second text type includes an excel type and a csv type.

[0014] Optionally, the adopting the initial segmentation method to perform data segmentation processing on the text to be segmented to obtain initial data blocks corresponding to the text to be segmented specifically includes:

[0015] When the text type of the text to be segmented is the first text type, identifying identifiers in the text to be segmented to obtain first identifiers and second identifiers;

[0016] The text to be segmented is segmented based on each of the first identifiers and each of the second identifiers to obtain each of the initial data blocks.

[0017] Optionally, the step of performing data segmentation processing on the text to be segmented by using the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented further includes:

[0018] When the text type of the to-be-segmented text is the second text type, identifying line break identifiers in the cell to obtain a plurality of line break identifiers;

[0019] Performing symbol conversion on each of the line break identifiers of the text to be segmented to obtain a first text;

[0020] The line header and each line of the first text are segmented to obtain each of the initial data blocks.

[0021] Optionally, the method of at least dynamically adjusting the length of the segmented blocks to perform data segmentation processing on each of the initial data blocks to obtain each target data block corresponding to the text to be segmented specifically includes:

[0022] For the first target data block, determine the first repeated block length and the first segmented block length;

[0023] Extract data from the first initial data block based on the first repeated block length and the first segmented block length to obtain a first target data block;

[0024] For a non-first target data block, dynamically adjusting the segmentation block length of the previous target data block and the repeated block length of the previous target data block based on the data in the previous target data block to obtain a second repeated block length and a second segmentation block length;

[0025] The second repeated block length and the second segmented block length are used to extract data from the remaining initial data blocks to obtain the non-first target data block.

[0026] Optionally, the method further includes:

[0027] When the length of the target initial data block is greater than the length of the first segmentation block or the length of the target initial data block is greater than the length of the second segmentation block, determining the target initial data block as the target data block;

[0028] When the length of the target initial data block is greater than the first repeated block length and the length of the target initial data block is less than or equal to the first segmented block length, determining the target initial data block as the target data block;

[0029] When the length of the target initial data block is greater than the second repeated block length and the length of the target initial data block is less than or equal to the second segmentation block length, the target initial data block is determined as the target data block.

[0030] Optionally, the method of at least dynamically adjusting the length of the segmented blocks to perform data segmentation processing on each of the initial data blocks to obtain each target data block corresponding to the text to be segmented also includes:

[0031] Identifying a row header data block in each of the initial data blocks to obtain a row header length;

[0032] Performing a subtraction operation based on the predetermined segmentation block length and the line header length to obtain a third segmentation block length;

[0033] Based on the third segmentation length, data segmentation processing is performed by adopting the method of patrolling the row header data block and the row data blocks in each of the initial data blocks to obtain each of the target data blocks.

[0034] In order to solve the above problems, the present application provides a text segmentation device, comprising:

[0035] The acquisition module is used to obtain the text to be segmented;

[0036] A determination module, used for determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0037] The first segmentation module is used to perform data segmentation processing on the text to be segmented by adopting the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented;

[0038] The second segmentation module is used to perform data segmentation processing on each of the initial data blocks by at least adopting a method of dynamically adjusting the segmentation block length to obtain each target data block corresponding to the text to be segmented.

[0039] In order to solve the above problem, the present application provides a storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above text segmentation method are implemented.

[0040] In order to solve the above problems, the present application provides an electronic device, characterized in that it includes at least a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above-mentioned text segmentation method when executing the computer program on the memory.

[0041] The beneficial effects of the present application are as follows: the present application adopts a method of dynamically adjusting the length of the segmented block or a method of dynamically adjusting the length of the segmented block and the length of the repeated block to perform data segmentation processing on each of the initial data blocks for different text types, and obtains each target data block corresponding to the text to be segmented, so that the original initial data block can ensure that each block has complete content, and the segmentation result is less likely to be ambiguous.

[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0044] Figure 1 This is a schematic diagram of an application environment of a text segmentation method in an embodiment of the present application;

[0045] Figure 2 A flowchart of a text segmentation method provided in an embodiment of the present application is shown;

[0046] Figure 3 A flowchart of a text segmentation method provided by another embodiment of the present application is shown;

[0047] Figure 4 A structural block diagram of a text segmentation device provided in another embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0049] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0050] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0051] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0052] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0053] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0054] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0055] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0056] The text segmentation method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the client communicates with the server through a network. The server can obtain the text to be segmented through the client, and determine the initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented; the text to be segmented is processed by the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented; at least the method of dynamically adjusting the segmentation block length is used to perform data segmentation on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented, and each target data block is fed back to the client. In the present application, the method of dynamically adjusting the segmentation block length is used to perform data segmentation on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented, so that the original segmentation can ensure that each block is complete content, and the segmentation result is not easy to be ambiguous.

[0057] Another embodiment of the present application provides a text segmentation method, such as Figure 2 As shown, including:

[0058] Step S101: obtaining the text to be segmented;

[0059] In the specific implementation process of this step, the type of the text to be segmented includes a first text type and a second text type; the first text type includes text types such as word type and txt type; the second text type includes texts of table types such as excel type and csv type. The text segmentation method of the present application can be applied to a medical cloud platform. Medical cloud refers to the use of "cloud computing" to create a medical and health service cloud platform based on new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the combination of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. For example, the current hospital appointment registration, electronic medical records, medical insurance, etc. are all products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout.

[0060] Step S102: determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0061] During the specific implementation of this step, when the text type of the text to be segmented is the first text type, the segmentation method using a preset paragraph extraction method and a preset identifier is determined as the initial segmentation method; when the text type of the text to be segmented is the second text type, the line-by-line segmentation method is determined as the initial segmentation method.

[0062] Step S103: performing data segmentation processing on the text to be segmented by using the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented;

[0063] In the specific implementation process of this step, when the text type of the text to be segmented is the first text type, the identifiers in the text to be segmented are identified to obtain each first identifier and each second identifier; the first identifier is a sentence end identifier, and the sentence end identifier can be a period, exclamation mark, question mark and other symbol identifiers; the second identifier can be a paragraph end identifier, and the paragraph end identifier can be an identifier symbol such as the enter key, representing the end of the paragraph. The text to be segmented is segmented based on each of the first identifiers and each of the second identifiers to obtain each of the initial data blocks. When the text type of the text to be segmented is the second text type, the line break identifiers in the cell are identified to obtain a number of line break identifiers; the line break identifiers of the text to be segmented are symbolically converted to obtain the first text; this can be done using The tag performs symbol conversion on each line break identifier. The line header and each line branch of the first text are segmented to obtain each initial data block.

[0064] Step S104: at least using a method of dynamically adjusting the length of the segmented blocks to perform data segmentation processing on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented.

[0065] In the specific implementation process of this step, when the text type of the text to be segmented is the first text type, the method of dynamically adjusting the segmentation block length and the method of dynamically adjusting the repeated block size are used to perform data segmentation processing on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented; when the text type of the text to be segmented is the second text type, the method of dynamically adjusting the segmentation block length is used to perform data segmentation processing on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented.

[0066] The present application uses a method of dynamically adjusting the length of the segmented block or a method of dynamically adjusting the length of the segmented block and the length of the repeated block to perform data segmentation processing on each of the initial data blocks for different text types, so as to obtain target data blocks corresponding to the text to be segmented. This can ensure that each block of the original initial data block has complete content, and can make the segmentation result less ambiguous.

[0067] Another embodiment of the present application provides another text segmentation method, such as Figure 3 As shown, including:

[0068] Step S201: obtaining the text to be segmented;

[0069] In the specific implementation process of this step, the type of the text to be segmented includes a first text type and a second text type; the first text type includes text types such as word type and txt type; the second text type includes texts of table types such as excel type and csv type. The text segmentation method of the present application can be applied to a medical cloud platform. Medical cloud refers to the use of "cloud computing" to create a medical and health service cloud platform based on new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, and the Internet of Things, combined with medical technology, to achieve the sharing of medical resources and the expansion of medical scope. Because of the combination of cloud computing technology, the medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. For example, the current hospital appointment registration, electronic medical records, medical insurance, etc. are all products of the combination of cloud computing and the medical field. The medical cloud also has the advantages of data security, information sharing, dynamic expansion, and overall layout.

[0070] Step S202: determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0071] In the specific implementation process of this step, when the text type of the text to be segmented is the first text type, the segmentation operation using a preset paragraph extraction method and a preset identifier is determined as the initial segmentation method; when the text type of the text to be segmented is the second text type, the line-by-line segmentation method is determined as the initial segmentation method; the first text type includes word type and txt type; the second text type includes excel type and csv type.

[0072] Step S203: When the text type of the text to be segmented is the first text type, identifying identifiers in the text to be segmented to obtain first identifiers and second identifiers;

[0073] In the specific implementation of this step, the first identifier is a sentence end identifier, which can be a period, exclamation mark, question mark or other symbol identifier; the second identifier can be a paragraph end identifier, which can be an identifier such as a carriage return key, representing the end of a paragraph. The first text type is a word type or a txt type text, and the format of this type of text is relatively standardized. The first identifier and the second identifier can be used to perform basic segmentation on the text to be segmented, so that each sentence is a separate initial data block.

[0074] Step S204: segmenting the text to be segmented based on the first identifiers and the second identifiers to obtain the initial data blocks;

[0075] In the specific implementation process of this step, the text to be segmented is segmented based on each of the first identifiers and each of the second identifiers to obtain an initial data block sequence composed of each of the initial data blocks. This method can be used to perform initial segmentation on the text without destroying the complete sentences, so that each sentence is a separate initial data block.

[0076] Step S205: performing data segmentation processing on each of the initial data blocks by dynamically adjusting the segmentation block length and the repeated block size to obtain target data blocks corresponding to the text to be segmented;

[0077] In the specific implementation process of this step, for the first target data block, the first repeated block length and the first split block length are determined; specifically, the repeated block length can be initialized to obtain the first repeated block length, the first repeated block length can be 40, and the first split block length can be 150; in actual applications, the first repeated block length and the first split block length can be set according to actual needs. Based on the first repeated block length and the first split block length, data extraction is performed from the first initial data block to obtain the first target data block; specifically, the repeated block is initialized to obtain the first repeated block length, the first repeated block length is 0; based on the first split block length and the first repeated block length, the first available split block length is calculated; that is, the first available split block length is 150-0=150; based on the first available split block length, data is read from the first initial data block, and when the current data length read reaches the first available split block length, the read initial data blocks and the initial data block at the current reading position are divided into the first target data block to obtain the first target data block. For a non-first target data block, dynamically adjust the segmentation block length of the previous target data block and the repeated block length of the previous target data block based on the data in the previous target data block to obtain a second repeated block length and a second segmentation block length; calculate and process based on the length of each sentence data in the previous target data block and the first repeated block length to obtain the second repeated block length; use the second repeated block length and the second segmentation block length to extract data from the remaining initial data blocks to obtain the non-first target data block. Specifically, firstly, the length of the last sentence data in the previous target data block is compared with the first repeated block length. When the length of the last sentence data in the previous target data block is less than the first repeated block length, the sum of the lengths of the two last sentences in the previous target data block is calculated to obtain a first sum value. The first sum value is compared with the first repeated block length. When the first sum value is greater than or equal to the first repeated block length, the length of the last sentence data in the previous target data block is determined as the second repeated block length. If the first sum value is less than the first repeated block length, the sum of the lengths of the three last sentences in the previous target data block is calculated to obtain a second sum value, and so on until the second repeated block length is determined. Based on the second repeated block length and the first segmentation block length, calculation processing is performed to obtain the current available segmentation block length. Then, data in the previous target data block is extracted based on the second repeated block length, and data in the remaining initial data blocks are extracted based on the current available segmentation block length to obtain the current non-first target data block.When the length of the target initial data block is greater than the first segmentation block length or the length of the target initial data block is greater than the second segmentation block length, the target initial data block is determined as the target data block; when the length of the target initial data block is greater than the first repeated block length and the length of the target initial data block is less than or equal to the first segmentation block length, the target initial data block is determined as the target data block; when the length of the target initial data block is greater than the second repeated block length and the length of the target initial data block is less than or equal to the second segmentation block length, the target initial data block is determined as the target data block.

[0078] Step S206: when the text type of the text to be segmented is the second text type, identifying the line break identifier in the cell to obtain a plurality of line break identifiers;

[0079] In the specific implementation of this step, when the second text type is a table type such as Excel, the line break in the cell is usually represented by the characters \n or \r\n, and the line break identifier needs to be replaced to obtain a complete row of data.

[0080] Step S207: performing symbol conversion on each of the line break identifiers of the text to be segmented to obtain a first text;

[0081] In the specific implementation process of this step, a preset lightweight markup language can be used to perform symbol conversion on each line break identifier of the text to be segmented, and the line break identifier \n or \r\n can be converted into The operation converts the text to be segmented into a markdown format to obtain the first text.

[0082] Step S208: segmenting the line header and each line of the first text to obtain each of the initial data blocks;

[0083] In the specific implementation process of this step, the line headers of the first text and the line data corresponding to each branch are separately divided into initial data blocks to obtain each of the initial data blocks. Due to the conversion to markdown format, a large number of redundant characters will be generated, and the format will exist. If the hard cutting method is still used, the table structure will be destroyed. This application uses the method of line break to divide each line of table data, and the method of bringing the line header to divide the first text to obtain each target data block.

[0084] Step S209: Perform data segmentation processing on each of the initial data blocks by dynamically adjusting the segmentation block length to obtain target data blocks corresponding to the text to be segmented.

[0085] In the specific implementation process of this step, the row header data block in each of the initial data blocks is identified to obtain the row header length; the row header data block includes row header field data and a separator mark; the data length of the row header field data and the separator mark is calculated to obtain the row header length. Based on the predetermined segmentation block length and the row header length, a subtraction operation is performed to obtain a third segmentation block length; based on the third segmentation length, data segmentation is performed by using a polling row header data block and a row data block in each of the initial data blocks to obtain each of the target data blocks. Specifically, for the first target data block, first read the row header data block in the initial data block; then extract data from the branch data blocks in the first initial data block based on the third segmentation block length, until the third segmentation length is reached, determine whether the data of the last branch data block is complete; when the data of the last branch data block is complete, determine the row header data block and the branch data blocks currently read as the first target data block; when the data of the last branch data block is incomplete, discard the last branch data block, and determine the row header data block and the branch data blocks currently read after the last branch data block is discarded as the first target data block; for non-first target data blocks, first read the row header data block in the initial data block Header data block; then based on the third segmentation block length, data is extracted from the branch data blocks of the remaining initial data blocks until the third segmentation length is reached, and whether the data of the last branch data block currently read is complete. If the data of the last branch data block currently read is complete, the line header data block and the branch data blocks currently read are determined as non-first target data blocks; if the data of the last branch data block currently read is incomplete, the last branch data block currently read is discarded, and the line header data block and the branch data blocks currently read after the last branch data block is discarded are determined as non-first target data blocks; and so on to obtain the target data blocks corresponding to the text to be segmented. When segmenting the text of the second text type, each target data block includes the same line header data and different branch data. And the different branch data in each target data block are complete.

[0086] The present application obtains the text to be segmented; determines the initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented; when the text type of the text to be segmented is the first text type, identifies the identifiers in the text to be segmented to obtain each first identifier and each second identifier; segments the text to be segmented based on each first identifier and each second identifier to obtain each initial data block; performs data segmentation on each initial data block using the method of dynamically adjusting the segmentation block length and dynamically adjusting the size of the repeated block to obtain each target data block corresponding to the text to be segmented; performs initial segmentation on the text to be segmented using the first identifier that identifies the end of the sentence and the second identifier of the paragraph technology, thereby ensuring the integrity of the sentence. The data segmentation on each initial data block using the method of dynamically adjusting the segmentation block length and dynamically adjusting the size of the repeated block can ensure the integrity of the original initial segmentation data, and can make the segmentation result less ambiguous. When the text type of the text to be segmented is the second text type, the line break identifier in the cell is identified to obtain a plurality of line break identifiers; the line break identifiers of the text to be segmented are symbolically converted to obtain the first text; the line header and each branch of the first text are segmented to obtain each initial data block; the method of dynamically adjusting the segmentation block length is used to perform data segmentation on each initial data block to obtain each target data block corresponding to the text to be segmented. The method of the present application prevents the structure of the table from being destroyed due to hard cutting. The method of the present application improves the coherence of the context, ensures that the content in each segment is more semantically coherent, avoids the loss of context information due to hard cutting, and improves the quality of retrieval and generation. The improved optimized segmentation method of the present application makes the answers provided by the large language model more accurate and coherent, and improves user satisfaction and trust.

[0087] Another embodiment of the present application provides a text segmentation device, such as Figure 4 As shown, including:

[0088] Acquisition module 1, used to acquire the text to be segmented;

[0089] Determining module 2, used for determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0090] The first segmentation module 3 is used to perform data segmentation processing on the text to be segmented by adopting the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented;

[0091] The second segmentation module 4 is used to perform data segmentation processing on each of the initial data blocks by at least adopting a method of dynamically adjusting the segmentation block length to obtain each target data block corresponding to the text to be segmented.

[0092] In the specific implementation process, the determination module 2 is specifically used for: when the text type of the text to be segmented is the first text type, the method of using a preset paragraph extraction method and a preset identifier for segmentation operation is determined as the initial segmentation method; when the text type of the text to be segmented is the second text type, the line-by-line segmentation method is determined as the initial segmentation method; the first text type includes word type and txt type; the second text type includes excel type and csv type.

[0093] In the specific implementation process, the first segmentation module 3 is specifically used to: when the text type of the text to be segmented is the first text type, identify the identifiers in the text to be segmented to obtain each first identifier and each second identifier; based on each first identifier and each second identifier, segment the text to be segmented to obtain each initial data block.

[0094] In the specific implementation process, the first segmentation module 3 is also used for: when the text type of the text to be segmented is the second text type, identifying the line break identifier in the cell to obtain a plurality of line break identifiers; performing symbol conversion on each of the line break identifiers of the text to be segmented to obtain the first text; and segmenting the line header and each branch of the first text respectively to obtain each of the initial data blocks.

[0095] In the specific implementation process, the second segmentation module 4 is specifically used to: determine the first repeated block length and the first segmentation block length for the first target data block; extract data from the first initial data block based on the first repeated block length and the first segmentation block length to obtain the first target data block; for the non-first target data block, dynamically adjust the segmentation block length and the repeated block length of the previous target data block obtained by segmentation based on the data in the previous target data block to obtain the second repeated block length and the second segmentation block length; use the second repeated block length and the second segmentation block length to extract data from the remaining initial data blocks to obtain the non-first target data block.

[0096] In a specific implementation process, the second segmentation module 4 is further used to: when the length of the target initial data block is greater than the first segmentation block length or the length of the target initial data block is greater than the second segmentation block length, determine the target initial data block as the target data block; when the length of the target initial data block is greater than the first repeated block length and the length of the target initial data block is less than or equal to the first segmentation block length, determine the target initial data block as the target data block; when the length of the target initial data block is greater than the second repeated block length and the length of the target initial data block is less than or equal to the second segmentation block length, determine the target initial data block as the target data block.

[0097] In the specific implementation process, the second segmentation module 4 is also used to: identify the row header data block in each of the initial data blocks to obtain the row header length; perform subtraction processing based on the predetermined segmentation block length and the row header length to obtain the third segmentation block length; based on the third segmentation length, use the patrol row header data block and the row data block in each of the initial data blocks to perform data segmentation processing to obtain each of the target data blocks.

[0098] The present application uses a method of dynamically adjusting the length of the segmented block or a method of dynamically adjusting the length of the segmented block and the length of the repeated block to perform data segmentation processing on each of the initial data blocks for different text types, so as to obtain target data blocks corresponding to the text to be segmented. This can ensure that each block of the original initial data block has complete content, and can make the segmentation result less ambiguous.

[0099] Another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0100] Step 1: Get the text to be segmented;

[0101] Step 2: determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0102] Step 3: using the initial segmentation method to perform data segmentation processing on the text to be segmented, and obtaining initial data blocks corresponding to the text to be segmented;

[0103] Step 4: At least use a method of dynamically adjusting the length of the segmented blocks to perform data segmentation processing on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0105] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0106] The specific implementation process of the above method steps can be found in the embodiments of any of the above text segmentation methods, and this embodiment will not be repeated here.

[0107] Another embodiment of the present application provides an electronic device, which may be a server, and the electronic device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external client via a network connection. When the electronic device program is executed by the processor, it implements the functions or steps of a text segmentation method server side.

[0108] In one embodiment, an electronic device is provided, which may be a client. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external server via a network connection. When the electronic device program is executed by the processor, the functions or steps on the client side of a text segmentation method are implemented.

[0109] Another embodiment of the present application provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the following method steps when executing the computer program in the memory:

[0110] Step 1: Get the text to be segmented;

[0111] Step 2: determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented;

[0112] Step 3: using the initial segmentation method to perform data segmentation processing on the text to be segmented, and obtaining initial data blocks corresponding to the text to be segmented;

[0113] Step 4: At least use a method of dynamically adjusting the length of the segmented blocks to perform data segmentation processing on each of the initial data blocks to obtain target data blocks corresponding to the text to be segmented.

[0114] The specific implementation process of the above method steps can be found in the embodiments of any of the above text segmentation methods, and this embodiment will not be repeated here.

[0115] The present application uses a method for dynamically adjusting the length of the segmented blocks or a method for dynamically adjusting the length of the segmented blocks and the length of the repeated blocks to perform data segmentation processing on each of the initial data blocks for different text types, and obtains each target data block corresponding to the text to be segmented, so that the original initial data block can ensure that each block has complete content, and the segmentation result is not easily ambiguous. The data block is subjected to data segmentation processing to obtain each target data block corresponding to the text to be segmented.

[0116] The non-company software tools or components that appear in the embodiments of this application are merely examples and do not represent actual use. The above embodiments are only exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to this application within the essence and scope of protection of this application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of this application.

Claims

1. A text segmentation method, characterized in that: include: Get the text to be segmented; Determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented; Using the initial segmentation method to perform data segmentation processing on the text to be segmented, and obtaining initial data blocks corresponding to the text to be segmented; At least one method of dynamically adjusting the length of the segmented blocks is used to perform data segmentation processing on each of the initial data blocks to obtain each target data block corresponding to the text to be segmented.

2. The method according to claim 1, characterized in that The determining of an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented specifically includes: When the text type of the to-be-segmented text is the first text type, a segmentation method using a preset paragraph extraction method and a preset identifier is determined as an initial segmentation method; When the text type of the to-be-segmented text is the second text type, determining the line-by-line segmentation mode as the initial segmentation mode; The first text type includes word type and txt type; The second text type includes an excel type and a csv type.

3. The method according to claim 2, characterized in that The method of using the initial segmentation method to perform data segmentation processing on the text to be segmented to obtain initial data blocks corresponding to the text to be segmented specifically includes: When the text type of the text to be segmented is the first text type, identifying identifiers in the text to be segmented to obtain first identifiers and second identifiers; The text to be segmented is segmented based on each of the first identifiers and each of the second identifiers to obtain each of the initial data blocks.

4. The method according to claim 2, characterized in that The method of performing data segmentation processing on the text to be segmented by using the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented also includes: When the text type of the to-be-segmented text is the second text type, identifying line break identifiers in the cell to obtain a plurality of line break identifiers; Performing symbol conversion on each of the line break identifiers of the text to be segmented to obtain a first text; The line header and each line of the first text are segmented to obtain each of the initial data blocks.

5. The method according to claim 3, characterized in that The method of at least dynamically adjusting the length of the segmented blocks is used to perform data segmentation processing on each of the initial data blocks to obtain each target data block corresponding to the text to be segmented, specifically comprising: For the first target data block, determine the first repeated block length and the first segmented block length; Extract data from the first initial data block based on the first repeated block length and the first segmented block length to obtain a first target data block; For a non-first target data block, dynamically adjusting the segmentation block length of the previous target data block and the repeated block length of the previous target data block based on the data in the previous target data block to obtain a second repeated block length and a second segmentation block length; The second repeated block length and the second segmented block length are used to extract data from the remaining initial data blocks to obtain the non-first target data block.

6. The method according to claim 5, characterized in that The method further comprises: When the length of the target initial data block is greater than the length of the first segmentation block or the length of the target initial data block is greater than the length of the second segmentation block, determining the target initial data block as the target data block; When the length of the target initial data block is greater than the first repeated block length and the length of the target initial data block is less than or equal to the first segmentation block length, determining the target initial data block as the target data block; When the length of the target initial data block is greater than the second repeated block length and the length of the target initial data block is less than or equal to the second segmentation block length, the target initial data block is determined as the target data block.

7. The method according to claim 4, characterized in that The method of at least dynamically adjusting the length of the segmented blocks is used to perform data segmentation processing on each of the initial data blocks to obtain each target data block corresponding to the text to be segmented, and further includes: Identifying a row header data block in each of the initial data blocks to obtain a row header length; Performing a subtraction operation based on the predetermined segmentation block length and the line header length to obtain a third segmentation block length; Based on the third segmentation length, data segmentation processing is performed by adopting the method of patrolling the row header data block and the row data blocks in each of the initial data blocks to obtain each of the target data blocks.

8. A text segmentation device, characterized in that: include: The acquisition module is used to obtain the text to be segmented; A determination module, used for determining an initial segmentation method for segmenting the text to be segmented based on the text type of the text to be segmented; The first segmentation module is used to perform data segmentation processing on the text to be segmented by adopting the initial segmentation method to obtain initial data blocks corresponding to the text to be segmented; The second segmentation module is used to perform data segmentation processing on each of the initial data blocks by at least adopting a method of dynamically adjusting the segmentation block length to obtain each target data block corresponding to the text to be segmented.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the text segmentation method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method comprises at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the text segmentation method according to any one of claims 1 to 7 when executing the computer program on the memory.

Images