Text segmentation method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of artificial intelligence and text processing, is applied to a text content segmentation scene in a document or clause, and relates to a text segmentation method and device, equipment and a storage medium thereof. Extracting a full amount of text content to be segmented; inputting into a pre-trained segmentation model; segmenting the total to-be-segmented text content to obtain all segmented sequence fragments contained in the total to-be-segmented text content; and outputting a segmentation processing result from the segmentation model according to the segmentation sequence fragments. When the text segmentation method is applied to an insurance service document or a text content segmentation scene in terms, the basis terms can be quickly screened out through the text segmentation method when claim settlement basis term search is carried out, and particularly, in the field of insurance services with a large number of protocol terms, the claim settlement basis term search efficiency is greatly improved. And underwriting or claim settlement processing personnel can be assisted to quickly find the basis terms, so that the business processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and text processing technology, and is applied to the text content segmentation scenario in documents or clauses, and in particular to a text segmentation method, device, equipment and storage medium thereof. Background Art

[0002] In the retrieval-enhanced dialogue system, reasonable and effective text segmentation plays a vital role. The system combines the retrieved text knowledge with the dialogue model to improve the information content and accuracy of the dialogue response. In this process, text segmentation is the link between the two key links of retrieval and generation, and has a profound impact on the final dialogue quality.

[0003] At present, there are mainly the following methods for text segmentation: rule-based text segmentation method, which segments text according to predefined rules, for example, it can be segmented according to punctuation marks such as periods and commas, or according to parts of speech, but the rules are difficult to formulate flexibly and cannot adapt to complex texts; statistical-based text segmentation method, which uses statistical models to learn the segmentation rules of text, for example, models such as hidden Markov models or conditional random fields can be used, but a large amount of labeled data is required, the training cost is high, and the effect depends on the data quality; deep learning-based text segmentation method, which uses deep neural networks to learn the segmentation features of text, for example, models such as convolutional neural networks or recursive neural networks can be used, but the training of neural network models is complex and requires a large amount of data and computing resources. In the modern financial insurance industry, it is crucial for business personnel to obtain accurate insurance terms and policy information in a timely and efficient manner. The above existing text segmentation methods all have certain limitations. In the insurance business field with a large number of agreement terms, they cannot assist underwriters or claims processing personnel to quickly find the basis terms, resulting in the problem that insurance document retrieval is usually inefficient. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a text segmentation method, device, equipment and storage medium thereof to solve the problem that existing text segmentation methods have certain limitations. In the insurance business field with a large number of agreement clauses, it is impossible to assist underwriters or claims processing personnel to quickly find the basis clauses, resulting in the problem that insurance document retrieval is usually inefficient.

[0005] In order to solve the above technical problems, the present application embodiment provides a text segmentation method, which adopts the following technical solutions:

[0006] A text segmentation method comprises the following steps:

[0007] Get the text to be segmented;

[0008] Extracting the entire text content to be segmented from the text to be segmented in sequence according to the row and column arrangement of the text content;

[0009] Inputting the entire text content to be segmented into the pre-trained segmentation model;

[0010] Based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the entire amount of text content to be segmented is segmented to obtain all segmentation sequence fragments contained in the entire amount of text content to be segmented;

[0011] According to all segmentation sequence segments contained in the full amount of text content to be segmented, the segmentation model outputs a segmentation processing result.

[0012] Furthermore, after executing the step of sequentially extracting the entire amount of the text content to be segmented from the text to be segmented according to the row and column arrangement of the text content, the method further includes:

[0013] Inputting the full amount of text content to be segmented into a pre-trained Seq2Seq-based text processing model to obtain a text sequence to be segmented containing segmentation marks generated by the text processing model, wherein the Seq2Seq-based text processing model includes a GPT-3 model;

[0014] Identify all segmentation marks in the to-be-segmented text sequence in turn, and identify position coding information of all segmentation marks in the to-be-segmented text sequence respectively;

[0015] Based on the position encoding information of all the segmentation marks in the text sequence to be segmented, all segmentation sequence fragments contained in the text sequence to be segmented are extracted.

[0016] Furthermore, after executing the step of extracting all segmentation sequence fragments contained in the to-be-segmented text sequence based on the position coding information of all segmentation marks in the to-be-segmented text sequence, the method further includes:

[0017] Comparing the consistency of all segmented sequence segments contained in the to-be-segmented text sequence with all segmented sequence segments contained in the full to-be-segmented text content;

[0018] According to the comparison result, it is identified whether all segmentation sequence segments included in the to-be-segmented text sequence are consistent with all segmentation sequence segments included in the full to-be-segmented text content.

[0019] Furthermore, before executing the step of inputting the entire amount of text content to be segmented into the pre-trained segmentation model, the method further includes:

[0020] Get the training set text;

[0021] Extracting the full amount of training text content from the training set text in sequence according to the row and column arrangement of the text content;

[0022] Inputting the full amount of training text content into the pre-trained Seq2Seq-based text processing model to obtain a training text sequence containing segmentation marks generated by the text processing model;

[0023] Identifying all segmentation marks in the training text sequence in sequence, and identifying position coding information of all segmentation marks in the training text sequence;

[0024] Extracting all segmentation sequence fragments contained in the training text sequence based on the position encoding information of all segmentation marks in the training text sequence respectively;

[0025] Constructing training input data based on the full amount of training text content and all segmented sequence segments contained in the training text sequence;

[0026] The training input data is input into the self-adjusting optimization segmentation model to be trained to obtain a pre-trained segmentation model.

[0027] Furthermore, the step of inputting the training input data into the self-adjusting optimization segmentation model to be trained to obtain a pre-trained segmentation model specifically includes:

[0028] Learning the segmentation mapping relationship between the full amount of training text content in the training input data and all segmentation sequence fragments contained in the training text sequence by machine learning;

[0029] The segmentation mapping relationship is set as the learned segmentation reference knowledge to the self-adjusting optimization segmentation model to be trained to obtain the pre-trained segmentation model.

[0030] Furthermore, the step of outputting the segmentation processing result from the segmentation model according to all the segmentation sequence segments contained in the full amount of text content to be segmented specifically includes:

[0031] If all segmentation sequence segments included in the to-be-segmented text sequence are consistent with all segmentation sequence segments included in the full amount of to-be-segmented text content, then all segmentation sequence segments included in the full amount of to-be-segmented text content are output from the segmentation model as the segmentation processing result;

[0032] If there is inconsistency between all segmentation sequence segments included in the to-be-segmented text sequence and all segmentation sequence segments included in the full to-be-segmented text content, then all inconsistent segmentation sequence segments are screened out and all consistent segmentation sequence segments are screened out;

[0033] Arrange all the inconsistent segmented sequence fragments into groups in pairs according to the segmented sequence fragment comparison relationship to obtain a group arrangement result;

[0034] Pushing the grouping results to a preset expert screening end to obtain the preferred segmentation sequence fragments selected by the expert screening end from the grouping results;

[0035] The preferred segmentation sequence segments and all the segmentation sequence segments that are consistent after screening are output from the segmentation model as the segmentation processing results.

[0036] Furthermore, after executing the step of pushing the grouping results to a preset expert screening end to obtain the preferred segmented sequence segments selected by the expert screening end from the grouping results, the method further includes:

[0037] Identifying text contents corresponding to all segmentation sequence segments in the preferred segmentation sequence segments from the entire amount of text content to be segmented;

[0038] According to the text contents respectively corresponding to all the segmented sequence segments in the preferred segmented sequence segments, the text processing model is optimized and adjusted for segmentation markings; and

[0039] According to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments, the segmentation optimization adjustment is performed on the pre-trained segmentation model.

[0040] In order to solve the above technical problems, the embodiment of the present application also provides a text segmentation device, which adopts the following technical solution:

[0041] A text segmentation device, comprising:

[0042] A module for obtaining text to be segmented, used for obtaining text to be segmented;

[0043] A module for extracting the content of the text to be segmented, used to extract the full amount of the content of the text to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns;

[0044] A text content input module to be segmented, used to input the full amount of text content to be segmented into a pre-trained segmentation model;

[0045] The text content segmentation module to be segmented is used to segment the entire text content to be segmented based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model to obtain all segmentation sequence fragments contained in the entire text content to be segmented;

[0046] The segmentation processing result output module is used to output the segmentation processing result from the segmentation model according to all the segmentation sequence fragments contained in the full amount of text content to be segmented.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0048] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned text segmentation method when executing the computer-readable instructions.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0050] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the text segmentation method as described above.

[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0052] The text segmentation method described in the embodiment of the present application is as follows: obtaining the text to be segmented; extracting the full text content to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns; inputting the full text content to be segmented into a pre-trained segmentation model; performing segmentation processing on the full text content to be segmented based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model to obtain all the segmentation sequence fragments contained in the full text content to be segmented; and outputting the segmentation processing result from the segmentation model based on all the segmentation sequence fragments contained in the full text content to be segmented. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses. When searching for claim basis clauses, the text segmentation method can be used to quickly filter out the basis clauses. Especially in the insurance business field with a large number of agreement clauses, it can assist underwriters or claims processing personnel to quickly find the basis clauses and improve business processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0055] Figure 2 is a flowchart of an embodiment of a text segmentation method according to the present application;

[0056] Figure 3 It is a flowchart of a specific embodiment of extracting all segmentation sequence fragments from the full amount of text content to be segmented based on the Seq2Seq text processing model in the text segmentation method described in the present application;

[0057] Figure 4 It is a flowchart of a specific embodiment of comparing the segmentation consistency of all segmented sequence fragments of the full amount of text content to be segmented extracted based on the Seq2Seq text processing model and the segmentation model in the text segmentation method described in the present application;

[0058] Figure 5 is a flowchart of a specific embodiment of pre-training the segmentation model in the text segmentation method described in the present application;

[0059] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 507 is shown;

[0060] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0061] Figure 8 It is a flowchart of a specific embodiment of optimizing and adjusting the model in the text segmentation method described in this application;

[0062] Fig. 9 is a structural schematic diagram of an embodiment of a text segmentation device according to the present application;

[0063] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0067] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0068] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0070] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0071] It should be noted that the text segmentation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the text segmentation device is generally set in the server.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] Continue to refer Figure 2 , shows a flow chart of an embodiment of a text segmentation method according to the present application. The text segmentation method comprises the following steps:

[0074] Step 201, obtaining the text to be segmented.

[0075] In this embodiment, the text to be segmented is, for example, an insurance agreement, an underwriting agreement, a claim settlement agreement, or any other electronic contract used to record text content.

[0076] In this embodiment, the obtaining of the text to be segmented specifically includes, for example, obtaining the claims-related electronic file text uploaded by the underwriter through a preset document upload interface during the claims business process.

[0077] Step 202: extract the entire text content to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns.

[0078] In this embodiment, before the step of extracting the entire amount of text content to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns, a document content scanning method can be adopted. First, the text content to be segmented is scanned. Then, the entire amount of text content to be segmented is extracted in sequence from the text content scanning results in sequence according to the arrangement of the text content in rows and columns.

[0079] Specifically, the entire text content to be segmented is extracted from the text to be segmented in sequence according to the row and column arrangement of the text content. The row and column arrangement of the text content includes the row and column arrangement from top to bottom and from left to right in the document according to the text content. The entire text content to be segmented refers to the text content to be extracted according to the actual extraction requirements. Assuming that the actual extraction requirement is to extract the liability clause content part in the insurance agreement, that is, the text to be segmented is only the liability clause content part in the target insurance agreement. At this time, it is only necessary to execute step 202 according to the extraction requirement, and extract all the liability clause contents in the target insurance agreement as the entire text content to be segmented according to the row and column arrangement from top to bottom and from left to right in the document according to the liability clause content part in the target insurance agreement.

[0080] In this embodiment, the extraction action can be performed by optical recognition extraction to extract text content. First, the target text content to be extracted is identified by optical recognition, and then the text data content is obtained according to the character stream transmission method.

[0081] In addition, after step 202, it is not excluded that the extracted full amount of text content to be segmented is subjected to preprocessing operations, for example: masking the text content involving sensitive characters, performing error correction on obviously erroneous text data content, cleaning and re-encoding unrecognizable characters with obvious encoding errors. Here, the preprocessing can be specifically set according to actual business needs, and its purpose is to facilitate computer processing and recognition.

[0082] The text to be segmented is obtained from step 201 to step 202, and the entire text content to be segmented is extracted from the text to be segmented in sequence according to the row and column arrangement of the text content, so as to facilitate the subsequent segmentation processing of the entire text content to be segmented.

[0083] Step 203: input the entire text content to be segmented into the pre-trained segmentation model.

[0084] Step 204, based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the entire amount of text content to be segmented is segmented to obtain all segmentation sequence fragments contained in the entire amount of text content to be segmented.

[0085] Through the pre-trained segmentation model and in combination with the segmentation reference knowledge, the model processing method is used to perform segmentation processing on the entire amount of text content to be segmented, and all segmentation sequence fragments contained in the entire amount of text content to be segmented are obtained, thereby realizing intelligent and automatic execution of segmentation processing operations.

[0086] Step 205: output a segmentation processing result from the segmentation model according to all segmentation sequence segments contained in the full amount of text content to be segmented.

[0087] Specifically, the text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses. When it is necessary to search for claim basis clauses, the text segmentation method can be used to quickly screen out the basis clauses. Especially in the fields of auto insurance, life insurance, and property insurance business with a large number of agreement clauses, it can assist underwriters or claims processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0088] In this embodiment, the text to be segmented is obtained; the entire text content to be segmented is sequentially extracted from the text to be segmented according to the arrangement of the text content in rows and columns; the entire text content to be segmented is input into a pre-trained segmentation model; based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the entire text content to be segmented is segmented to obtain all the segmentation sequence fragments contained in the entire text content to be segmented; and the segmentation processing result is output from the segmentation model according to all the segmentation sequence fragments contained in the entire text content to be segmented. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses. When searching for claim basis clauses, the text segmentation method can be used to quickly filter out the basis clauses, especially in the insurance business field with a large number of agreement clauses. It can assist underwriters or claim processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0089] Continue to refer Figure 3 In some optional implementations, after step 202, a step of extracting all segmentation sequence fragments from the entire text content to be segmented according to a Seq2Seq-based text processing model is also included. Figure 3 It is a flowchart of a specific embodiment of extracting all segmentation sequence fragments from the full amount of text content to be segmented based on the Seq2Seq text processing model in the text segmentation method described in the present application, comprising the following steps:

[0090] Step 301, inputting the full amount of text content to be segmented into a pre-trained Seq2Seq-based text processing model to obtain a text sequence to be segmented containing segmentation marks generated by the text processing model, wherein the Seq2Seq-based text processing model includes a GPT-3 model;

[0091] Specifically, assuming that the full text to be segmented is "abcdefgh", assuming that the segmentation mark is " / ", the text sequence to be segmented containing the segmentation mark generated by the text processing model is, for example, "ab / cde / fg / h". By using a text processing model directly based on Seq2Seq, including the GPT-3 model, and using a large language model for semantic segmentation, there is no need for complex feature engineering, and the powerful semantic understanding ability of the large model is fully utilized, so that the text segmentation result will be more reasonable and semantically coherent.

[0092] Step 302, identifying all segmentation marks in the to-be-segmented text sequence in turn, and identifying position coding information of all segmentation marks in the to-be-segmented text sequence respectively;

[0093] Specifically, continuing with the above embodiment, assuming that the text sequence to be segmented is "ab / cde / fg / h", the position encoding information of all the segmentation marks in the text sequence to be segmented are 3, 7 and 10 respectively, or the segmentation marks are "between b and c", "between e and f" and "between g and h".

[0094] Step 303: extract all segmentation sequence fragments contained in the text sequence to be segmented based on the position coding information of all segmentation marks in the text sequence to be segmented.

[0095] The Seq2Seq-based text processing model is used to extract all segmentation sequence fragments from the entire text content to be segmented, so as to facilitate subsequent consistency verification and comparison with all segmentation sequence fragments output by the segmentation model, and also to optimize and update the segmentation model according to the segmentation sequence fragments, so as to make the final text segmentation more accurate.

[0096] Continue to refer Figure 4 In some optional implementations, after step 303, a step of comparing the segmentation consistency of all segmented sequence fragments of the full amount of text content to be segmented extracted by the Seq2Seq-based text processing model and the segmentation model is also included. Figure 4 It is a flowchart of a specific embodiment of comparing the segmentation consistency of all segmented sequence fragments of the full amount of text content to be segmented extracted based on the Seq2Seq text processing model and the segmentation model in the text segmentation method described in the present application, comprising the following steps:

[0097] Step 401, performing consistency comparison between all segmentation sequence segments included in the to-be-segmented text sequence and all segmentation sequence segments included in the full to-be-segmented text content;

[0098] Step 402: According to the comparison result, it is identified whether all segmentation sequence segments included in the to-be-segmented text sequence are consistent with all segmentation sequence segments included in the full to-be-segmented text content.

[0099] By comparing the segmentation consistency of all segmented sequence fragments of the full text content to be segmented extracted by the Seq2Seq-based text processing model and the segmentation model, it is used to segment and update the text processing model on the one hand, and to optimize and update the segmentation model on the other hand, thereby ensuring timely updating and optimization of relevant processing models and improving the efficiency and accuracy of text segmentation.

[0100] Continue to refer Figure 5 In some optional implementations, a step of pre-training the segmentation model is also included before step 203. Figure 5 It is a flowchart of a specific embodiment of pre-training the segmentation model in the text segmentation method described in the present application, comprising the following steps:

[0101] Step 501, obtaining training set text;

[0102] Specifically, the training set text is a batch text of the same version as the text to be segmented. For example, if the text to be segmented is an insurance agreement text of a specified version, then the training set text is a batch of insurance agreement texts of the specified version.

[0103] Step 502, extracting the full amount of training text content from the training set text in sequence according to the row and column arrangement of the text content;

[0104] Specifically, the full amount of training text content is the text content of the same section or the same unit as the full amount of text content to be segmented in the insurance agreement text. For example, the full amount of text content to be segmented is the text content of legal terms in the insurance agreement text, then the full amount of training text content is all the text content of legal terms extracted from the training set text.

[0105] Step 503: input the full amount of training text content into the pre-trained Seq2Seq-based text processing model to obtain a training text sequence containing segmentation marks generated by the text processing model;

[0106] Step 504, identifying all segmentation marks in the training text sequence in sequence, and identifying position coding information of all segmentation marks in the training text sequence;

[0107] Step 505, extracting all segmentation sequence segments contained in the training text sequence based on the position coding information of all segmentation marks in the training text sequence;

[0108] Step 506, constructing training input data based on the full amount of training text content and all segmented sequence segments contained in the training text sequence;

[0109] Step 507: input the training input data into the self-adjusting optimization segmentation model to be trained to obtain a pre-trained segmentation model.

[0110] The self-adjusting optimization segmentation model is used to learn the segmentation mapping relationship between the full amount of training text content in the training input data and all segmentation sequence fragments contained in the training text sequence by machine learning.

[0111] By pre-training the segmentation model, the pre-trained segmentation model can be directly used to segment the entire amount of text content to be segmented, thereby improving the text segmentation efficiency in actual use.

[0112] Continue to refer Figure 6 , in some optional implementations, Figure 6 yes Figure 5 The flowchart of a specific embodiment of step 507 shown includes the following steps:

[0113] Step 601, learning the segmentation mapping relationship between the full amount of training text content in the training input data and all segmentation sequence segments contained in the training text sequence by machine learning;

[0114] Step 602: setting the segmentation mapping relationship as the learned segmentation reference knowledge to the self-adjusting optimization segmentation model to be trained to obtain the pre-trained segmentation model.

[0115] The segmentation mapping relationship between the full training text content in the training input data and all segmentation sequence fragments contained in the training text sequence is learned by machine learning, thereby pre-training the segmentation model and obtaining the pre-trained segmentation model, thereby ensuring the data basis of the segmentation model when performing text segmentation.

[0116] Continue to refer Figure 7 , in some optional implementations, Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes the following steps:

[0117] Step 701: if all segmentation sequence segments included in the to-be-segmented text sequence are consistent with all segmentation sequence segments included in the full to-be-segmented text content, then all segmentation sequence segments included in the full to-be-segmented text content are output from the segmentation model as the segmentation processing result;

[0118] Specifically, if all the segmented sequence fragments contained in the text sequence to be segmented are consistent with all the segmented sequence fragments contained in the full text content to be segmented, it means that the segmentation results of the two models, namely the text processing model and the segmentation model, are consistent, and all the segmented sequence fragments contained in the full text content to be segmented can be directly output from the segmentation model as the segmentation processing results.

[0119] Step 702: if there is inconsistency between all segmentation sequence segments included in the to-be-segmented text sequence and all segmentation sequence segments included in the full to-be-segmented text content, then all inconsistent segmentation sequence segments are screened out and all consistent segmentation sequence segments are screened out;

[0120] Step 703, grouping all the inconsistent segmented sequence segments in pairs according to the segmented sequence segment comparison relationship to obtain a grouping result;

[0121] Step 704, pushing the grouping results to a preset expert screening end, and obtaining the preferred segmentation sequence segments selected by the expert screening end from the grouping results;

[0122] Step 705: Output the preferred segmentation sequence fragment and all the screened consistent segmentation sequence fragments as the segmentation processing results from the segmentation model.

[0123] Specifically, if there is an inconsistency between all the segmented sequence fragments contained in the text sequence to be segmented and all the segmented sequence fragments contained in the full text content to be segmented, all consistent segmented sequence fragments are screened out, and the inconsistent segmented sequence fragments are also screened and grouped in pairs to obtain the preferred segmented sequence fragments selected by the expert screening end, and the preferred segmented sequence fragments and all the screened out consistent segmented sequence fragments are output from the segmentation model as the segmentation processing results.

[0124] The above two segmentation result output modes reflect the cautious selection of segmentation results by the text segmentation method to ensure the accuracy of the text segmentation results.

[0125] Continue to refer Figure 8 In some optional implementations, after step 704, a step of optimizing and adjusting the model is also included. Figure 8It is a flowchart of a specific embodiment of optimizing and adjusting the model in the text segmentation method described in this application, comprising the following steps:

[0126] Step 801, identifying text contents corresponding to all segmentation sequence segments in the preferred segmentation sequence segments from the entire amount of text content to be segmented;

[0127] Step 802, optimizing and adjusting the segmentation marks of the text processing model according to the text contents corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments, and,

[0128] Step 803: performing segmentation optimization adjustment on the pre-trained segmentation model according to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments.

[0129] Specifically, the step of optimizing and adjusting the segmentation marks of the text processing model according to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments includes: re-acquiring the position coding information of all the segmentation marks in the text sequence to be segmented according to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments, and feeding back the position coding information to the text processing model to optimize and adjust the segmentation marks of the text processing model.

[0130] Specifically, the step of optimizing and adjusting the segmentation of the pre-trained segmentation model according to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments includes: optimizing and updating the segmentation reference knowledge in the pre-trained segmentation model according to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments, so as to achieve segmentation optimization and adjustment of the pre-trained segmentation model.

[0131] By performing segmentation mark optimization and adjustment on the text processing model and segmentation optimization and adjustment on the pre-trained segmentation model, it is achieved that when actually performing text segmentation, the processing models involved, such as the text processing model and the segmentation model, are continuously and timely optimized and adjusted to ensure that as the text segmentation processing data increases and the segmentation changes, the processing model is updated and optimized in a timely manner, thereby improving the accuracy and efficiency of text segmentation.

[0132] In this embodiment, when actually performing document segmentation, a large language model based on Seq2Seq and a segmentation model pre-trained by machine learning are successively used. According to the two document segmentation results, the final file segmentation result is determined, thereby ensuring the accuracy and credibility of the document segmentation.

[0133] The present application obtains the text to be segmented; extracts the full text content to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns; inputs the full text content to be segmented into a pre-trained segmentation model; based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the full text content to be segmented is segmented to obtain all the segmentation sequence fragments contained in the full text content to be segmented; and outputs the segmentation processing result from the segmentation model based on all the segmentation sequence fragments contained in the full text content to be segmented. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses. When searching for claim basis clauses, the text segmentation method can be used to quickly filter out the basis clauses, especially in the insurance business field with a large number of agreement clauses. It can assist underwriters or claims processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0134] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0135] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0136] In an embodiment of the present application, the text to be segmented is obtained; the entire text content to be segmented is sequentially extracted from the text to be segmented according to the arrangement of the text content in rows and columns; the entire text content to be segmented is input into a pre-trained segmentation model; based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the entire text content to be segmented is segmented to obtain all segmentation sequence fragments contained in the entire text content to be segmented; and the segmentation processing result is output from the segmentation model based on all segmentation sequence fragments contained in the entire text content to be segmented. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses. When searching for claim basis clauses, the text segmentation method can be used to quickly filter out the basis clauses, especially in the insurance business field with a large number of agreement clauses. It can assist underwriters or claims processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0137] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a text segmentation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0138] like Fig. 9 As shown, the text segmentation device 900 described in this embodiment includes: a text acquisition module 901 to be segmented, a text content extraction module 902 to be segmented, a text content input module 903 to be segmented, a text content segmentation module 904 to be segmented, and a segmentation processing result output module 905. Among them:

[0139] The to-be-segmented text acquisition module 901 is used to acquire the to-be-segmented text;

[0140] The module 902 for extracting the content of the text to be segmented is used to extract the full content of the text to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns;

[0141] The text content to be segmented input module 903 is used to input the full amount of text content to be segmented into the pre-trained segmentation model;

[0142] The text content segmentation module 904 is used to segment the entire text content to be segmented based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model to obtain all segmentation sequence fragments contained in the entire text content to be segmented;

[0143] The segmentation processing result output module 905 is used to output the segmentation processing result from the segmentation model according to all the segmentation sequence segments contained in the full amount of text content to be segmented.

[0144] The present application obtains the text to be segmented; extracts the full text content to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns; inputs the full text content to be segmented into a pre-trained segmentation model; based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the full text content to be segmented is segmented to obtain all the segmentation sequence fragments contained in the full text content to be segmented; and outputs the segmentation processing result from the segmentation model based on all the segmentation sequence fragments contained in the full text content to be segmented. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses. When searching for claim basis clauses, the text segmentation method can be used to quickly filter out the basis clauses, especially in the insurance business field with a large number of agreement clauses. It can assist underwriters or claims processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0146] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0147] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0148] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0149] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0150] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 10. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store the operating system and various application software installed on the computer device 10, such as a computer-readable instruction of a text segmentation method, etc. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or are to be output.

[0151] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions stored in the memory 10a or process data, such as computer-readable instructions for running the text segmentation method.

[0152] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0153] The computer device proposed in this embodiment belongs to the field of artificial intelligence and text processing technology, and is applied to the text content segmentation scenario in documents or clauses. This application obtains the text to be segmented; according to the arrangement of the text content in rows and columns, the full amount of text content to be segmented is extracted from the text to be segmented in sequence; the full amount of text content to be segmented is input into the pre-trained segmentation model; based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the full amount of text content to be segmented is segmented to obtain all the segmentation sequence fragments contained in the full amount of text content to be segmented; according to all the segmentation sequence fragments contained in the full amount of text content to be segmented, the segmentation processing result is output from the segmentation model. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses, and the basis clauses can be quickly screened out through the text segmentation method when searching for the claim basis clauses, especially in the insurance business field with a large number of agreement clauses, which can assist underwriters or claim processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0154] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the text segmentation method as described above.

[0155] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence and text processing technology, and is applied to the text content segmentation scenario in documents or clauses. This application obtains the text to be segmented; extracts the full amount of text content to be segmented from the text to be segmented in sequence according to the row and column arrangement of the text content; inputs the full amount of text content to be segmented into a pre-trained segmentation model; based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the full amount of text content to be segmented is segmented to obtain all the segmentation sequence fragments contained in the full amount of text content to be segmented; according to all the segmentation sequence fragments contained in the full amount of text content to be segmented, the segmentation processing result is output from the segmentation model. The text segmentation method is applied to the text content segmentation scenario in insurance business documents or clauses, and the basis clauses can be quickly screened out through the text segmentation method when searching for the claim basis clauses, especially in the insurance business field with a large number of agreement clauses, which can assist underwriters or claim processing personnel to quickly find the basis clauses and improve business processing efficiency.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0157] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A text segmentation method, characterized in that: The steps include: Get the text to be segmented; Extracting the entire text content to be segmented from the text to be segmented in sequence according to the row and column arrangement of the text content; Inputting the entire text content to be segmented into the pre-trained segmentation model; Based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model, the entire amount of text content to be segmented is segmented to obtain all segmentation sequence fragments contained in the entire amount of text content to be segmented; According to all segmentation sequence segments contained in the full amount of text content to be segmented, the segmentation model outputs a segmentation processing result.

2. The text segmentation method according to claim 1, characterized in that: After executing the step of sequentially extracting the entire amount of the text content to be segmented from the text to be segmented according to the row and column arrangement of the text content, the method further includes: Inputting the full amount of text content to be segmented into a pre-trained Seq2Seq-based text processing model to obtain a text sequence to be segmented containing segmentation marks generated by the text processing model, wherein the Seq2Seq-based text processing model includes a GPT-3 model; Identify all segmentation marks in the to-be-segmented text sequence in turn, and identify position coding information of all segmentation marks in the to-be-segmented text sequence respectively; Based on the position encoding information of all the segmentation marks in the text sequence to be segmented, all segmentation sequence fragments contained in the text sequence to be segmented are extracted.

3. The text segmentation method according to claim 2, characterized in that: After executing the step of extracting all segmentation sequence fragments contained in the to-be-segmented text sequence based on the position encoding information of all segmentation marks in the to-be-segmented text sequence, the method further includes: Comparing the consistency of all segmented sequence segments contained in the to-be-segmented text sequence with all segmented sequence segments contained in the full to-be-segmented text content; According to the comparison result, it is identified whether all segmentation sequence segments included in the to-be-segmented text sequence are consistent with all segmentation sequence segments included in the full to-be-segmented text content.

4. The text segmentation method according to claim 1, characterized in that: Before executing the step of inputting the entire amount of text content to be segmented into the pre-trained segmentation model, the method further includes: Get the training set text; Extracting the full amount of training text content from the training set text in sequence according to the row and column arrangement of the text content; Inputting the full amount of training text content into the pre-trained Seq2Seq-based text processing model to obtain a training text sequence containing segmentation marks generated by the text processing model; Identifying all segmentation marks in the training text sequence in sequence, and identifying position coding information of all segmentation marks in the training text sequence; Extracting all segmentation sequence fragments contained in the training text sequence based on the position encoding information of all segmentation marks in the training text sequence respectively; Constructing training input data based on the full amount of training text content and all segmented sequence segments contained in the training text sequence; The training input data is input into the self-adjusting optimization segmentation model to be trained to obtain a pre-trained segmentation model.

5. The text segmentation method according to claim 4, characterized in that: The step of inputting the training input data into the self-adjusting optimization segmentation model to be trained to obtain a pre-trained segmentation model specifically includes: Learning the segmentation mapping relationship between the full amount of training text content in the training input data and all segmentation sequence fragments contained in the training text sequence by machine learning; The segmentation mapping relationship is set as the learned segmentation reference knowledge to the self-adjusting optimization segmentation model to be trained to obtain the pre-trained segmentation model.

6. The text segmentation method according to claim 3, characterized in that: The step of outputting the segmentation processing result from the segmentation model according to all the segmentation sequence segments contained in the full amount of text content to be segmented specifically includes: If all segmentation sequence segments included in the to-be-segmented text sequence are consistent with all segmentation sequence segments included in the full amount of to-be-segmented text content, then all segmentation sequence segments included in the full amount of to-be-segmented text content are output from the segmentation model as the segmentation processing result; If there is inconsistency between all segmentation sequence segments included in the to-be-segmented text sequence and all segmentation sequence segments included in the full to-be-segmented text content, then all inconsistent segmentation sequence segments are screened out and all consistent segmentation sequence segments are screened out; Arrange all the inconsistent segmented sequence fragments into groups in pairs according to the segmented sequence fragment comparison relationship to obtain a group arrangement result; Pushing the grouping results to a preset expert screening end to obtain the preferred segmentation sequence fragments selected by the expert screening end from the grouping results; The preferred segmentation sequence segments and all the segmentation sequence segments that are consistent after screening are output from the segmentation model as the segmentation processing results.

7. The text segmentation method according to claim 6, characterized in that: After executing the step of pushing the grouping results to a preset expert screening end to obtain the preferred segmented sequence segments selected by the expert screening end from the grouping results, the method further includes: Identifying text contents corresponding to all segmentation sequence segments in the preferred segmentation sequence segments from the entire amount of text content to be segmented; According to the text contents respectively corresponding to all the segmented sequence segments in the preferred segmented sequence segments, the segmentation mark optimization adjustment is performed on the text processing model, and, According to the text contents respectively corresponding to all the segmentation sequence segments in the preferred segmentation sequence segments, the segmentation optimization adjustment is performed on the pre-trained segmentation model.

8. A text segmentation device, characterized in that: include: A module for obtaining text to be segmented, used for obtaining text to be segmented; A module for extracting the content of the text to be segmented, used to extract the full amount of the content of the text to be segmented from the text to be segmented in sequence according to the arrangement of the text content in rows and columns; A text content input module to be segmented, used to input the full amount of text content to be segmented into a pre-trained segmentation model; The text content segmentation module to be segmented is used to segment the entire text content to be segmented based on the segmentation reference knowledge learned by machine learning in the pre-trained segmentation model to obtain all segmentation sequence fragments contained in the entire text content to be segmented; The segmentation processing result output module is used to output the segmentation processing result from the segmentation model according to all the segmentation sequence fragments contained in the full amount of text content to be segmented.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text segmentation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text segmentation method according to any one of claims 1 to 7 are implemented.

Images