Text segmentation method, system and computer storage medium based on machine learning

Through the machine learning-based text sentence breaking method, irregular text information is processed, and the problem of low efficiency and accuracy of text information processing in the existing technology is solved, and efficient and accurate text sentence breaking is achieved.

CN112667769BActive Publication Date: 2025-03-11GUANGZHOU HUIRUI SITONG INFORMATION SCI & TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201910944700.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-30
Publication Date
2025-03-11
Estimated Expiration
2039-09-30

AI Technical Summary

Technical Problem

The prior art is difficult to deal with text information full of online terms, abuse of homophones, omitting or abuse of punctuation marks, resulting in troubles in some occasions of natural language processing methods.

Method used

Using the text sentence breaking method based on machine learning, we obtain the sentence breaking text by obtaining the text library, building the training set model, and preset text input training model. The method includes removing punctuation marks and spaces, adding padding characters, building a training set, training a model, extracting non-zero numeric strings, corresponding punctuation marks and calibrating.

Benefits of technology

It improves the processing efficiency and accuracy of text sentence breaking, increases the throughput of business, and solves the natural language processing problems caused by irregular text information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112667769B_ABST
    Figure CN112667769B_ABST
Patent Text Reader

Abstract

The present invention relates to a text segmentation method, system and computer storage medium based on machine learning, the method comprising: obtaining various texts to create a text library; using all continuous texts in the text library as training samples to construct a training set; inputting the training set into a machine learning model to construct a training set model; inputting a preset text into the training set model for training to obtain a segmented text. The technical solution provided by the present invention adopts machine learning to construct a training model, which greatly improves the processing efficiency and accuracy of segmentation and increases the throughput of the business.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer algorithms, and in particular to a text segmentation method, system and computer storage medium based on machine learning. Background Art

[0002] In recent years, with the advancement of computer technology and the development of science and technology, artificial intelligence technology has entered all aspects of our lives, and natural language processing technology, as one of its main branches, has greatly facilitated our lives.

[0003] However, in some cases, text information is often not standardized, such as being full of Internet slang, abusing various homophones, omitting or abusing punctuation marks, etc. This will cause great trouble for some natural language processing methods that require standardized input or have low tolerance.

[0004] Therefore, it is necessary to provide a text segmentation method, system and computer storage medium based on machine learning to address the deficiencies of the prior art. Summary of the invention

[0005] In order to solve the problems in the prior art, the present invention provides a text segmentation method, system and computer storage medium based on machine learning.

[0006] This application provides a text segmentation method based on machine learning, including:

[0007] Get various texts to create a text library;

[0008] Taking all the continuous texts in the text library as training samples, a training set is constructed;

[0009] Inputting the training set into a machine learning model to construct a training set model;

[0010] The preset text is input into the training set model for training to obtain the sentence segmentation text.

[0011] Furthermore, all the continuous texts in the text library are used as training samples to construct a training set, including:

[0012] According to the preset rules, each punctuation mark corresponds to a non-zero number;

[0013] Adding a padding character before the training sample;

[0014] Each character in the training sample corresponds to a number to form a training set; the number corresponding to non-punctuation characters is 0.

[0015] Furthermore, before inputting the preset text into the training set model for training to obtain the segmented text, the method further includes:

[0016] Removing punctuation marks and spaces from the text to be segmented to obtain a punctuation-free text, and storing the punctuation marks and spaces in a preset storage space;

[0017] A fill character is added before the text without punctuation marks to obtain a preset text.

[0018] Furthermore, the preset text is input into the training set model for training to obtain the sentence segmentation text, including:

[0019] Inputting the preset text into the training set model for training to obtain a trained digital string;

[0020] Extracting non-zero digits from the digital string to obtain a non-zero digital string;

[0021] According to a preset rule, the non-zero digital string is mapped to various punctuation marks;

[0022] The punctuation marks are introduced into the preset text to obtain the sentence-breaking text.

[0023] Furthermore, before the punctuation marks are introduced into the preset text to obtain the segmented text, the method further includes:

[0024] The various punctuation marks are calibrated by punctuation marks and spaces stored in a preset storage space.

[0025] Further, calibrating each punctuation mark by using the punctuation marks and spaces stored in a preset storage space includes:

[0026] Comparing each punctuation mark with punctuation marks and spaces stored in a preset storage space;

[0027] Pick out the punctuation marks that are different after comparison;

[0028] Selecting a punctuation mark to be determined according to the text content of the preset text;

[0029] The same punctuation marks and the undetermined punctuation marks are put into the preset text to obtain the segmented text.

[0030] The present invention also provides a text segmentation system based on machine learning, comprising:

[0031] The acquisition module is used to acquire various texts to create a text library;

[0032] A first construction module is used to construct a training set by taking all continuous texts in the text library as training samples;

[0033] A second construction module is used to input the training set into a machine learning model to construct a training set model;

[0034] The determination module is used to input the preset text into the training set model for training to obtain the sentence segmentation text.

[0035] Furthermore, the first building block includes:

[0036] The first corresponding submodule is used to correspond a non-zero number to each punctuation mark according to a preset rule;

[0037] An adding submodule, used for adding a padding character before the training sample;

[0038] A submodule is constructed to correspond each character in the training sample to a number to form a training set; the number corresponding to the non-punctuation character is 0.

[0039] Furthermore, the system further comprises:

[0040] A removal module, used for removing punctuation marks and spaces in the text to be segmented, obtaining a punctuation-free text, and storing the punctuation marks and spaces in a preset storage space;

[0041] The adding module is used to add a filling character before the text without punctuation marks to obtain a preset text.

[0042] The present invention also provides a computer storage medium, on which a text segmentation method program based on machine learning is stored. When the text segmentation method program based on machine learning is executed by a processor, the steps of any of the above-mentioned text segmentation methods based on machine learning are implemented.

[0043] The technical solution of the present invention has the following advantages compared with the closest prior art:

[0044] The technical solution provided by the present invention first obtains various texts to create a text library, then uses all continuous texts in the text library as training samples to construct a training set, then inputs the training set into a machine learning model to construct a training set model, and finally inputs a preset text into the above training set model for training to obtain a segmented text. The technical solution provided by the present invention uses machine learning to construct a training model, which greatly improves the processing efficiency and accuracy of segmentation and increases the throughput of the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a text segmentation method based on machine learning in an embodiment of the present invention;

[0046] Figure 2Schematic diagram of a text segmentation system based on machine learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 The present application provides a text segmentation method based on machine learning. The text segmentation method based on machine learning may include the following steps:

[0049] Get various texts to create a text library;

[0050] Taking all the continuous texts in the text library as training samples, a training set is constructed;

[0051] Inputting the training set into a machine learning model to construct a training set model;

[0052] The preset text is input into the training set model for training to obtain the sentence segmentation text.

[0053] In the embodiment of the present application, various texts are first obtained to create a text library, and then all continuous texts in the text library are used as training samples to construct a training set, and then the training set is input into the machine learning model to construct a training set model, and finally the preset text is input into the above training set model for training to obtain the segmented text. The technical solution provided by the present invention adopts machine learning to construct the training model, which greatly improves the processing efficiency and accuracy of segmentation and increases the throughput of the business.

[0054] In a possible implementation of the present application, all the continuous texts in the text library are used as training samples to construct a training set, which specifically includes:

[0055] According to the preset rules, each punctuation mark corresponds to a non-zero number;

[0056] Adding a padding character before the training sample;

[0057] Each character in the training sample corresponds to a number to form a training set; the number corresponding to non-punctuation characters is 0.

[0058] Specifically, first, each punctuation mark corresponds to a non-zero number according to the preset rules, for example: "." corresponds to "1", "," corresponds to "2", ";" corresponds to "3", etc. Then add a padding character before the training sample, and finally correspond each character in the training sample to a number to form a training set, that is, the training set is a pure digital label.

[0059] The number corresponding to the non-punctuation character is 0, that is, the number corresponding to the text is "0".

[0060] In a possible implementation manner of the present application, before inputting the preset text into the training set model for training to obtain the segmented text, the method further includes:

[0061] Removing punctuation marks and spaces from the text to be segmented to obtain a punctuation-free text, and storing the punctuation marks and spaces in a preset storage space;

[0062] A filler character is added before the text without punctuation marks to obtain a preset text.

[0063] In the embodiment of the present application, punctuation marks and spaces in the text to be segmented are first removed and moved to a preset storage space to obtain a punctuation-free text, and then a filler character is added before the text of the punctuation-free text to obtain a preset text. The text to be segmented is processed to remove punctuation marks so that it can be input into the training set model for segmentation processing, and the removed punctuation marks are saved, which can be used for calibration after the subsequent segmentation processing is completed, so that the segmentation is more accurate.

[0064] In a possible implementation manner of the present application, the preset price text is input into the training set model for training to obtain the segmented text, which specifically includes:

[0065] Inputting the preset text into the training set model for training to obtain a trained digital string;

[0066] Extracting non-zero digits from the digital string to obtain a non-zero digital string;

[0067] According to a preset rule, the non-zero digital string is mapped to various punctuation marks;

[0068] The punctuation marks are introduced into the preset text to obtain the sentence-breaking text.

[0069] In an embodiment of the present application, the preset text is first input into the training set model obtained above for training to obtain a trained digital string, and then the non-zero array in the digital string is extracted to obtain a non-zero digital string, and the non-zero array string is mapped to various punctuation marks according to preset rules, and finally the punctuation marks are brought into the preset text to obtain a segmented text. In an embodiment of the present application, the preset text is trained to obtain a digital string, and then mapped to corresponding punctuation marks, and brought back to the preset text to obtain a segmented text.

[0070] In a possible implementation manner of the present application, before the punctuation marks are introduced into the preset text to obtain the segmented text, the method further includes:

[0071] The various punctuation marks are calibrated by punctuation marks and spaces stored in a preset storage space.

[0072] That is, the punctuation marks obtained through training are calibrated by the removed punctuation marks and spaces stored above, so that the sentence segmentation is more accurate.

[0073] In a possible implementation manner of the present application, calibrating the punctuation marks by the punctuation marks and spaces stored in a preset storage space specifically includes:

[0074] Comparing each punctuation mark with punctuation marks and spaces stored in a preset storage space;

[0075] Pick out the punctuation marks that are different after comparison;

[0076] Selecting a punctuation mark to be determined according to the text content of the preset text;

[0077] The same punctuation marks and the undetermined punctuation marks are put into the preset text to obtain the segmented text.

[0078] In the embodiment of the present application, the removed punctuation marks are compared with the punctuation marks obtained by training, and then the differences are selected, and then the punctuation marks are determined according to the semantics of the original text, and finally the calibrated sentence segmentation text is obtained. Through this calibration method, the sentence segmentation is more accurate and more efficient.

[0079] The specific process of this application is as follows:

[0080] First, obtain various texts to create a text library, which is the raw data preparation stage.

[0081] Specifically, the acquisition method can be to crawl texts containing complete Chinese sentences through the Internet or other channels, including but not limited to: Chinese Wikipedia content, Baidu Encyclopedia content, People's Daily news texts over the years, public Chinese novels, and various texts containing standard characters on major portals.

[0082] Then, the acquired text data is preprocessed.

[0083] That is, the large text data prepared in the above steps is divided into short sentences for model training. In order to prevent data leakage, a fixed length is used to intercept the text so that punctuation marks (such as periods, question marks, etc.) do not always fall at the end of the sample. When the remaining text is less than the fixed length, it can be directly included in the processed sample. Because padding will be performed later, a fixed length is not a necessary condition.

[0084] In a specific embodiment of the present application, all punctuation marks correspond to a non-zero number, starting from 1, and each punctuation mark corresponds to a non-zero number.

[0085] For example: "." corresponds to "1", "," corresponds to "2", ";" corresponds to "3", and so on.

[0086] Then add a padding character before the sample, and then delete all punctuation marks in the sample. The label corresponding to this sample is a digital sequence with the same length as the number of words in the sample. If there is no punctuation mark after the word corresponding to this bit, this bit of the label is 0, otherwise it is equal to the number corresponding to the corresponding punctuation mark.

[0087] For example, for the sentence "I am XXX.", the corresponding label is "000001". The original text has only 5 characters except for punctuation, and the label has 6 characters because padding characters are added in the front. The purpose of adding padding characters is to make the first character have a corresponding position. You can also use a bidirectional model, in which case you also need to add padding characters at the end of the text.

[0088] By doing this for each piece of text, a complete training set is constructed.

[0089] The model building process is as follows:

[0090] Through the above text processing steps, machine learning methods can be used for modeling and training, including hidden Markov models and random vector fields including probabilistic graph models, simple RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), Attention (attention model) or more complex sequence labeling models for modeling and training.

[0091] After building a complete and reasonable sequence labeling model, you can use the complete training set established in the above steps for training and verification. In particular, in order to ensure the actual effect of the model in practical applications, not only the accuracy of the detection results but also the recall rate should be considered as the criterion for judging the quality of the model training results.

[0092] The process of inputting the preset text into the training set model for training is as follows:

[0093] During the usage phase of the system, i.e. the prediction phase: the system receives a text input that does not contain punctuation or has incomplete punctuation, and outputs text with completed punctuation.

[0094] The specific working steps are as follows: first, remove the punctuation or spaces contained in the input text in the sample and store it in the preset storage space, then add a filler character in front of the text that does not contain any punctuation, and send it to the above-built and trained model. The model will output a string of labels. The numbers in it indicate whether there is a punctuation mark after each character and what punctuation mark it is.

[0095] The punctuation sequence obtained by the model is compared with the punctuation sequence information previously stored in the preset storage space and fused. The final punctuation is determined based on the context information at different positions of the two sequences.

[0096] Finally, the above method can be used to obtain a text sequence with complete punctuation information.

[0097] Based on the text after punctuation completion and the meaning of punctuation marks, the text can be segmented in a very simple way.

[0098] In another embodiment of the present application, a text segmentation system based on machine learning is also provided. Figure 2 As shown, the text segmentation system based on machine learning may include the following modules: an acquisition module, a first construction module, a second construction module and a determination module.

[0099] The acquisition module is configured to acquire various texts to create a text library;

[0100] The first construction module is configured to use all the continuous texts in the text library as training samples to construct a training set;

[0101] The second building module is configured to input the training set into the machine learning model to build a training set model;

[0102] The determination module is configured to input the preset text into the training set model for training to obtain the segmented text.

[0103] In the embodiment of the present application, firstly, various texts are acquired through the acquisition module to create a text library, then all continuous texts in the text library are used as training samples through the first construction module to construct a training set, and then the training set is input into the machine learning model through the second construction module to construct a training set model, and finally the preset text is input into the above training set model through the determination module for training to obtain the segmented text. The technical solution provided by the present invention adopts machine learning to construct the training model, which greatly improves the processing efficiency and accuracy of segmentation and increases the throughput of the business.

[0104] Furthermore, the first construction module specifically includes: a first corresponding submodule, an adding submodule and a construction submodule.

[0105] The first corresponding submodule is configured to correspond a non-zero number to each punctuation mark according to a preset rule;

[0106] The adding submodule is configured to add a padding character before the training sample;

[0107] The construction submodule is configured to correspond each character in the training sample to a number to form a training set; the number corresponding to the non-punctuation character is 0.

[0108] Furthermore, the text segmentation system based on machine learning may also include: a removal module and an addition module.

[0109] The removal module is configured to remove punctuation marks and spaces in the text to be segmented to obtain a punctuation-free text, and store the punctuation marks and spaces in a preset storage space;

[0110] The adding module is configured to add a filling character before the text without punctuation marks to obtain a preset text.

[0111] Furthermore, the determination module specifically includes: an input submodule, an extraction submodule, a second corresponding submodule and a first determination submodule.

[0112] The input submodule is configured to input the preset text into the training set model for training to obtain a trained digital string;

[0113] The extraction submodule is configured to extract non-zero digits in the digital string to obtain a non-zero digital string;

[0114] The second corresponding submodule is configured to correspond the non-zero digital string to various punctuation marks according to a preset rule;

[0115] The first determining submodule is configured to bring the punctuation marks into the preset text to obtain a sentence-segmented text.

[0116] Furthermore, the text segmentation system based on machine learning may also include: a calibration module.

[0117] The calibration module is configured to calibrate the various punctuation marks using punctuation marks and spaces stored in a preset storage space.

[0118] Furthermore, the calibration module specifically includes: a comparison submodule, a selection submodule, a selection submodule and a second determination submodule.

[0119] The comparison submodule is configured to compare each punctuation mark with the punctuation marks and spaces stored in a preset storage space;

[0120] The selection submodule is configured to select punctuation marks that are different after comparison;

[0121] The selection submodule is configured to select a pending punctuation mark according to the text content of the preset text;

[0122] The second determining submodule is configured to put the same punctuation marks and the undetermined punctuation marks into a preset text to obtain a segmented text.

[0123] In another embodiment of the present disclosure, a computer storage medium is further provided, on which a text segmentation method program based on machine learning is stored. When the text segmentation method program based on machine learning is executed by a processor, the following steps are implemented:

[0124] Get various texts to create a text library;

[0125] Taking all the continuous texts in the text library as training samples, a training set is constructed;

[0126] Inputting the training set into a machine learning model to construct a training set model;

[0127] The preset text is input into the training set model for training to obtain the sentence segmentation text.

[0128] In the embodiment of the present application, various texts are first obtained to create a text library, and then all continuous texts in the text library are used as training samples to construct a training set, and then the training set is input into the machine learning model to construct a training set model, and finally the preset text is input into the above training set model for training to obtain the segmented text. The technical solution provided by the present invention adopts machine learning to construct the training model, which greatly improves the processing efficiency and accuracy of segmentation and increases the throughput of the business.

[0129] It should be noted that, in this document, relational terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0130] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0131] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0132] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0133] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks or optical disks.

[0136] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text segmentation method based on machine learning, characterized in that: include: Obtain various texts to create a text library; Performing preprocessing operations on the text data in the text library; The preprocessing operation includes dividing the text into a plurality of short sentences; Taking all the continuous texts in the text library as training samples, a training set is constructed; Inputting the training set into a machine learning model to construct a training set model; Inputting the preset text into the training set model for training to obtain the sentence segmentation text; All the continuous texts in the text library are used as training samples to construct a training set, including: According to the preset rules, each punctuation mark corresponds to a non-zero number; Adding a padding character before the training sample; Each character in the training sample, including the filler character, is assigned a number to form a training set; the number corresponding to the character without punctuation is 0, and the number corresponding to the character with punctuation is selected according to the preset rules; Input the preset text into the training set model for training to obtain the sentence segmentation text, including: Inputting the preset text into the training set model for training to obtain a trained digital string; Extracting non-zero digits from the digital string to obtain a non-zero digital string; According to a preset rule, the non-zero digital string is mapped to various punctuation marks; Bring the punctuation marks into the preset text to obtain the sentence-breaking text; Before the punctuation marks are introduced into the preset text to obtain the sentence-segmented text, the method further includes: Calibrate each punctuation mark by using the punctuation marks and spaces stored in a preset storage space; Wherein, calibrating each punctuation mark by using the punctuation marks and spaces stored in a preset storage space includes: Comparing each punctuation mark with punctuation marks and spaces stored in a preset storage space; Pick out the punctuation marks that are different after comparison; Selecting a punctuation mark to be determined according to the text content of the preset text; The same punctuation marks and the undetermined punctuation marks are put into the preset text to obtain the segmented text.

2. A text segmentation method based on machine learning according to claim 1, characterized in that: Before inputting the preset text into the training set model for training to obtain the segmented text, the method further includes: Removing punctuation marks and spaces from the text to be segmented to obtain a punctuation-free text, and storing the punctuation marks and spaces in a preset storage space; A fill character is added before the text without punctuation marks to obtain a preset text.

3. A text segmentation system based on machine learning, characterized in that: include: An acquisition module is used to acquire various texts to create a text library; and to perform preprocessing operations on the text data in the text library; The preprocessing operation includes dividing the text into a plurality of short sentences; A first construction module is used to construct a training set by taking all continuous texts in the text library as training samples; A second construction module is used to input the training set into a machine learning model to construct a training set model; A determination module is used to input a preset text into the training set model for training to obtain a segmented text; Wherein, all the continuous texts in the text library are used as training samples to construct a training set, including: According to the preset rules, each punctuation mark corresponds to a non-zero number; Adding a padding character before the training sample; Each character in the training sample, including the filler character, is assigned a number to form a training set; the number corresponding to the character without punctuation is 0, and the number corresponding to the character with punctuation is selected according to the preset rules; Input the preset text into the training set model for training to obtain the sentence segmentation text, including: Inputting the preset text into the training set model for training to obtain a trained digital string; Extracting non-zero digits from the digital string to obtain a non-zero digital string; According to a preset rule, the non-zero digital string is mapped to various punctuation marks; Bring the punctuation marks into the preset text to obtain the sentence-breaking text; Before the punctuation marks are introduced into the preset text to obtain the segmented text, the determination module further performs the following operations: Calibrate each punctuation mark by using the punctuation marks and spaces stored in a preset storage space; Wherein, calibrating each punctuation mark by using the punctuation marks and spaces stored in a preset storage space includes: Comparing each punctuation mark with punctuation marks and spaces stored in a preset storage space; Pick out the punctuation marks that are different after comparison; Selecting a punctuation mark to be determined according to the text content of the preset text; The same punctuation marks and the undetermined punctuation marks are put into the preset text to obtain the segmented text.

4. A text segmentation system based on machine learning according to claim 3, characterized in that: The first building block comprises: The first corresponding submodule is used to correspond a non-zero number to each punctuation mark according to a preset rule; An adding submodule, used for adding a padding character before the training sample; A submodule is constructed to correspond each character in the training sample to a number to form a training set; the number corresponding to the non-punctuation character is 0.

5. The text segmentation system based on machine learning according to claim 3, characterized in that: The system further comprises: A removal module, used for removing punctuation marks and spaces in the text to be segmented, obtaining a punctuation-free text, and storing the punctuation marks and spaces in a preset storage space; The adding module is used to add a filling character before the text without punctuation marks to obtain a preset text.

6. A computer storage medium, characterized in that: The computer storage medium stores a text segmentation method program based on machine learning, and when the text segmentation method program based on machine learning is executed by a processor, the steps of the text segmentation method based on machine learning as described in any one of claims 1-2 above are implemented.

Citation Information

Patent Citations

  • Text punctuation model establishment method, punctuation method and device and computer equipment

    CN107247706A

  • Chinese punctuation adding method based on a neural network

    CN109918666A