Text processing method and device, readable storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a text processing method and device, a readable storage medium and electronic equipment. According to the embodiment of the invention, the text processing method includes: analyzing the data through at least one processor; obtaining at least one first text, segmenting the at least one first text, determining a second text set, obtaining feature informationof the second text through the at least one processor, determining evaluation information according to the feature information of the second text, and determining at least one segmentation keyword inthe at least one second text according to the evaluation information; determining a final segmentation mode of the first text and a final segmentation keyword corresponding to the final segmentationmode; and training a word segmentation model according to the feature information, the final segmentation mode and the final segmentation keyword. Through the word segmentation model trained by the method, the text can be automatically and accurately segmented, the consumption of human resources is reduced, and the accuracy of text word segmentation is improved.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a text processing method, device, readable storage medium and electronic equipment. Background technique [0002] With the advancement of science and technology, the application of natural language processing is becoming more and more extensive. Text segmentation is the basis of other processing in natural language processing, so the accuracy of text segmentation is very important. [0003] In the prior art, word segmentation methods based on dictionaries or word segmentation models based on statistics / deep learning are usually used for text segmentation. Specifically, word segmentation methods based on dictionaries require a large number of manual maintenance dictionaries, and word segmentation models based on statistics / deep learning require a large number of manual annotations. Both of the above two methods waste a lot of human resources, and the error rate is high during manual...

AI Technical Summary

Problems solved by technology

[0003] In the prior art, word segmentation methods based on dictionaries or word segmentation models based on statistics / deep learning are usually used for text segmentation. Specifically, word segmentation methods based on dictionaries require a large number of manual maintenance dictionaries, and word segmentation models based on statistics / deep learning require a large number of manual annotations. The above two methods waste a lot of human resources, and the error rate of manual maintenance or labeling is high, which affects the accuracy of text segmentation

Images