Text processing method and device, model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a text processing method and device, a model training method and device, equipment and a storage medium, and the method comprises the steps: firstly obtaining a to-be-detected text containing a plurality of statements, and then extracting statement feature vectors corresponding to the plurality of statements; the type of each statement is determined according to the statement feature vector corresponding to the statement, and then the type of the whole to-be-detected text is determined according to the type of each statement. It can be seen that classification of different levels is achieved in the classification process, that is, whether the whole text relates to the sensitive content or not is determined, meanwhile, which statements in the text relate to the sensitive content can be specifically determined, and therefore the classification granularity of the text is refined. In the practical application, the classification results of the statement level and the document level can be output to the user, so that the user can process the whole text or some statements in the text according to the classification results of different levels, and the effect of refining the processing granularity of the long text data is also achieved.

Description

technical field [0001] The present invention relates to the field of artificial intelligence, and in particular to text processing methods, model training methods, devices, equipment and storage media Background technique [0002] For the massive text data existing on the Internet, there are often sensitive contents such as pornography, violence, gambling, and poisoning. In order to limit the widespread dissemination of such sensitive content, it needs to be accurately identified and dealt with accordingly. [0003] In the prior art, for long text data with obscure description and difficult semantic understanding, it is often only possible to roughly identify whether the long text data as a whole is a sensitive type, and cannot identify the specific location of sensitive content in the long text data, which makes the long text data The classification granularity is relatively rough, which also makes the processing granularity of long text data relatively rough. SUMMARY OF...

AI Technical Summary

Problems solved by technology

[0003] In the prior art, for long text data with ambiguous description and difficult semantic understanding, it is often only possible to roughly identify whether the long text data as a whole is a sensitive type, and cannot identify the specific location of the sensitive content in the long text data, making the long text data The granularity of the classification is relatively rough, which also makes the processing granularity of long text data relatively rough

Images