NLP text security auditing multi-level retrieval system

Abstract

The invention discloses an NLP text security auditing multi-level retrieval system, which utilizes a data structure of a compressed prefix tree to store and search data, is high in query speed, reduces memory occupation by more than two thousands of times compared with a dictionary tree data structure, and improves the retrieval efficiency. The keyword matching sub-module, the sentence similarity matching sub-module and the text classification deep learning sub-module form a hierarchical search structure of three-level search, the query accuracy is high, search of dominant sensitive words can be covered, meanwhile, text content security auditing can be conducted semantically, the accuracy, the error-tolerant rate and the coverage rate are guaranteed, and the search efficiency is improved. The problems that an existing NLP text security auditing system uses a data structure of a Trie tree for storage, the occupied storage space is large, the memory cost of a server is increased, large-scale deployment on the same server is inconvenient, the performance is difficult to achieve optimization, the model generalization ability is limited, and the system reliability is poor are solved. And the prediction accuracy is unstable.

Description

technical field [0001] The invention relates to the technical field of text security audit, in particular to an NLP text security audit multi-level retrieval system. Background technique [0002] Text content security auditing is essentially a text classification problem, that is, given a text, and then what is the security intent of the text, the security intent here is the text label. The NLP text security auditing system is mainly used for security auditing in user text chats. The auditing areas generally include advertisements, blacklists, and prohibitions. In the existing NLP text security auditing system, three combined technologies of Trie tree, sentence similarity matching and deep learning text classification model are used for hierarchical search, and customized text preprocessing technology is matched at the same time. The advantage of the data structure of Trie tree is that the query speed is very fast, but the problem is that the storage space is very large, wh...

AI Technical Summary

Problems solved by technology

[0003] The present invention provides a multi-level retrieval system for NLP text security review, which is used to solve the problem that the existing NLP text security review system uses a Trie tree data structure for storage, occupies a large storage space, increases the memory cost of the server, and is inconvenient Large-scale deployment on the same server, it is difficult to achieve optimal performance, and the generalization ability of the model is limited, and the prediction accuracy is unstable.

Images