A method for detecting harmful information on the Internet by combining OCR and NLP models

Abstract

This invention provides a method for detecting harmful information on the Internet by combining an OCR model and an NLP model, comprising the following steps: obtaining first text data and image data from a website through a web crawler; processing the first text data using a trained first improved NLP model to obtain a first discrimination result; identifying text targets in the image data using an OCR model and obtaining second text data; processing the identified second text data using a trained second improved NLP model to obtain a second discrimination result; concatenating the first and second text data to obtain third text data; processing the third text data using a trained third improved NLP model to obtain a third discrimination result; and inputting the first, second, and third discrimination results into a trained comprehensive judgment model for comprehensive judgment to obtain a final classification result.

Description

Technical Field

[0001] This invention relates to the field of harmful website identification, and more specifically, to a method for detecting harmful information on the Internet that combines an OCR model and an NLP model. Background Technology

[0002] With the development of internet technology, web articles on the internet have gradually evolved into multi-source heterogeneous articles, which are usually composed of text, related image sequences, or video information.

[0003] The rapid development of the internet has brought about a large number of websites and their data. Due to the openness and low barriers to entry of the internet, the quality of internet content varies greatly, and there is an urgent need for efficient internet content detection methods to strengthen the supervision of internet content and clean up the internet environment.

[0004] In the early days, malicious websites could be identified by directly extracting text content from their source code using web crawlers. Analyzing this text was sufficient to identify harmful information. However, as criminals have become more aware of their methods and have made technological advancements, the amount of harmful features that can be directly extracted from the text has decreased. More text with harmful features is now presented as images, which circumvents the earlier method of directly analyzing harmful content from the source code. This has resulted in low accuracy in detecting harmful content on the internet, posing a significant challenge to creating a clean and healthy internet environment.

[0005] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for detecting harmful information on the Internet that combines OCR and NLP models.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] The first aspect provides a method for detecting harmful information on the Internet that combines an OCR model and an NLP model, including the following steps:

[0009] Obtain the initial text and image data of a website through web crawlers;

[0010] The first improved NLP model, which has been trained, is used to process the first text data to obtain the first discrimination result;

[0011] The OCR model is used to identify text targets in image data, and the text targets are then recognized to obtain second text data;

[0012] The trained second improved NLP model is used to process the identified second text data to obtain the second discrimination result;

[0013] The first text data is concatenated with the second text data to obtain the third text data. The trained third improved NLP model is then used to process the third text data to obtain the third discrimination result.

[0014] The first, second, and third discrimination results are fed into the trained comprehensive judgment model for comprehensive judgment to obtain the final classification result.

[0015] Preferably, the comprehensive evaluation model is an SVM model; the first improved NLP model includes a BERT pre-trained model, an LSTM, a fully connected layer, and a softmax layer connected in sequence; the second improved NLP model includes a PERT pre-trained model, an LSTM, a fully connected layer, and a softmax layer connected in sequence; and the third improved NLP model includes an ERNIE-Doc pre-trained model, an LSTM, a fully connected layer, and a softmax layer connected in sequence.

[0016] The proposed method for detecting harmful information on the Internet utilizes OCR technology to extract text content from webpage images. Combined with text content from the webpage source code, different improved NLP techniques are used to extract features from the two texts and their concatenated text, train a model, and use them together as a criterion for identifying harmful URLs, thereby achieving effective detection and early warning of harmful information on the Internet.

[0017] Furthermore, before processing the corresponding text data using the trained first, second, or third improved NLP model, a keyword dictionary vector for the text is constructed using a harmful information database, and then fused with the feature vector of the corresponding text data to enrich the text features.

[0018] A second aspect of this invention provides an internet harmful information detection system that combines an OCR model and an NLP model, comprising:

[0019] The data collection module is used to obtain the initial text and image data of a website through web crawlers;

[0020] The first discrimination module has a built-in first improved NLP model, which is used to process the first text data and obtain the first discrimination result;

[0021] The image-to-text conversion module has a built-in OCR module. The OCR module includes a character detection and localization model and a character recognition model. The character detection and localization model is used to identify text targets in image data, and the character recognition model is used to recognize text targets to obtain second text data.

[0022] The second discrimination module has a built-in second improved NLP model, which is used to process the second text data and obtain the second discrimination result;

[0023] The text concatenation module is used to concatenate the first text data with the second text data to obtain the third text data.

[0024] The third discrimination module has a built-in third improved NLP model, which is used to process the third text data and obtain the third discrimination result;

[0025] The comprehensive judgment model is used to make a comprehensive judgment based on the first, second and third discrimination results to obtain the final classification result.

[0026] A third aspect of the present invention provides an electronic device, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor performs the steps of the aforementioned Internet harmful information detection method.

[0027] A fourth aspect of the present invention provides a storage medium storing computer-readable instructions thereon, characterized in that, when the computer-readable instructions are executed by a processor, the processor performs the steps of the aforementioned Internet harmful information detection method.

[0028] This invention has significant substantive features and remarkable progress compared to existing technologies. Specifically, this invention uses an OCR model to perform character detection and recognition on website images. The obtained text is then analyzed using a first improved NLP model to obtain a first discrimination result. A second NLP model is used to analyze the text directly crawled from the website to obtain a second discrimination result. The first and second texts are concatenated to obtain third text data. The trained third improved NLP model is then used to process the third text data to obtain a third discrimination result. Based on the first, second, and third discrimination results, a comprehensive judgment is made to obtain the final classification result. The above-mentioned scheme makes full use of the effective information of the website, greatly improving the data utilization rate, and is suitable for the data shortage in the scenario of malicious website detection. In response to the counter-detection trend of unstable quality of directly crawlable text and unclear harmful features of malicious websites, the present invention has high adversarial power and can significantly improve the accuracy of malicious website identification. Moreover, compared with the detection and analysis methods that only use crawled text or only use website images, the present invention uses three classification results to comprehensively judge the final classification result, which greatly improves the error tolerance of the method and improves the accuracy of malicious website identification. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the method for detecting harmful internet information according to the present invention.

[0030] Figure 2 This is a diagram illustrating the training of an OCR model.

[0031] Figure 3 This is a schematic diagram illustrating the training of an improved NLP model.

[0032] Figure 4 This is a schematic diagram of the Internet harmful information detection system of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0034] This invention employs a combined OCR and NLP model for detecting harmful information on the Internet. It can fully utilize the multimodal information of images and text on websites, significantly improve the accuracy of identifying harmful URLs, combat the counter-detection trend of harmful URLs, and uncover cunning and hidden harmful URLs. It achieves an innovative breakthrough in the processing of dual-modal data of text and vision, opening up a new technical path for dual-modal analysis and detection of Internet articles.

[0035] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] This embodiment provides a method for detecting harmful information on the Internet that combines an OCR model and an NLP model, such as... Figure 1 As shown, it includes the following steps:

[0038] Obtain the initial text and image data of a website through web crawlers;

[0039] The first improved NLP model, which has been trained, is used to process the first text data to obtain a first discrimination result; preferably, the first discrimination result includes a first discrimination category and a first discrimination score.

[0040] The OCR model is used to identify text targets in image data, and the text targets are then recognized to obtain second text data;

[0041] The trained second improved NLP model is used to process the identified second text data to obtain a second discrimination result; preferably, the second discrimination result includes a second discrimination category and a second discrimination score;

[0042] The first text data is concatenated with the second text data to obtain the third text data. The third text data is then processed using a trained third improved NLP model to obtain the third discrimination result. Preferably, the third discrimination result includes a third discrimination category and a third discrimination score.

[0043] The first, second, and third discrimination results are fed into the trained comprehensive judgment model for comprehensive judgment to obtain the final classification result.

[0044] The comprehensive judgment model is an SVM model. During training, the first discrimination category, the first discrimination score, the second discrimination category, the second discrimination score, the third discrimination category, and the third discrimination score, along with their labels, are used as features and fed into the SVM model for training to obtain reasonable weight parameters, which serve as the final comprehensive judgment model.

[0045] As can be seen, the above scheme makes full use of the website's effective information, greatly improves data utilization, and is suitable for the data shortage in harmful website detection scenarios; moreover, the final classification result is based on a comprehensive judgment of three types of text, which greatly improves the error tolerance of this method and improves the accuracy of harmful website identification.

[0046] In its implementation, the OCR model employs a CTPN+CRNN+CTC network. CTPN is used for character detection and text target localization; the CRNN+CTC network is used for text target recognition to obtain the second text data. Specifically, its training steps are as follows: Figure 2 As shown, it will not be elaborated further here.

[0047] In specific implementation, the first improved NLP model includes a BERT pre-trained model, an LSTM, a fully connected Linear layer, and a softmax layer connected in sequence.

[0048] BERT, short for Bidirectional Encoder Representation from Transformers, is a pre-trained language representation model. It emphasizes that instead of using traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training, it employs a new masked language model (MLM) to generate deep bidirectional language representations.

[0049] In practice, the second improved NLP model includes a PERT pre-trained model, an LSTM, a fully connected Linear layer, and a softmax layer connected in sequence.

[0050] Among them, PERT, short for PRE-TRAINING BERT WITH PERMUTED LANGUAGE MODEL, is a pre-trained model based on a scrambled language model. The network architecture used by PERT is a multi-layer Transformer structure with an encoder layer. It replaces the masked word prediction task of the MLM language model with the word order prediction task. That is, it learns the semantic information of the text in a self-supervised manner without introducing a mask. It randomly scrambles part of the word order of a text and then predicts the original position of the scrambled words.

[0051] In practice, the third improved NLP model includes a pre-trained ERNIE-Doc model, an LSTM, a fully connected Linear layer, and a softmax layer connected in sequence.

[0052] ERNIE-Doc is a pre-training-fine-tuning framework for modeling long texts at the chapter level. Inspired by the human reading method of skimming first and then reading carefully, it proposes a retrospective modeling mechanism and an enhanced memory mechanism, breaking through the modeling bottleneck of Transformer with text length. ERNIE-Doc has achieved bidirectional modeling of infinitely long texts at the full chapter level for the first time in the industry, and has achieved state-of-the-art (SOTA) results on 13 authoritative Chinese and English long text language understanding tasks, including reading comprehension, information extraction, chapter classification, and language modeling.

[0053] It is understood that in this invention, the first text data from the webpage screenshot is relatively standardized and has a relatively ideal semantic structure, making it suitable for modeling with the BERT model; while the second text data from the webpage source code has the characteristic of being out of order. Therefore, this embodiment uses the PERT model to obtain the vector representation of the second text data, which can more effectively extract text features; and since the harmful classification is extremely imbalanced, while the other classification samples are relatively few, the first and second text data are concatenated to form the third text data as data augmentation. Since the third text data is very long, the ERNIE-Doc model is used to effectively extract the feature information of the third text data.

[0054] To address the issue of high repetition in harmful URL text derived from webpage source code, a bag-of-words model is constructed before feeding the first, second, and third text data into the corresponding improved NLP model. Using the bag-of-words model, word vectors are created for each text corpus. Principal component analysis is then performed on each text corpus based on these word vectors, thereby retaining words and phrases with key components in the text corpus. This achieves the goal of removing useless repetitive words and reducing redundant information.

[0055] Furthermore, addressing the issue that BERT and PERT models struggle to model excessively long malicious URL text, this embodiment employs a slicing + pooling approach to extract the sentence content of the malicious URL text. Taking the PERT model as an example, a malicious URL text is first sliced ​​into N segments using a 510 sliding window, and these segments are fed into a batch of input to the PERT model. After the sliced ​​data is fed into PERT, the [CLS] output of the PERT model is taken, at which point the output dimension is (N, 768). This vector is then polled to obtain a 1-dimensional vector, i.e., (1, 768). Specifically, the polling operation can be Max-Pooling, Mean-Pooling, or Max-Pooling and Mean-Pooling respectively, followed by concatenation before being fed into a fully connected network layer (FC).

[0056] Since the Pooling method incorporates all sequences into the model, the above approach takes into account the global information of the text and performs well for tasks with long and truncation-sensitive text.

[0057] The LSTM model is a variant of the Recurrent Neural Network (RNN) used to process time-series data and other tasks requiring memory mechanisms, such as speech recognition and text classification. Compared to traditional RNN models, LSTM introduces a gating mechanism to control the information passing and forgetting within the network. LSTM includes three gating mechanisms: input gate, output gate, and forget gate, as well as an internal cell state, making it unaffected by gradient vanishing and gradient exploding, thus enabling it to better learn long sequences and handle data with irregular time intervals.

[0058] A fully connected Linear layer is a simple fully connected network used to project the vector output by the LSTM onto a larger Logits vector. Assuming the output vocabulary is 10,000, the output Logits vector will also be 10,000, with each number in the vector corresponding to a single word.

[0059] Softmax layer: Used to transform the output of the fully connected Linear layer into probabilities between [0, 1], the sum of which is 1.

[0060] It's important to note that the training process for the first, second, and third improved NLP models is largely the same as that for the OCR model. The key difference is that the training of these three models utilizes NLP pre-model transfer learning. During training, the pre-trained NLP model is fine-tuned based on the task. Specifically, backpropagation is performed based on the defined model to adjust network hyperparameters, optimize the model network, and increase and optimize training data to obtain the final NLP pre-model. For a detailed training process, please refer to [link to relevant documentation]. Figure 3 It is important to note that, unlike the pre-training phase, the PERT model uses the normal input sequence during the fine-tuning phase, rather than a shuffled sequence.

[0061] Furthermore, before processing the corresponding text data using the trained first, second, or third improved NLP model, a keyword dictionary vector for the text is constructed using a harmful information database, and then fused with the feature vector of the corresponding text data to enrich the text features. Specifically, a harmful information database is constructed by annotating and extracting keywords from a massive amount of harmful website content, as well as accumulating prior keywords. This harmful information database contains keyword information from various harmful categories. Before feeding the text data into the corresponding improved NLP model, a keyword dictionary vector for the text data is constructed using the harmful information database, and then further fused with the feature vector of the text data. The fused text vector is then used as the input to the improved NLP model.

[0062] Example 2

[0063] This embodiment provides an internet harmful information detection system that combines an OCR model and an NLP model, such as... Figure 4 As shown, it includes:

[0064] The data collection module is used to obtain the initial text and image data of a website through web crawlers;

[0065] The first discrimination module has a built-in first improved NLP model, which is used to process the first text data and obtain the first discrimination result;

[0066] The image-to-text conversion module has a built-in OCR module. The OCR module includes a character detection and localization model and a character recognition model. The character detection and localization model is used to identify text targets in image data, and the character recognition model is used to recognize text targets to obtain second text data.

[0067] The second discrimination module has a built-in second improved NLP model, which is used to process the second text data and obtain the second discrimination result;

[0068] The text concatenation module is used to concatenate the first text data with the second text data to obtain the third text data.

[0069] The third discrimination module has a built-in third improved NLP model, which is used to process the third text data and obtain the third discrimination result;

[0070] The comprehensive judgment model is used to make a comprehensive judgment based on the first, second and third discrimination results to obtain the final classification result.

[0071] The comprehensive evaluation model is an SVM model; the first improved NLP model includes a BERT pre-trained model, an LSTM, a fully connected layer, and a softmax layer connected in sequence; the second improved NLP model includes a PERT pre-trained model, an LSTM, a fully connected layer, and a softmax layer connected in sequence; and the third improved NLP model includes an ERNIE-Doc pre-trained model, an LSTM, a fully connected layer, and a softmax layer connected in sequence.

[0072] The first discrimination module and the second discrimination module both include a slicing model. The slicing model uses a sliding window to slice the first text data or the second text data into N segments and puts them into a batch input to the first improved NLP model or the second improved NLP model.

[0073] Furthermore, the Internet harmful information detection system also includes a harmful information database, which includes keyword information from various harmful categories; the harmful information database is used to construct a keyword dictionary vector for the text, and is fused with the feature vectors of the first text data, the second text data, and the third text data as input to the corresponding improved NLP model.

[0074] Example 3

[0075] This embodiment provides an electronic device, including: a processor; and a memory storing a computer program. When the computer program is executed by the processor, the processor performs the steps of the Internet harmful information detection method described in Embodiment 1.

[0076] Example 4

[0077] This embodiment provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by a processor, the processor performs the steps of the Internet harmful information detection method described in Embodiment 1.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for detecting Internet harmful information by combining an OCR model and an NLP model, characterized in that, The method comprises the following steps: obtaining first text data and picture data of a website through a web crawler; processing the first text data by using a trained first improved NLP model to obtain a first discrimination result; recognizing a character target in the picture data by using an OCR model, and identifying the character target to obtain second text data; processing the recognized second text data by using a trained second improved NLP model to obtain a second discrimination result; splicing the first text data and the second text data to obtain third text data, and processing the third text data by using a trained third improved NLP model to obtain a third discrimination result; sending the first discrimination result, the second discrimination result and the third discrimination result into a trained comprehensive judgment model for comprehensive judgment to obtain a final classification result; the first improved NLP model comprises a BERT pre-training model, an LSTM, a full connection layer and a softmax layer connected in sequence; the second improved NLP model comprises a PERT pre-training model, an LSTM, a full connection layer and a softmax layer connected in sequence; the third improved NLP model comprises an ERNIE-Doc pre-training model, a full connection layer and a softmax layer connected in sequence; and the comprehensive judgment model is an SVM model. 2.The internet harmful information detection method combining OCR model and NLP model according to claim 1, characterized in that, Before processing the corresponding text data by using the trained first improved NLP model, the second improved NLP model or the third improved NLP model, a keyword dictionary vector of the text is constructed by using a harmful information library, and is fused with a feature vector of the corresponding text data to enrich the text features. 3.The method of claim 2, wherein the method further comprises: Before processing the corresponding text data by using the trained first improved NLP model or the second improved NLP model, the corresponding text data is sliced into N segments by using a sliding window, and is put into a batch input and sent into the first improved NLP model or the second improved NLP model. 4.The method of claim 2, wherein, Before sending the first text data, the second text data and the third text data into the corresponding improved NLP model, a bag-of-words model is constructed, a word vector of each text corpus is established by using the bag-of-words model, and principal component analysis is performed on each text corpus based on the word vector.

5. An Internet harmful information detection system combining an OCR model and an NLP model, characterized by, The method comprises the following steps: a data collection module for obtaining first text data and picture data of a website through a web crawler; a first discrimination module having a first improved NLP model built therein, for processing the first text data to obtain a first discrimination result; a picture text conversion module having an OCR module built therein, wherein the OCR module comprises a character detection positioning model and a character recognition model, the character detection positioning model is used for recognizing a character target in the picture data, and the character recognition model is used for identifying the character target to obtain second text data; a second discrimination module having a second improved NLP model built therein, for processing the second text data to obtain a second discrimination result; a text splicing module for splicing the first text data and the second text data to obtain third text data a third discrimination module having a third improved NLP model built therein, for processing the third text data to obtain a third discrimination result; The comprehensive research and judgment model is used for comprehensive research and judgment based on the first judgment result, the second judgment result and the third judgment result to obtain a final classification result. The comprehensive research and judgment model is an SVM model. The first improved NLP model comprises a BERT pre-training model, an LSTM, a full connection layer and a softmax layer connected in sequence; the second improved NLP model comprises a PERT pre-training model, an LSTM, a full connection layer and a softmax layer connected in sequence; and the third improved NLP model comprises an ERNIE-Doc pre-training model, an LSTM, a full connection layer and a softmax layer connected in sequence. 6.The Internet harmful information detection system of claim 5, wherein, Further comprising a harmful information library comprising keyword information in each harmful category; a keyword dictionary vector of the text is constructed by using the harmful information library, and is fused with the feature vectors of the first text data, the second text data and the third text data as the input of the corresponding improved NLP model.

7. An electronic device, comprising: A processor; A memory storing a computer program, characterized in that when the computer program is executed by the processor, the processor executes the steps of the internet harmful information detection method according to any one of claims 1-4.

8. A storage medium having stored thereon computer readable instructions, characterized in that, When the computer readable instructions are executed by the processor, the processor executes the steps of the internet harmful information detection method according to any one of claims 1-4.

Images