AI-based Internet content detection method and system

Abstract

The present invention provides an AI-based Internet content detection method and system, which uses different feature extraction methods for image types or text types by identifying the types of different data packets. Dimensionality reduction sampling and sliding window sampling are used for image types to obtain image features containing high-dimensional local features. Sentence segmentation and clustering are used for text types to obtain sequence features after reorganization of sentences. After fusing image features and sequence features according to certain rules, a feature matrix is ​​obtained, and finally a random forest method is used to obtain the classification result. Through the above processing process, the problem that the existing single convolutional neural network or recurrent neural network is not sufficient to cope with complex network environments is overcome.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to an AI-based Internet content detection method and system. Background Art

[0002] Existing content detection methods mostly use convolutional neural networks or recurrent neural networks. Convolutional neural networks can extract image features, reduce their dimensionality using pooling layers, and perform nonlinear transformations on them using flattening and densely connected layers to achieve classification results. Recurrent neural networks are used to extract features from training data sequences. By traversing all elements of the data, they generate data of specific dimensions to represent the sequence characteristics.

[0003] However, the Internet is complex, including both image and text data. A single convolutional neural network or recurrent neural network is not sufficient to cope with this complex network environment.

[0004] Therefore, there is an urgent need for a targeted AI-based Internet content detection method and system. Summary of the Invention

[0005] The purpose of the present invention is to provide an AI-based Internet content detection method and system to solve the problem that existing single convolutional neural networks or recurrent neural networks are not sufficient to cope with complex network environments.

[0006] In a first aspect, the present application provides an AI-based Internet content detection method, the method comprising:

[0007] Collecting different types of collected data packets from the Internet network, and identifying the type of the data packet as an image type or a text type based on metadata carried by the collected data packet;

[0008] When the image type is identified, the data stream of the collected data packet is discretized and sampled according to time domain continuity to obtain a discrete data stream after dimensionality reduction, which is converted into a grayscale image; the first image feature of the grayscale image is extracted, vectorized and input into N layers of convolution units, and a first intermediate result after convolution is output; the width of the sliding window is determined based on the eigenvalue distribution of the first intermediate result, and the data stream of the collected data packet is sampled again using the sliding window to directly extract the second image feature therefrom;

[0009] When the text type is identified, sentences are extracted from the stream sequence of the collected data packet, and the sentences are input into the syntactic model for preliminary segmentation to obtain the first word components. According to the pre-set mapping relationship between phrase type and weight value, all the first word components after preliminary segmentation are analyzed, and the first word components with weight values ​​greater than the threshold are clustered to form a new sentence, from which sequence features are extracted.

[0010] The second image feature and the sequence feature are fused and fed into a convolutional layer of a recognition model. Local feature components are selected using sliding windows of different sizes, and the local feature components are concatenated to obtain a first feature matrix. The feature matrix is ​​fed into a pooling layer of the recognition model. The pooling layer selects valid eigenvalues ​​by selecting a pooling function, and the results are concatenated again to obtain a second feature matrix.

[0011] The second feature matrix is ​​fed into the random forest of the recognition model for classification. The random forest extracts the second feature matrix in n rounds to obtain n training sets. The extracted n training sets are used to train n decision trees by randomly sampling columns and using a specified number of feature values. The n decision trees obtain a classification result by voting.

[0012] The collected data packets are managed according to the classification result.

[0013] In a second aspect, the present application provides an AI-based Internet content detection system, the system comprising:

[0014] The collection and identification unit is used to collect different types of collected data packets from the Internet network and identify the type of the data packet as an image type or a text type based on the metadata carried by the collected data packet;

[0015] An image feature extraction unit is configured to, when the image type is identified, discretize the data stream of the collected data packet, sample it according to time domain continuity, obtain a discrete data stream after dimensionality reduction, and convert it into a grayscale image; extract a first image feature of the grayscale image, vectorize it, and input it into an N-layer convolution unit, outputting a first intermediate result after convolution; determine the width of a sliding window based on the eigenvalue distribution of the first intermediate result, use the sliding window to sample the data stream of the collected data packet again, and directly extract a second image feature therefrom;

[0016] a sequence feature extraction unit configured to extract sentences from the stream sequence of the collected data packet when the data packet is identified as a text type, input the sentences into a syntactic model, perform preliminary sentence segmentation, and obtain first word components; analyze all first word components after preliminary sentence segmentation according to a preset mapping relationship between phrase type and weight value, cluster first word components with weight values ​​greater than a threshold, form new sentences, and extract sequence features therefrom;

[0017] a fusion unit, configured to fuse the second image features and the sequence features, and feed the results into a convolutional layer of a recognition model; select local feature components using sliding windows of different sizes; concatenate the local feature components to obtain a first feature matrix; and feed the feature matrix into a pooling layer of the recognition model; the pooling layer selects valid eigenvalues ​​by selecting a pooling function, and concatenates the results again to obtain a second feature matrix;

[0018] a classification unit, configured to input the second feature matrix into a random forest of the recognition model for classification, wherein the random forest performs n rounds of extraction on the second feature matrix to obtain n training sets, and uses the extracted n training sets to randomly train n decision trees using a specified number of feature values ​​by column sampling, and the n decision trees obtain a classification result by voting;

[0019] A management unit is used to manage the collected data packets according to the classification result.

[0020] In a third aspect, the present application provides an AI-based Internet content detection system, the system comprising a processor and a memory:

[0021] The memory is used to store program code and transmit the program code to the processor;

[0022] The processor is configured to execute any one of the four possible methods of the first aspect according to instructions in the program code.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute any one of the four possible methods in the first aspect.

[0024] Beneficial effects

[0025] The present invention provides an AI-based Internet content detection method and system, which uses different feature extraction methods for image types or text types by identifying the types of different data packets. Dimensionality reduction sampling and sliding window sampling are used for image types to obtain image features containing high-dimensional local features. Sentence segmentation and clustering are used for text types to obtain sequence features after reorganization of sentences. After fusing image features and sequence features according to certain rules, a feature matrix is ​​obtained, and finally a random forest method is used to obtain the classification result. Through the above processing process, the problem that the existing single convolutional neural network or recurrent neural network is not sufficient to cope with complex network environments is overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 This is a general flow chart of the AI-based Internet content detection method of the present invention;

[0028] Figure 2 This is an architectural diagram of the AI-based Internet content detection system of the present invention. DETAILED DESCRIPTION

[0029] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0030] Figure 1 This is a general flow chart of the AI-based Internet content detection method provided in this application, which includes:

[0031] Collecting different types of collected data packets from the Internet network, and identifying the type of the data packet as an image type or a text type based on metadata carried by the collected data packet;

[0032] When the image type is identified, the data stream of the collected data packet is discretized and sampled according to time domain continuity to obtain a discrete data stream after dimensionality reduction, which is converted into a grayscale image; the first image feature of the grayscale image is extracted, vectorized and input into N layers of convolution units, and a first intermediate result after convolution is output; the width of the sliding window is determined based on the eigenvalue distribution of the first intermediate result, and the data stream of the collected data packet is sampled again using the sliding window to directly extract the second image feature therefrom;

[0033] When the text type is identified, sentences are extracted from the stream sequence of the collected data packet, and the sentences are input into the syntactic model for preliminary segmentation to obtain the first word components. According to the pre-set mapping relationship between phrase type and weight value, all the first word components after preliminary segmentation are analyzed, and the first word components with weight values ​​greater than the threshold are clustered to form a new sentence, from which sequence features are extracted.

[0034] The second image feature and the sequence feature are fused and fed into a convolutional layer of a recognition model. Local feature components are selected using sliding windows of different sizes, and the local feature components are concatenated to obtain a first feature matrix. The feature matrix is ​​fed into a pooling layer of the recognition model. The pooling layer selects valid eigenvalues ​​by selecting a pooling function, and the results are concatenated again to obtain a second feature matrix.

[0035] The second feature matrix is ​​fed into the random forest of the recognition model for classification. The random forest extracts the second feature matrix in n rounds to obtain n training sets. The extracted n training sets are used to train n decision trees by randomly sampling columns and using a specified number of feature values. The n decision trees obtain a classification result by voting.

[0036] The collected data packets are managed according to the classification result.

[0037] In some preferred embodiments, the recognition model is trained by minimizing the entropy loss function through back propagation to avoid oversaturation. When the accuracy of the recognition model meets the threshold requirement, the recognition model training is completed and can then be used for data verification.

[0038] In some preferred embodiments, the fusion of the second image features and the sequence features includes: extracting the second image features one by one in rows or columns and writing them into a single-dimensional matrix; extracting the sequence features one by one and writing them into another single-dimensional matrix; and weighting or accumulating the two single-dimensional matrices to obtain a fused feature matrix.

[0039] Each decision tree has a targeted classification capability. The specified eigenvalues ​​are derived based on different classifications. By using the same eigenvector matrix to classify from different perspectives through a decision tree, the integration of different classification capabilities is achieved. This results in higher classification performance than a single classifier.

[0040] The average generalization error of a decision tree in a random forest is related to the regression function.

[0041] In some preferred embodiments, the voting method includes weighted accumulation of the output results of each decision tree.

[0042] Figure 2 This is an architectural diagram of the AI-based Internet content detection system provided in this application, which includes:

[0043] The collection and identification unit is used to collect different types of collected data packets from the Internet network and identify the type of the data packet as an image type or a text type based on the metadata carried by the collected data packet;

[0044] An image feature extraction unit is configured to, when the image type is identified, discretize the data stream of the collected data packet, sample it according to time domain continuity, obtain a discrete data stream after dimensionality reduction, and convert it into a grayscale image; extract a first image feature of the grayscale image, vectorize it, and input it into an N-layer convolution unit, outputting a first intermediate result after convolution; determine the width of a sliding window based on the eigenvalue distribution of the first intermediate result, use the sliding window to sample the data stream of the collected data packet again, and directly extract a second image feature therefrom;

[0045] a sequence feature extraction unit configured to extract sentences from the stream sequence of the collected data packet when the data packet is identified as a text type, input the sentences into a syntactic model, perform preliminary sentence segmentation, and obtain first word components; analyze all first word components after preliminary sentence segmentation according to a preset mapping relationship between phrase type and weight value, cluster first word components with weight values ​​greater than a threshold, form new sentences, and extract sequence features therefrom;

[0046] a fusion unit, configured to fuse the second image features and the sequence features, and feed the results into a convolutional layer of a recognition model; select local feature components using sliding windows of different sizes; concatenate the local feature components to obtain a first feature matrix; and feed the feature matrix into a pooling layer of the recognition model; the pooling layer selects valid eigenvalues ​​by selecting a pooling function, and concatenates the results again to obtain a second feature matrix;

[0047] a classification unit, configured to input the second feature matrix into a random forest of the recognition model for classification, wherein the random forest performs n rounds of extraction on the second feature matrix to obtain n training sets, and uses the extracted n training sets to randomly train n decision trees using a specified number of feature values ​​by column sampling, and the n decision trees obtain a classification result by voting;

[0048] A management unit is used to manage the collected data packets according to the classification result.

[0049] The present application provides an AI-based Internet content detection system, the system comprising: the system comprising a processor and a memory:

[0050] The memory is used to store program code and transmit the program code to the processor;

[0051] The processor is configured to execute the method described in any one of all embodiments of the first aspect according to instructions in the program code.

[0052] The present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the method described in any one of the embodiments of the first aspect.

[0053] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program that, when executed, may include some or all of the steps of various embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0054] Those skilled in the art will clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention or certain portions of the embodiments.

[0055] In particular, for the embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0056] The above-described embodiments of the present invention do not limit the protection scope of the present invention.

Claims

1. An AI-based Internet content detection method, characterized in that: The method comprises: Collecting different types of collected data packets from the Internet network, and identifying the type of the data packet as an image type or a text type based on metadata carried by the collected data packet; When the image type is identified, the data stream of the collected data packet is discretized and sampled according to time domain continuity to obtain a discrete data stream after dimensionality reduction, which is converted into a grayscale image; the first image feature of the grayscale image is extracted, vectorized and input into N layers of convolution units, and a first intermediate result after convolution is output; the width of the sliding window is determined based on the eigenvalue distribution of the first intermediate result, and the data stream of the collected data packet is sampled again using the sliding window to directly extract the second image feature therefrom; When the text type is identified, sentences are extracted from the stream sequence of the collected data packet, and the sentences are input into the syntactic model for preliminary segmentation to obtain the first word components. According to the pre-set mapping relationship between phrase type and weight value, all the first word components after preliminary segmentation are analyzed, and the first word components with weight values ​​greater than the threshold are clustered to form a new sentence, from which sequence features are extracted. The second image feature and the sequence feature are fused and fed into a convolutional layer of a recognition model. Local feature components are selected using sliding windows of different sizes, and the local feature components are concatenated to obtain a first feature matrix. The feature matrix is ​​fed into a pooling layer of the recognition model. The pooling layer selects valid eigenvalues ​​by selecting a pooling function, and the results are concatenated again to obtain a second feature matrix. The second feature matrix is ​​fed into the random forest of the recognition model for classification. The random forest extracts the second feature matrix in n rounds to obtain n training sets. The extracted n training sets are used to train n decision trees by randomly sampling columns and using a specified number of feature values. The n decision trees obtain a classification result by voting. The collected data packets are managed according to the classification result.

2. The method according to claim 1, wherein: When the recognition model is trained, the entropy loss function is minimized by back propagation to avoid oversaturation. When the accuracy of the recognition model meets the threshold requirement, it indicates that the recognition model training is completed.

3. The method according to claim 1, wherein: The fusing of the second image features and the sequence features includes: extracting the second image features one by one in rows or columns and writing them into a single-dimensional matrix; extracting the sequence features one by one and writing them into another single-dimensional matrix; and weighting or accumulating the two single-dimensional matrices to obtain a fused feature matrix.

4. The method according to any one of claims 2 or 3, characterized in that: The voting method includes weighted accumulation of the output results of each decision tree.

5. An AI-based Internet content detection system, characterized in that: The system comprises: The collection and identification unit is used to collect different types of collected data packets from the Internet network and identify the type of the data packet as an image type or a text type based on the metadata carried by the collected data packet; An image feature extraction unit is configured to, when the image type is identified, discretize the data stream of the collected data packet, sample it according to time domain continuity, obtain a discrete data stream after dimensionality reduction, and convert it into a grayscale image; extract a first image feature of the grayscale image, vectorize it, and input it into an N-layer convolution unit, outputting a first intermediate result after convolution; determine the width of a sliding window based on the eigenvalue distribution of the first intermediate result, use the sliding window to sample the data stream of the collected data packet again, and directly extract a second image feature therefrom; a sequence feature extraction unit configured to extract sentences from the stream sequence of the collected data packet when the data packet is identified as a text type, input the sentences into a syntactic model, perform preliminary sentence segmentation, and obtain first word components; analyze all first word components after preliminary sentence segmentation according to a preset mapping relationship between phrase type and weight value, cluster first word components with weight values ​​greater than a threshold, form new sentences, and extract sequence features therefrom; a fusion unit, configured to fuse the second image features and the sequence features, and feed the results into a convolutional layer of a recognition model; select local feature components using sliding windows of different sizes; concatenate the local feature components to obtain a first feature matrix; and feed the feature matrix into a pooling layer of the recognition model; the pooling layer selects valid eigenvalues ​​by selecting a pooling function, and concatenates the results again to obtain a second feature matrix; a classification unit, configured to input the second feature matrix into a random forest of the recognition model for classification, wherein the random forest performs n rounds of extraction on the second feature matrix to obtain n training sets, and uses the extracted n training sets to randomly train n decision trees using a specified number of feature values ​​by column sampling, and the n decision trees obtain a classification result by voting; A management unit is used to manage the collected data packets according to the classification result.

6. An AI-based Internet content detection system, characterized in that: The system includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to implement the method according to any one of claims 1 to 4 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the method according to any one of claims 1 to 4.

Images