A network information detection method and system based on big data

Abstract

The present invention discloses a network information detection method and system based on big data, and relates to the field of network information detection. The network information detection method based on big data obtains network information data to be detected, and the network information data includes text information data, file information data, network flow information data, and image information data; performs feature extraction on the network information data to be detected, and obtains text features, file features, network flow features, and image features respectively. The present invention performs multi-dimensional feature extraction on the network information data to be detected, including multiple types of features such as text, file, network flow, and image, so as to more comprehensively reflect the attributes and features of the information, so as to improve the accuracy and robustness of information detection, and through comprehensive analysis of the features of different types of information, it can more accurately judge whether the information is bad information, thereby effectively curbing the spread of bad information in the network.

Description

Technical Field

[0001] The present invention relates to the field of network information detection, and in particular to a network information detection method and system based on big data. Background Art

[0002] In today's digital age, the explosive growth of network information is increasing day by day. At the same time, bad information on the network is becoming more rampant. In response to this challenge, network information detection methods based on big data have emerged and have become an important means to effectively deal with bad information. In addition, big data technology provides powerful data processing and analysis capabilities, allowing massive amounts of network information to be efficiently collected, stored, processed and analyzed. Through big data technology, we can monitor the information flow on the network in real time, identify and filter out bad content, thereby ensuring the clarity and security of cyberspace.

[0003] Prior art, such as the bad information detection method and server disclosed in the invention patent application with announcement number: CN102880636A, includes: the server receives the information release request sent by the client; the information carried by the information release request is matched with the preset filtering word library by word segmentation, and the relevance is calculated; when the relevance is greater than the preset threshold, the information is filtered. The present invention maintains a filtering word library, performs word segmentation matching on each piece of information, calculates the relevance of the information with the sensitive words in the filtering word library, and compares the relevance with the preset threshold. When the relevance is greater than the preset threshold, the information is filtered and marked accordingly in the database, so that the information is only visible to the publisher himself, but not to others. This effectively prevents the spread of bad information, enhances the maintainability and good interactivity of the website; in addition, due to the reduction of manual participation, the efficiency of website information management is improved.

[0004] With respect to the above technical solution, it is found that the current limitation of network information detection is that the current network information detection method only detects a single type of information, but network information contains a lot of information, so the network information detection method is difficult to process multiple types of network information data. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a network information detection method and system based on big data, which solves the problem that the current network information detection method only detects a single type of information, but network information contains a lot of information, so the network information detection method is difficult to process multiple types of network information data.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a network information detection method based on big data, obtaining network information data to be detected, wherein the network information data includes text information data, file information data, network traffic information data, and image information data; performing feature extraction on the network information data to be detected, and obtaining text features, file features, network traffic features, and image features respectively; performing analysis and processing based on the text features, file features, network traffic features, and image features respectively, and judging whether the text information data, file information data, and network traffic information data are bad information according to the analysis and processing results; and taking corresponding processing measures for the corresponding information data judged as bad information.

[0007] Furthermore, the text feature specifically refers to all word information data in the text information data, and the specific process of determining whether the text information data is bad information is as follows: read the text information data and divide it; remove stop words from the divided text information data; perform word form restoration on the text information data after the stop words removal process to obtain all word information in the text information data; match all word information in the text information data with a bad word library to obtain the number of bad words in the text information data; calculate the text bad coefficient of the text information data based on all word information of the text information data and the number of bad words, and compare the text bad coefficient of the text information data with a preset text bad coefficient threshold; if the bad coefficient of the text information data is higher than the preset bad coefficient threshold, the text information data is regarded as bad information.

[0008] Furthermore, the specific calculation formula of the defect coefficient is as follows: ;in, is the bad coefficient of text information data, The number of bad words obtained after matching the bad word library in the text information data. is the total number of words in the text information data, is a matching correction factor; the specific steps for calculating the matching correction factor are: obtaining word information data of several text information data, and matching the word information data of several text information data with the bad word library respectively, and obtaining the measured quantity value of the bad words of each text information data respectively; at the same time, obtaining the actual quantity value of the bad words of each text information data after matching the word information data of each text information data with the bad word library respectively; based on the measured quantity value and the actual quantity value of the bad words of each text information data and combined with the moving exponential average method, a comprehensive analysis is performed to obtain the matching correction factor.

[0009] Furthermore, the file feature is specifically a file type in the file information data, and the file types include text files, image files, audio files and binary files. The specific process of determining whether the file information data is bad information is as follows: if the file feature is a text file, then based on the file operation function in the programming language, the content of the text file is read into the memory and marked as a string. After the file reading is completed, the file handle is closed to release resources, and the string is hashed based on the hash algorithm to obtain a text hash value, and the text hash value is matched with the bad text hash library. If the match is successful, the file information data is regarded as bad information; if the file feature is an image file, the image information in the image file is adjusted to a preset size, the adjusted image information is grayed out, the pixel mean is calculated based on the image information, and a string is constructed based on the pixel mean, which is the image. Hash value, and match the image hash value with the bad image hash library. If the match is successful, the file information data is regarded as bad information. If the file feature is an audio file, the spectrogram in the audio file is extracted, and the spectrogram is converted into an audio string. The audio string is hashed based on the hash algorithm to obtain the audio hash value, and the audio hash value is matched with the bad audio hash library. If the match is successful, the file information data is regarded as bad information. If the file feature is a binary file, based on the file operation function in the programming language, the content of the binary file is read into the memory and marked as a string. After the file reading is completed, the file handle is closed to release resources. The string is hashed based on the hash algorithm to obtain the binary hash value, and the binary hash value is matched with the bad binary hash library. If the match is successful, the file information data is regarded as bad information.

[0010] Furthermore, the network traffic characteristics specifically include the number of bytes of data packets in the network traffic information data, the data transmission rate, the source IP address and target IP address of the data packets, and the source port number and target port number of the data packets. The specific process of judging whether the network traffic information data is bad information is as follows: normalizing based on the extracted number of bytes of the data packets, the data transmission rate, the source IP address and target IP address of the data packets, and the source port number and target port number of the data packets; performing comprehensive analysis and calculation based on the number of bytes of the data packets after normalization, the data transmission rate, the source IP address and target IP address of the data packets, and the source port number and target port number of the data packets to obtain a network traffic characteristic value; comparing the network traffic characteristic value with a preset network traffic characteristic threshold. If the network traffic characteristic value is higher than the preset network traffic characteristic threshold, the network traffic information data is regarded as bad information.

[0011] Furthermore, the image information data includes picture information data and video information data, the image features include picture features extracted based on the picture information data and video features extracted based on the video information data, the picture features specifically include the color histogram, edge features, texture features, and depth features of the picture information data, and the specific process of judging whether the picture information data is bad information is as follows: performing feature fusion on the color histogram, edge features, texture features, and depth features of the extracted picture information data to obtain a picture feature comprehensive vector; reading the picture feature comprehensive vector and inputting it into a classification model for classification, and judging whether the picture information data is bad information based on the classification result.

[0012] Furthermore, the video features specifically include color histograms, edge features, texture features, and depth features of the picture information data of each key frame in the video information data. The specific process of determining whether the video information data is bad information is as follows: reading the color histograms, edge features, texture features, and depth features of the picture information data of each key frame in the extracted video information data, and inputting them into a classification model for classification to obtain the category of the picture information data of each key frame; determining whether there is a key frame with picture information data classified as bad; if so, the video information data is deemed to be bad information.

[0013] Furthermore, corresponding processing measures are taken for the corresponding information data judged as bad information, specifically: if the bad information is text information data or image information data, the text data and image information data containing the bad information are deleted to prevent further propagation, and the text content and image content subsequently uploaded by the user are monitored in real time to promptly discover and process the bad information; if the bad information is file information data, isolation measures are taken for the files containing the bad information to prevent further propagation, and the file content subsequently uploaded by the user is monitored in real time to promptly discover and process the bad files; if the bad information is network traffic information data, network security equipment is used to prevent the propagation of bad traffic, and access control measures are implemented for the source of the bad traffic to limit its access rights to network resources.

[0014] A network information detection system based on big data comprises: a data acquisition subsystem, a feature extraction subsystem, an analysis and processing subsystem, and a treatment measure subsystem; the data acquisition subsystem is used to acquire network information data to be detected based on network monitoring software, and the network information data comprises text information data, file information data, network flow information data, and image information data; the feature extraction subsystem is used to extract features from the network information data to be detected, and obtain text features, file features, network flow features, and image features respectively; the analysis and processing subsystem is used to perform analysis and processing based on text features, file features, network flow features, and image features, and judge whether the text information data, file information data, and network flow information data are bad information according to the analysis and processing results; the treatment measure subsystem is used to take corresponding treatment measures for the corresponding information data judged as bad information.

[0015] Furthermore, the analysis and processing subsystem includes: a text analysis and processing module, a file analysis and processing module, a network traffic analysis and processing module, and an image analysis and processing module; the text analysis and processing module is used to perform analysis and processing based on text features to determine whether text information data is bad information; the file analysis and processing module is used to perform analysis and processing based on file features to determine whether file information data is bad information; the network traffic analysis and processing module is used to perform analysis and processing based on network traffic features to determine whether network traffic information data is bad information; the image analysis and processing module is used to perform analysis and processing based on image features to determine whether image information data is bad information.

[0016] The present invention has the following beneficial effects:

[0017] (1) This network information detection method based on big data extracts multi-dimensional features of the network information data to be detected, including various types of features such as text, files, network traffic and images, so as to more comprehensively reflect the attributes and characteristics of the information, thereby improving the accuracy and robustness of information detection. In addition, by comprehensively analyzing the features of different types of information, it can more accurately determine whether the information is bad information, thereby effectively curbing the spread of bad information in the network.

[0018] (2) This network information detection method based on big data can monitor in real time and take targeted treatment measures for data judged as bad information while detecting network information. Corresponding treatment strategies are adopted for different types of bad information, such as deleting text information, image information, isolating file information, blocking network traffic, etc. This real-time monitoring and targeted treatment mechanism can effectively prevent the continued spread of bad information and protect network security and user interests.

[0019] (3) This big data-based network information detection system realizes modular processing by dividing the analysis and processing subsystem into text, file, network traffic and image analysis and processing modules. Each module focuses on the analysis and processing of a specific type of information, which enhances the flexibility and scalability of the system. The modular design enables the system to dynamically adjust the functions and configurations of each module according to actual needs, and better adapt to network information detection tasks in different scenarios.

[0020] (4) This network information detection system based on big data realizes the detailed processing of tasks through the text, file, network traffic and image analysis and processing modules, each of which is responsible for the analysis and processing of corresponding types of information. This detailed processing enables the system to perform information detection more efficiently and improves the processing speed and accuracy. By decomposing the processing tasks into multiple modules for parallel processing, the system can make full use of computing resources, complete information detection tasks more quickly, and effectively respond to the processing needs of large-scale data.

[0021] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a network information detection method based on big data in the present invention.

[0023] Figure 2 The present invention is a flowchart of the specific steps of determining whether text information data is bad information in a network information detection method based on big data.

[0024] Figure 3 The present invention is a flowchart of the specific steps of determining whether file information data is bad information in a network information detection method based on big data.

[0025] Figure 4 The present invention is a flowchart of the specific steps of determining whether network traffic information data is bad information in a network information detection method based on big data.

[0026] Figure 5 The present invention is a flowchart of the specific steps of determining whether image information data is bad information in a network information detection method based on big data.

[0027] Figure 6 The present invention is a flowchart of the specific steps of determining whether video information data is bad information in a network information detection method based on big data.

[0028] Figure 7 This is a block diagram of a network information detection system based on big data in the present invention.

[0029] Figure 8This is a block diagram of an analysis and processing subsystem in a network information detection system based on big data in the present invention. DETAILED DESCRIPTION

[0030] The embodiment of the present application uses a network information detection method and system based on big data to solve the problem that the current network information detection method only detects a single type of information, but network information contains a lot of information, so the network information detection method is difficult to process multiple types of network information data.

[0031] The overall idea of ​​the problem in the embodiment of this application is as follows:

[0032] First, the network information data to be detected is obtained from the network, including text information data, file information data, network flow information data, and image information data. Then, the obtained network information data is subjected to feature extraction to obtain text features, file features, network flow features, and image features respectively. Then, based on the extracted features, the text information data, file information data, network flow information data, and image information data are analyzed and processed respectively. For the text information data, the text badness coefficient is calculated according to the vocabulary information; for the file information data, the bad information is judged according to the file type and features; for the network flow information data, the bad information is analyzed according to the characteristics of the data packet; for the image information data, the bad information is judged according to the image features. Finally, for the data judged as bad information, corresponding processing measures are taken. For the text information data and the image information data, deletion or shielding is adopted to prevent further propagation, and the subsequent uploaded content is monitored in real time; for the file information data, isolation or deletion is adopted; for the network flow information data, network security equipment is used to block it, and access control measures are implemented to limit its access rights to network resources.

[0033] See also Figure 1 The embodiment of the present invention provides a technical solution: a network information detection method based on big data, comprising the following steps: obtaining network information data to be detected based on network monitoring software, the network information data including text information data, file information data, network flow information data, and image information data; performing feature extraction on the network information data to be detected, and obtaining text features, file features, network flow features, and image features respectively; performing analysis and processing based on the text features, file features, network flow features, and image features respectively, and judging whether the text information data, file information data, and network flow information data are bad information respectively according to the analysis and processing results; and taking corresponding processing measures for the corresponding information data judged as bad information.

[0034] Specifically, Figure 2As shown, the text features are specifically all the word information data in the text information data. The specific process of determining whether the text information data is bad information is as follows: Read the text information data and divide it in the form of characters. This process can split the text into discrete words or tokens, which are used as the basic units of features; Perform stop word removal processing on the divided text information data to remove the stop words in the text. These words usually appear frequently in the language but do not carry much information, such as "de", "shi", "zai", etc. Perform lemmatization processing on the text information data after stop word removal to obtain all the word information in the text information data. Specifically, it merges different forms of words into their original forms, and lemmatization is to restore words to their basic forms in the dictionary. For example, change "was" to "be". Match all the word information in the text information data with the bad word library respectively to obtain the number of bad words in the text information data. The bad word library is specifically an information database containing all bad words; Calculate the text bad coefficient of the text information data based on all the word information and the number of bad words in the text information data, and compare the text bad coefficient of the text information data with the preset text bad coefficient threshold. The text bad coefficient is used to represent the bad degree of the text information data; If the bad coefficient of the text information data is higher than the preset bad coefficient threshold, then the text information data is regarded as bad information. The specific calculation formula of the bad coefficient is as follows: ; where, is the bad coefficient of the text information data, is the number of bad words obtained after matching the text information data with the bad word library, is the total number of words in the text information data, is the matching correction factor.

[0035] The specific steps for calculating the matching correction factor are as follows: Obtain the word information data of several text information data, and match the word information data of several text information data with the bad word library respectively to obtain the measured quantity values of the bad words of each text information data; At the same time, obtain the actual quantity values of the bad words of each text information data after matching the word information data of each text information data with the bad word library; Based on the measured quantity values and actual quantity values of the bad words of each text information data and combined with the moving average method for comprehensive analysis to obtain the matching correction factor. The matching correction factor can be obtained not only through text processing technology but also through more accurate calculation by combining the moving average method. Its calculation formula is as follows: ; where, is the measured quantity value of the bad words of the th text information data after matching with the bad word library, is the The actual number of bad words obtained after matching the bad word library in the text information data, For the The measured value of bad words after matching the bad word library with the text information data, For the The actual number of bad words obtained after matching the bad word library in the text information data, , is the amount of text information data, for The proportionality coefficient, for The proportionality factor.

[0036] In this embodiment, by splitting the text information data into words or marks, using words as the basic unit of features, this method can capture the features of text information more comprehensively, and the word form restoration process can merge words of different forms into their original forms, making feature extraction more accurate and complete. This comprehensiveness and flexibility enable the system to cope with text information data of different types and forms, improving the accuracy and applicability of detection. At the same time, by calculating the text bad coefficient and combining the preset text bad coefficient threshold, the bad degree of text information data can be quantified, and it can be judged whether it is bad information. The introduction of the matching correction factor further improves the accuracy. By comprehensively analyzing the measured quantity value and the actual quantity value, it can better reflect the matching of bad words and reduce the possibility of misjudgment. This method is practical and can help the network information detection system to efficiently identify and filter out bad information, thereby protecting the safety and health of the network environment.

[0037] Specifically, Figure 3As shown, the file feature is specifically the file type in the file information data, and the file types include text files, image files, audio files, and binary files. The specific process of judging whether the file information data is bad information is as follows: If the file feature is a text file, based on the file operation function in the programming language, the content of the text file is read into the memory and marked as a string. This string contains all the content of the text file. After the file reading is completed, the file handle is closed to release resources, that is, the text file is closed, and the system resources occupied by the file are released. The string is hashed based on the hash algorithm to obtain a text hash value, and the text hash value is matched with the bad text hash library. If the match is successful, the file information data is marked as bad information. The bad text hash library is specifically an information database containing all bad text hash values, and the bad text hash value is specifically a value that exceeds a preset bad text hash threshold. If the file feature is an image file, the image information in the image file is adjusted to a preset size, and the adjusted image information is grayed out. The pixel mean is calculated based on the image information, and the image information is specifically the pixel value information of all pixels in the image file. All pixels greater than the average value are set to 1, and all pixels less than the average value are set to 0. A string is constructed based on the pixel mean, which is the image hash value. The binary values ​​of all pixels are connected to form a long binary string, which is the image hash value, and the image hash value is compared with the bad image. The hash library is used for matching. If the match is successful, the file information data is regarded as bad information. The bad image hash library is specifically an information database containing all bad image hash values. The bad image hash value is specifically a value that exceeds a preset bad image hash threshold. If the file feature is an audio file, the spectrogram in the audio file is extracted, and the spectrogram is converted into an audio string. The audio string is hashed based on a hash algorithm to obtain an audio hash value, and the audio hash value is matched with the bad audio hash library. If the match is successful, the file information data is regarded as bad information. The bad audio hash library is specifically an information database containing all bad audio hash values. The bad audio hash value is specifically a value that exceeds a preset bad audio hash threshold. If the file feature is a binary file, based on the file operation function in the programming language, the content of the binary file is read into the memory and marked as a string. This string contains all the content of the binary file. After the file reading is completed, the file handle is closed and the resources are released, that is, the binary file is closed and the system resources occupied by the file are released. The string is hashed based on the hash algorithm to obtain a binary hash value, and the binary hash value is matched with a bad binary hash library. If the match is successful, the file information data is regarded as bad information. The bad binary hash library is specifically an information database including all bad binary hash values, and the bad binary hash value is specifically a value that exceeds a preset bad binary hash threshold.

[0038] The calculation steps of the pixel mean are as follows: first, remove the pixel points with pixel value information in the image file, and then calculate the pixel mean based on the image information of the removed image file. In addition to being obtained through computer vision technology, the pixel mean can also be obtained by combining the harmonic mean method for more accurate calculation. The calculation formula is as follows: ;in, is the pixel mean, is the first The pixel value information of each pixel point, , is the number of pixels in the image file after removal.

[0039] The specific process of extracting the spectrogram from the audio file is as follows: the audio signal is divided into short-term segments, usually framed using overlapping windows, for example, each window is 20-30 milliseconds long, a window function, such as a Hanning window or a Hamming window, is applied to each segment to reduce boundary effects, the audio signal in each window is Fourier transformed, the time domain signal is converted into a frequency domain signal, the frequency domain signal in each window is squared to obtain an energy spectrum, the energy spectrum is then summed and the logarithm is taken to obtain the power spectrum density of each time segment, the power spectrum density is represented as an image, namely a spectrogram, with the horizontal axis being time and the vertical axis being frequency, and the color depth representing the power value.

[0040] The specific steps of converting a spectrogram into a string are as follows: compress the spectrogram according to a certain compression algorithm to reduce the amount of data. Common compression algorithms include Fourier transform, wavelet transform, etc., and then encode the compressed data to obtain a string representation. The encoding method can be to convert each data point into a character, or to convert a piece of data into a substring, and connect all the encoded strings to obtain a long string, which is the string representation of the spectrogram.

[0041] In this embodiment, by identifying different types of files, such as text files, image files, audio files and binary files, it can be applied to various forms of information data, thereby improving the applicability and diversity of the system, and different types of files are processed in different ways. This flexibility enables the system to effectively deal with different types of bad information, providing all-round protection for network security. For different types of files, targeted processing methods are adopted, such as hash calculation for text files, pixel mean calculation and grayscale processing for image files, spectrogram extraction for audio files, hash calculation for binary files, etc. These processing methods can extract file features more accurately, and by matching with bad hash libraries, it can reliably determine whether the file is bad information, thereby reducing the possibility of misjudgment and improving the accuracy and reliability of detection. The specific process of determining whether the file information data is bad information adopts the method of automatically processing files by computer programs, which improves processing efficiency and practicality. Through the file operation function in the program language, the file content can be quickly read into the memory, and corresponding processing and calculation can be performed, thereby realizing rapid analysis and judgment of file information. This efficiency and practicality enable the system to timely identify and filter out bad information, ensuring the safety and health of the network environment.

[0042] Specifically, Figure 4 As shown, the network traffic characteristics are specifically the number of bytes of the data packet in the network traffic information data, the data transmission rate, the source IP address and the target IP address of the data packet, the source port number and the target port number of the data packet, and the specific process of judging whether the network traffic information data is bad information is as follows: based on the extracted number of bytes of the data packet, the data transmission rate, the source IP address and the target IP address of the data packet, and the source port number and the target port number of the data packet, normalization processing is performed; based on the number of bytes of the data packet after normalization processing, the data transmission rate, the source IP address and the target IP address of the data packet, and the source port number of the data packet The network traffic characteristic value is obtained by comprehensively analyzing and calculating the target port number and the target port number; wherein, for the IPv4 address, it is split into four bytes, and the value of each byte is divided by 255, and the result is a value in the range of [0,1]. For the IPv6 address, similar normalization processing can also be performed. For the port number, the port number is divided by 65535, and the result is also a value in the range of [0,1]; the network traffic characteristic value is compared with the preset network traffic characteristic threshold. If the network traffic characteristic value is higher than the preset network traffic characteristic threshold, the network traffic information data is regarded as bad information.

[0043] In addition to obtaining network traffic characteristic values ​​through network packet capture tools such as Wireshark, network traffic characteristic values ​​can also be obtained through more accurate calculations using the weighted average method. The calculation formula is as follows: ;in, is the network traffic characteristic value, is the number of bytes of the normalized data packet, is the normalized data transmission rate, is the source IP address value of the normalized data packet. is the destination IP address value of the normalized data packet, is the source port number value of the normalized data packet, is the destination port number value of the normalized data packet, for The proportionality coefficient, for The proportionality coefficient, for The proportionality coefficient, for The proportionality coefficient, for The proportionality coefficient, for The proportionality coefficient, .

[0044] In this embodiment, a comprehensive analysis of network traffic is performed by extracting multiple features of network traffic data, such as the number of bytes of a data packet, transmission rate, source IP address, target IP address, source port number and target port number. This comprehensive feature extraction can provide a more comprehensive understanding of the situation of network traffic, help discover and identify bad information, improve the accuracy and reliability of detection, and use normalization to standardize network traffic features so that different types of feature values ​​have the same dimension, which is convenient for comprehensive analysis and comparison. By normalizing data such as IPv4 addresses and port numbers to a range of 0 to 1, the order of magnitude difference between different feature values ​​is eliminated, making the comparison of feature values ​​more fair and accurate, improving the comparability and reliability of detection, and the feature extraction process is flexible and diverse, and can be adjusted and optimized according to the different characteristics of network traffic. At the same time, by setting a preset network traffic feature threshold, customized detection of different types of network traffic can be performed according to specific circumstances, thereby meeting the security requirements in different scenarios, improving the applicability and flexibility of the system, and judging whether the network traffic information data is bad information The specific process uses an automated method to process network traffic data, which improves processing efficiency and practicality. By automatically extracting network traffic features and normalizing them through computer programs, it is possible to quickly and accurately analyze network traffic data, detect abnormal situations in a timely manner and take corresponding measures. This efficiency and practicality enables the system to respond to network security threats in a timely manner and ensure the security and stability of the network environment.

[0045] Specifically, Figure 5 As shown, the image information data includes picture information data and video information data, and the image features include picture features extracted based on the picture information data and video features extracted based on the video information data. The picture features are specifically the color histogram, edge features, texture features, and depth features of the picture information data, and the video features are specifically the color histogram, edge features, texture features, and depth features of the picture information data of each key frame in the video information data. The specific process of judging whether the picture information data is bad information is as follows: the color histogram, edge features, texture features, and depth features of the extracted picture information data are subjected to feature fusion in a weighted summation manner to obtain a picture feature comprehensive vector, which is used to describe the comprehensive features of the picture information data, and the picture feature comprehensive vector is read and input into a classification model for classification, and whether the picture information data is bad information is judged based on the classification result. The classification result is specifically divided into two categories: normal and bad. If it is bad, the picture information data is regarded as bad information.

[0046] The specific process of extracting image features is as follows: input the image information data into the image feature extraction model for feature extraction to obtain the color histogram, edge features, texture features, and depth features of the image information data.

[0047] The specific training process of the image feature extraction model is as follows: obtain a number of image information data to be extracted and establish an image information data set; divide the image information data set into an image information training set and an image information verification set; read the image information training set to train the image feature extraction model and calculate the image loss function; evaluate the model performance based on the image information verification set, and adjust the model parameters according to the verification results until the color histogram, edge features, texture features, and depth features predicted by the model are consistent with the actual color histogram, edge features, texture features, and depth features; the input and output data of the image feature extraction model are the image information data to be extracted and the color histogram, edge features, texture features, and depth features of the image information data, respectively. Features, and the picture feature extraction model includes: an input layer for receiving picture information data to be extracted, a plurality of convolutional layers for extracting features from the picture information data, and each convolutional layer includes a filter and an activation function, the activation function is used to introduce nonlinear characteristics, increase the expression ability of the model, and is used to reduce the spatial dimension of the feature map, retain important information and reduce the amount of calculation. The pooling layer, a fully connected layer for mapping the features extracted by the convolutional layer to the final output category, an output layer for outputting the color histogram, edge features, texture features, and depth features of the final predicted picture information data of the model, a loss function for measuring the difference between the model output and the true label, and an optimizer for adjusting the model parameters to minimize the loss function.

[0048] The training process of the classification model is as follows: obtain several comprehensive image feature vectors, and establish a comprehensive image feature vector data set, and each comprehensive image feature vector is accompanied by a label, which is specifically the category of the image: normal or bad; divide the comprehensive image feature vector data set into a picture feature training set and a picture feature verification set; read the picture feature training set and train the classification model, and calculate the picture feature loss function; evaluate the model performance based on the picture feature verification set, and adjust the model parameters according to the verification results until the label of the picture feature comprehensive vector predicted by the model meets the actual label of the feature comprehensive vector; and the training process of the classification model passes the gradient of the loss function back to the model through the back propagation algorithm to update the parameters of the model, and the training process usually includes multiple iterations. The classification model includes: an input layer for receiving a feature vector as input, each feature as an input node, a plurality of hidden layers for performing nonlinear transformation and feature extraction on the input features, each hidden layer comprising a plurality of neurons, each neuron is connected to all nodes of the input layer and has a weight and a bias, an activation function layer ReLU for introducing nonlinear characteristics and increasing the expressive power of the model, and an output layer, the output layer comprising two neurons, representing normal and bad categories respectively, each neuron is connected to all nodes of the hidden layer and has a weight and a bias, a loss function for measuring the difference between the model output and the true label, and an optimizer for adjusting the model parameters to minimize the loss function.

[0049] The color histogram is the statistical information of the pixel color distribution in the image. It divides the color space in the image into several regions, counts the number of pixels in each region, and represents the result as a histogram. The color histogram can reflect the distribution of different colors in the image and help describe the color characteristics of the image.

[0050] Edge features describe the edge information in an image, that is, the area in the image where the grayscale value changes greatly. Commonly used edge detection algorithms include Sobel, Canny, etc., which can detect the edge position and direction in the image and extract the edge information to describe the contour and shape characteristics of the image.

[0051] Texture features describe the texture information in an image, that is, the spatial distribution of color and grayscale in a local area of ​​the image. Commonly used texture descriptors include local binary patterns, histograms of oriented gradients, etc., which can be used to describe the texture roughness, direction, frequency and other characteristics of an image.

[0052] Deep features are high-level semantic features learned from images through deep learning models. Deep features have strong representation capabilities and can capture abstract information in images, such as objects and scenes in images. Commonly used deep learning models include VGG, ResNet, Inception, etc. The convolutional layers in these models can extract deep features of images.

[0053] In this implementation, by extracting multi-dimensional features such as color histogram, edge features, texture features, depth features, etc. of the image information data, and performing comprehensive analysis through feature fusion and classification models, it is possible to more comprehensively describe the features of the image, thereby improving the accuracy of identifying bad information. By training the model, it is enabled to have the ability to judge normal and bad images, thereby achieving efficient bad image detection and filtering. This method is not only applicable to static image data, but also to key frames in video information data. By extracting different features from different types of image data and combining them with training models for classification, it can adapt to various forms of image data, has strong flexibility and applicability, and can cope with diverse network security needs. By constructing image feature extraction models and classification models, it is achieved that the image is detected and filtered. The automatic extraction and classification of features have greatly improved the processing efficiency and intelligence level. This automated processing method can quickly process large amounts of image data, and timely discover and filter bad information, effectively protecting the security of the network environment. By continuously collecting and annotating image information data, conducting model training and verification, and continuously optimizing and improving the performance of the model, it can adapt to the dynamic changes in the network environment and the evolving security threats. The continuous optimization process makes the model more robust and adaptable, and can maintain a stable detection effect in a complex network environment. Through the timely identification and filtering of bad image information, it helps to prevent various security threats to the network environment caused by bad information, thereby improving the overall security level of the network and protecting the legitimate rights and interests of users and the healthy development of cyberspace.

[0054] Specifically, Figure 6 As shown, the video features are specifically the color histogram, edge features, texture features, and depth features of the picture information data of each key frame in the video information data. The specific process of determining whether the video information data is bad information is as follows: read the color histogram, edge features, texture features, and depth features of the picture information data of each key frame in the extracted video information data, and input them into the classification model for classification to obtain the category of the picture information data of each key frame; determine whether there is a key frame with picture information data classified as bad; if so, the video information data is regarded as bad information.

[0055] The specific extraction process of the key frame is as follows: read the video information data and divide it to obtain the picture information data of all frames of the video information data; based on the picture information data of all frames of the video information data, respectively count the color histogram of the picture information data of each frame; the color histogram specifically includes multiple intervals, each interval represents a color range, and counts the number of pixels in the image that fall into each color range; based on the color histogram of the picture information data of each frame, respectively calculate the similarity of the picture information data between each frame, and take the number of frames of the picture information data corresponding to the similarity exceeding the similarity threshold as the key frame. In addition to being calculated and obtained through image processing and computer vision technology, the similarity can also be obtained through a more accurate calculation method, and its calculation formula is as follows: The calculation formula for similarity is as follows: ;in, For the The picture information data of the frame is the same as the The similarity of the picture information data of the frame, For the The color histogram of the frame's picture information data The statistical value of a color interval, the statistical value is specifically the number of pixels in the color interval, For the The color histogram of the frame's picture information data The statistical value of the color interval, , is the number of frames of video information data, , It is the number of color intervals of the color histogram in the picture information data of each frame.

[0056] In this embodiment, by extracting the multi-dimensional features such as color histogram, edge features, texture features, depth features, etc. of the image information data of each key frame, and classifying them through the classification model, the video content can be comprehensively analyzed, and the recognition accuracy of bad information is improved. Compared with only analyzing the video as a whole, feature extraction and classification for key frames can more accurately capture the bad content in the video, and the extraction of key frames can greatly reduce the processing cost of the video content, avoiding the complex calculation process of frame-by-frame analysis of the entire video. By extracting and classifying features for key frames, it is possible to improve processing efficiency while ensuring detection accuracy, saving computing resources and time costs, and realizing automatic processing and intelligent analysis of video content by constructing a classification model and an algorithm for extracting key frames. This automated processing method can quickly process a large amount of video data, and timely discover and filter bad information, improve processing efficiency and intelligence level, and timely identify and filter bad information in video content, which helps to prevent various security threats caused by bad information to the network environment, thereby improving the overall security level of the network, protecting the legitimate rights and interests of users and the healthy development of cyberspace.

[0057] Specifically, corresponding processing measures are taken for the corresponding information data judged as bad information: if the bad information is text information data or image information data, the text data and image information data containing the bad information are deleted to prevent further propagation, and the text content and image content subsequently uploaded by the user are monitored in real time to promptly discover and process the bad information; if the bad information is file information data, isolation is taken for the files containing the bad information to prevent further propagation, and the file content subsequently uploaded by the user is monitored in real time to promptly discover and process the bad files; if the bad information is network traffic information data, network security equipment, such as firewalls, intrusion detection systems, etc., are used to prevent the spread of bad traffic, and access control measures are implemented for the source of the bad traffic to limit its access rights to network resources.

[0058] In this implementation scheme, deleting text data and image data containing bad information is a timely and effective measure to prevent the further spread of bad information. By real-time monitoring of the text content and image content subsequently uploaded by users, bad information can be discovered and processed in a timely manner to prevent its spread, protect users from the impact of bad content, and maintain the health and order of the network environment. For files containing bad information, isolating them is an effective measure to prevent the spread of bad information without affecting the use of other legal content. By real-time monitoring of the content of files subsequently uploaded by users, bad files can be discovered and processed in a timely manner to protect the security and stability of the network environment. Network security equipment such as firewalls and intrusion detection systems can be used to Blocking and filtering bad traffic is a key measure to ensure network security. Implementing access control measures on the source of bad traffic and limiting its access rights to network resources can effectively prevent the spread of bad information and attacks on the network and maintain network stability and security. By deleting, isolating or blocking bad information, users can be effectively protected from bad content and their sense of security and comfort in the network environment. Timely processing of bad information can purify the network environment and create a safe, harmonious and healthy network space, which is conducive to the healthy development of the network ecology. Taking targeted processing measures can effectively prevent the spread of bad information and attacks on the network, improve the security level of the network, and ensure the security of network systems and user data.

[0059] See also Figure 7The embodiment of the present invention provides a technical solution: a network information detection system based on big data, including: a data acquisition subsystem, a feature extraction subsystem, an analysis and processing subsystem, and a treatment measure subsystem; the data acquisition subsystem is used to acquire the network information data to be detected based on the network monitoring software, the network information data includes text information data, file information data, network flow information data, and image information data, and is specifically responsible for collecting text information data from the network, including text content and news articles on social media, etc., and is responsible for collecting file information data from the network, including various types of files, such as text files, image files, audio files, and binary files, and is responsible for capturing and recording network flow data, including the number of bytes of data packets, transmission rate, IP address and port number, etc., and is responsible for collecting image information data from the network, including pictures and videos; the feature extraction subsystem is used to extract features of the network information data to be detected, and obtain text features, file features, network flow features, and image features respectively, and is specifically responsible for extracting features from the text information data, extracting vocabulary information and other related features of the text, and is responsible for extracting file information data Feature extraction, extracting corresponding features according to file type, such as text hash value, image hash value, audio hash value, etc., responsible for feature extraction of network traffic information data, extracting relevant features of data packets, such as number of bytes, transmission rate, IP address, etc., responsible for feature extraction of image information data, extracting color histogram, edge features, texture features, and depth features of images; analysis and processing subsystem, used to analyze and process based on text features, file features, network traffic features, and image features, and judge whether text information data, file information data, and network traffic information data are bad information according to the analysis and processing results; processing measure subsystem, used to take corresponding processing measures for corresponding information data judged as bad information, specifically, taking corresponding processing measures for text information data judged as bad information, such as deletion, shielding, etc., taking corresponding processing measures for file information data judged as bad information, such as isolation, deletion, etc., taking corresponding processing measures for network traffic information data judged as bad information, such as traffic filtering, access control, etc., and taking corresponding processing measures for image information data judged as bad information, such as deletion, review, etc.

[0060] Specifically, Figure 8As shown, the analysis and processing subsystem includes: a text analysis and processing module, a file analysis and processing module, a network traffic analysis and processing module, and an image analysis and processing module; the text analysis and processing module is used to perform analysis and processing based on text features to determine whether text information data is bad information; the file analysis and processing module is used to perform analysis and processing based on file features to determine whether file information data is bad information; the network traffic analysis and processing module is used to perform analysis and processing based on network traffic features to determine whether network traffic information data is bad information; the image analysis and processing module is used to perform analysis and processing based on image features to determine whether image information data is bad information.

[0061] In this implementation scheme, the text analysis and processing module focuses on the analysis and processing of text information data. By analyzing text features, it can identify keywords and semantic information in the text, and then determine whether the text information data is bad information. For example, through word frequency statistics and semantic analysis, sensitive words or bad content in the text can be found, so that corresponding processing measures can be taken in time. The file analysis and processing module is used to process file information data. By analyzing file features, feature extraction and analysis of files, abnormal content or malicious code in the files can be detected, and then determine whether the file information data is bad information. For example, through features such as file format, file size, and file structure, potential bad files can be quickly identified to protect the security of the system and users. The network traffic analysis and processing module is dedicated to analyzing and processing network traffic information data. By analyzing network traffic features, it can detect abnormal network behavior and traffic data, and identify whether there is transmission of bad information. For example, by analyzing network traffic Monitoring and analysis can detect abnormal data packets, malicious network connections, etc., and take defensive measures in time to protect the security and stability of the network. The image analysis and processing module is used to process image information data. By analyzing image features, extracting and analyzing features of the image, it can identify abnormal content or bad information in the image, and then determine whether the image information data is bad information. For example, through the color, texture, shape and other features of the image, the bad content in the image can be detected and processed in time. Through the detailed analysis and processing of different types of information data, the system can more accurately identify and process bad information, effectively protect the user's data security and the stability of the network environment. Timely discovery and processing of bad information can effectively purify the network environment, improve the user's experience on the network, and enhance the user's trust in the system. Through monitoring and analysis of network traffic, malicious attacks and network intrusions can be discovered and prevented in time, improving the overall security level of the network and ensuring the security of the network system and user data.

[0062] In summary, this application has at least the following effects:

[0063] By performing multi-dimensional feature extraction on the network information data to be detected, including various types of features such as text, files, network traffic and images, it is possible to more comprehensively reflect the attributes and characteristics of the information, thereby improving the accuracy and robustness of information detection. By comprehensively analyzing the characteristics of different types of information, it is possible to more accurately determine whether the information is bad information, thereby effectively curbing the spread of bad information in the network.

[0064] While detecting network information, it can monitor in real time and take targeted measures to deal with data that is judged to be bad information. Corresponding processing strategies are adopted for different types of bad information, such as deleting text information, image information, isolating file information, blocking network traffic, etc. This real-time monitoring and targeted processing mechanism can effectively prevent the continued spread of bad information and protect network security and user interests.

[0065] By dividing the analysis and processing subsystem into text, file, network traffic and image analysis and processing modules, modular processing is achieved. Each module focuses on the analysis and processing of a specific type of information, enhancing the flexibility and scalability of the system. The modular design enables the system to dynamically adjust the functions and configurations of each module according to actual needs, and better adapt to network information detection tasks in different scenarios.

[0066] By having text, file, network traffic and image analysis and processing modules each responsible for the analysis and processing of corresponding types of information, detailed processing of tasks is achieved. This detailed processing enables the system to perform information detection more efficiently, improves the processing speed and accuracy. By decomposing the processing tasks into multiple modules for parallel processing, the system can make full use of computing resources, complete information detection tasks more quickly, and effectively respond to the processing needs of large-scale data.

[0067] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0068] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A network information detection method based on big data, characterized in that: The following steps are involved: Acquire network information data to be detected, wherein the network information data includes text information data, file information data, network flow information data, and image information data; Perform feature extraction on the network information data to be detected, and obtain text features, file features, network traffic features, and image features respectively; Analyze and process the text features, file features, network traffic features, and image features respectively, and determine whether the text information data, file information data, and network traffic information data are bad information according to the analysis and processing results; And take corresponding processing measures for the corresponding information data judged as bad information; The text feature is specifically all word information data in the text information data. The specific process of judging whether the text information data is bad information is as follows: Read text information data and divide it; Remove stop words from the divided text information data; Performing word form restoration on the text information data after the stop words are removed to obtain all word information in the text information data; Match all word information in the text information data with the bad word library respectively to obtain the number of bad words in the text information data; Calculating a text badness coefficient of the text information data based on all word information of the text information data and the number of bad words, and comparing the text badness coefficient of the text information data with a preset text badness coefficient threshold; If the bad coefficient of the text information data is higher than the preset bad coefficient threshold, the text information data is regarded as bad information; The specific calculation formula of the defect coefficient is as follows: ； in, is the bad coefficient of text information data, The number of bad words obtained after matching the bad word library in the text information data. is the total number of words in the text information data, is the matching correction factor; The specific steps of calculating the matching correction factor are: obtaining word information data of a plurality of text information data, and matching the word information data of the plurality of text information data with the bad word library respectively, and obtaining the measured quantity value of the bad words of each text information data respectively; At the same time, the actual number value of bad words in each text information data after each text information data word information data is matched with the bad word library; Based on the measured quantity value and actual quantity value of the bad words in each text information data and combined with the moving exponential average method, a comprehensive analysis is performed to obtain a matching correction factor; The file feature is specifically the file type in the file information data, and the file type includes text files, image files, audio files, and binary files. The specific process of determining whether the file information data is bad information is as follows: If the file feature is a text file, the content of the text file is read into the memory based on the file operation function in the programming language and marked as a string. After the file is read, the file handle is closed to release resources. The string is hashed based on the hash algorithm to obtain a text hash value, and the text hash value is matched with the bad text hash library. If the match is successful, the file information data is regarded as bad information. If the file feature is an image file, the image information in the image file is adjusted to a preset size, the adjusted image information is grayed out, the pixel mean is calculated based on the image information, and a character string is constructed based on the pixel mean, which is the image hash value, and the image hash value is matched with the bad image hash library. If the match is successful, the file information data is regarded as bad information; If the file feature is an audio file, extract the spectrogram from the audio file and convert the spectrogram into an audio string. Perform hash calculation on the audio string based on a hash algorithm to obtain an audio hash value. Match the audio hash value with a bad audio hash library. If the match is successful, the file information data is regarded as bad information. If the file feature is a binary file, the contents of the binary file are read into the memory based on the file operation function in the programming language and marked as a string. After the file is read, the file handle is closed to release resources. The string is hashed based on the hash algorithm to obtain a binary hash value, and the binary hash value is matched with the bad binary hash library. If the match is successful, the file information data is regarded as bad information. The network traffic characteristics are specifically the number of bytes of the data packet in the network traffic information data, the data transmission rate, the source IP address and the destination IP address of the data packet, the source port number and the destination port number of the data packet. The specific process of judging whether the network traffic information data is bad information is as follows: Normalization is performed based on the number of bytes of the extracted data packets, the data transmission rate, the source IP address and the destination IP address of the data packets, and the source port number and the destination port number of the data packets; Based on the number of bytes of the normalized data packet, the data transmission rate, the source IP address and the destination IP address of the data packet, the source port number and the destination port number of the data packet, a comprehensive analysis and calculation is performed to obtain the network traffic characteristic value; The network traffic characteristic value is compared with a preset network traffic characteristic threshold. If the network traffic characteristic value is higher than the preset network traffic characteristic threshold, the network traffic information data is regarded as bad information.

2. The network information detection method based on big data according to claim 1 is characterized in that: The image information data includes picture information data and video information data, the image features include picture features extracted based on the picture information data and video features extracted based on the video information data, the picture features are specifically the color histogram, edge features, texture features, and depth features of the picture information data, and the specific process of determining whether the picture information data is bad information is as follows: The color histogram, edge features, texture features, and depth features of the extracted image information data are fused to obtain a comprehensive image feature vector; The comprehensive feature vector of the image is read and input into the classification model for classification, and whether the image information data is bad information is determined based on the classification result.

3. The network information detection method based on big data according to claim 2 is characterized in that: The video features are specifically the color histogram, edge features, texture features, and depth features of the image information data of each key frame in the video information data. The specific process of determining whether the video information data is bad information is as follows: Read the color histogram, edge features, texture features, and depth features of the image information data of each key frame in the extracted video information data, and input them into the classification model for classification to obtain the category of the image information data of each key frame; Determine whether there is a key frame of image information data classified as bad; If it exists, the video information data is regarded as bad information.

4. The network information detection method based on big data according to claim 1 is characterized in that: And take corresponding processing measures for the corresponding information data judged as bad information, specifically: If the harmful information is text or image data, the text or image data containing the harmful information shall be deleted to prevent further dissemination, and the text or image content subsequently uploaded by the user shall be monitored in real time to promptly detect and handle the harmful information; If the bad information is file information data, the file containing the bad information will be isolated to prevent further spread, and the content of the files subsequently uploaded by the user will be monitored in real time to detect and handle the bad files in a timely manner; If the bad information is network traffic information data, use network security equipment to prevent the spread of bad traffic, and implement access control measures for the source of the bad traffic to limit its access rights to network resources.

5. A network information detection system based on big data, applying the network information detection method based on big data according to any one of claims 1 to 4, characterized in that: include: Data acquisition subsystem, feature extraction subsystem, analysis and processing subsystem, and treatment measures subsystem; The data acquisition subsystem is used to acquire network information data to be detected based on the network monitoring software, and the network information data includes text information data, file information data, network flow information data, and image information data; The feature extraction subsystem is used to extract features from the network information data to be detected, and obtain text features, file features, network traffic features, and image features respectively; The analysis and processing subsystem is used to perform analysis and processing based on text features, file features, network traffic features, and image features, and to determine whether the text information data, file information data, and network traffic information data are bad information based on the analysis and processing results; The processing measure subsystem is used to take corresponding processing measures for the corresponding information data judged as bad information.

6. The network information detection system based on big data according to claim 5 is characterized in that: The analysis and processing subsystem includes: a text analysis and processing module, a file analysis and processing module, a network traffic analysis and processing module, and an image analysis and processing module; The text analysis and processing module is used to analyze and process the text information data according to the text features to determine whether the text information data is bad information; The file analysis and processing module is used to analyze and process the file according to the file characteristics to determine whether the file information data is bad information; The network traffic analysis and processing module is used to analyze and process network traffic characteristics to determine whether network traffic information data is bad information; The image analysis and processing module is used to perform analysis and processing based on image features to determine whether the image information data is bad information.

Images