Internet intelligent detection method, system and device based on big data DPI technology

Abstract

The invention relates to the technical field of electronic information, in particular to an internet intelligent detection method, system and device based on the big data DPI technology, and the method comprises the steps: carrying out the deep analysis of a data package in network flow based on the DPI technology, and extracting the information of an application layer; the extracted information is stored by adopting a big data storage technology, and abnormal behaviors and bad contents are identified by adopting a big data analysis technology; an intelligent detection algorithm is constructed, key features of abnormal behaviors and bad contents are extracted, and the abnormal behaviors and the bad contents are automatically identified by using machine learning and deep learning technologies. According to the method, the application layer information of the network traffic is deeply analyzed, and a big data storage and analysis technology is combined, so that efficient processing and accurate identification of the network traffic are realized. Meanwhile, machine learning and deep learning algorithms in the artificial intelligence technology are introduced, bad content and abnormal behaviors in the network are automatically learned and recognized, and the intelligent level and accuracy of detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and specifically to an Internet intelligent detection method, system and device based on big data DPI technology. Background Art

[0002] With the rapid development of the Internet, the explosive growth of network traffic has brought challenges to network security and content compliance. As a technology that can deeply analyze the application layer information of network traffic, deep packet inspection (DPI) technology analyzes the content of data packets in network traffic, identifies and controls network traffic, thereby improving network security and management efficiency, providing strong support for solving these problems.

[0003] However, with the continuous growth of network traffic and the diversification of network applications, traditional DPI technology faces the following performance and security challenges, including performance bottlenecks, high power consumption and cost, low efficiency of data collection and analysis, limited detection capabilities for encrypted traffic, algorithm and platform constraints, system complexity and maintenance difficulty, inadequacy of matching algorithms, and low accuracy and throughput.

[0004] Therefore, in the era of big data, how to combine DPI technology with big data technology to realize intelligent detection of Internet content has become an issue that needs to be studied urgently. Summary of the invention

[0005] In order to overcome the problems existing in the prior art, the purpose of the present invention is to provide an intelligent Internet detection method, system and device based on big data DPI technology, which collects network traffic data in real time, uses big data storage and analysis technology to deeply analyze and process network traffic, identifies bad content and abnormal behavior, and realizes intelligent detection of Internet content.

[0006] To achieve the above object, the present invention provides the following technical solution: an intelligent Internet detection method based on big data DPI technology, comprising the following steps:

[0007] S1: Based on DPI technology, the data packets in the network traffic are deeply analyzed to extract the application layer information;

[0008] S2: using big data storage technology to store the information extracted in step S1, and then using big data analysis technology to analyze the information extracted in step S1 to identify abnormal behavior and inappropriate content;

[0009] S3: Build intelligent detection algorithms to extract features of abnormal behaviors and inappropriate content, and use machine learning and deep learning technologies to automatically identify abnormal behaviors and inappropriate content;

[0010] S4: Visualize the detection results.

[0011] The present invention is further configured as follows: Step S1 specifically uses a data capture module to capture data packets of network traffic passing through a network interface in real time, a traffic preprocessing module preprocesses the captured data packets, and then uses a deep analysis engine to parse the data packets. Based on the analysis results, an application layer information extraction module extracts application layer information, and then stores the analysis results in a data storage module to encrypt data transmission and storage.

[0012] The present invention is further configured as follows: the application layer information extraction module extracts application layer information by using a content identification algorithm to identify the data content of the parsing result; using a source identification algorithm to identify the data source of the parsing result; the application layer information includes URL, file type and user identity.

[0013] The present invention is further configured as follows: the content identification algorithm includes keyword matching and pattern recognition; the source identification algorithm includes IP address analysis and port analysis.

[0014] The present invention is further configured as follows: Step S2 specifically adopts Hadoop Distributed File System (HDFS) as the main storage system, combined with a column-based database to store and manage data; uses data mining technology to identify patterns and associations in data packets of network traffic, builds a real-time data processing pipeline, performs real-time analysis on data packets of network traffic, uses machine learning algorithms to analyze data packets, predicts and identifies abnormal behaviors and bad content; and takes encryption measures to ensure data security during storage and analysis.

[0015] The present invention is further configured as follows: Step S3 specifically comprises the following steps:

[0016] S31: Data collection and preprocessing, extracting key features of abnormal behavior and bad content;

[0017] S32: Establish machine learning models and deep learning models, and combine machine learning models and deep learning models through integrated learning methods;

[0018] S33: Model training and optimization;

[0019] S34: Establish a feedback mechanism to allow manual review of the model's detection results and feed the review results back into model training to form a closed-loop learning;

[0020] S35: Monitor algorithm performance, regularly update the model with new data, and collect user feedback to improve model design, adjust hyperparameters, or optimize system architecture.

[0021] The present invention is further configured as follows: Step S31 specifically includes: building a data collection system to collect network traffic data, wherein the network traffic data includes HTTP requests, DNS queries, IP packet header information, and application layer data; removing irrelevant data, processing missing values ​​and abnormal values, and ensuring data quality; manually annotating a portion of the data set to distinguish between normal and abnormal behaviors, and the types of inappropriate content; extracting key features of abnormal behaviors and inappropriate content, wherein the key features include traffic size, access frequency, access time, IP address distribution, and request type.

[0022] The Internet content intelligent detection system based on big data DPI technology is applicable to the above-mentioned Internet intelligent detection method based on big data DPI technology, including: a deep message analysis system, a big data storage and analysis system and an intelligent detection system; the deep message analysis system performs deep analysis on data packets in network traffic based on DPI technology, the big data storage and analysis system is used for efficient storage and in-depth analysis of data packets, and the intelligent detection system is used for automatically identifying bad content and abnormal behavior in the network.

[0023] It should be noted that each system architecture should consider integration with the existing network architecture and ensure that the system has good scalability so that storage capacity and computing power can be easily increased in the future.

[0024] The present invention is further configured as follows: the deep message parsing system comprises:

[0025] Data capture module: responsible for capturing the network traffic passing through the network interface in real time;

[0026] Traffic preprocessing module: performs preliminary processing on captured data packets, such as reassembling TCP streams, removing duplicate data packets, etc.

[0027] Deep parsing engine: responsible for deep parsing of pre-processed data packets;

[0028] Application layer information extraction module: extracts application layer information from the parsed data;

[0029] Data storage module: stores parsed and extracted information for subsequent analysis;

[0030] User interface module: provides a user interaction interface to display analysis results and statistical information;

[0031] The big data storage and analysis system includes:

[0032] Distributed file system: Hadoop Distributed File System (HDFS) is used as the main storage solution;

[0033] Column-oriented databases: used to store and manage data, such as Apache Cassandra or HBase;

[0034] Big data analysis module: used to identify patterns and associations in network traffic, analyze network traffic data, and identify abnormal behavior and inappropriate content;

[0035] Visualization module: used to display the analysis results in the form of charts and graphs;

[0036] The intelligent detection system comprises:

[0037] Data collection system: used to collect network traffic data;

[0038] Data preprocessing module: used to implement manual labeling, data cleaning and feature extraction;

[0039] Algorithm module: Combine machine learning and deep learning models to improve the generalization ability of the model through integrated learning methods (such as stacking and blending);

[0040] Real-time monitoring and feedback system: Realize real-time processing of data streams, quickly respond to changes in network traffic, manually review the detection results of the model, and feed the results back to the algorithm module to form a closed-loop learning.

[0041] Performance monitoring system: used to monitor the performance of the intelligent detection system and ensure the stable operation of the detection algorithm.

[0042] The present invention also relates to a computer device, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned Internet intelligent detection method based on big data DPI technology.

[0043] In summary, the beneficial effects of the above technical solution of the present invention are as follows:

[0044] The present invention proposes an intelligent detection method for Internet content that integrates deep packet analysis, big data analysis, and artificial intelligence technology. This method achieves efficient processing and accurate identification of network traffic by deeply analyzing the application layer information of network traffic and combining it with big data storage and analysis technology. At the same time, it introduces machine learning and deep learning algorithms in artificial intelligence technology to automatically learn and identify bad content and abnormal behaviors in the network, significantly improving the intelligence level and accuracy of detection. This innovation provides a new solution for network security and content compliance management, which has important theoretical significance and practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 This is a flow chart of the Internet intelligent detection method based on big data DPI technology described in Example 1. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention. Based on the embodiments of the present invention, other similar embodiments obtained by ordinary technicians in the field without making any creative work should all fall within the scope of protection of the present invention.

[0048] The present invention will be further described below in conjunction with the accompanying drawings and preferred embodiments.

[0049] Embodiment 1:

[0050] like Figure 1 As shown in the figure, a preferred embodiment of the present invention is an Internet intelligent detection method based on big data DPI technology, which includes the following steps:

[0051] S1: Based on DPI technology, the data packets in the network traffic are deeply analyzed to extract the application layer information;

[0052] The data capture module is used to capture the data packets of the network traffic passing through the network interface in real time. The traffic preprocessing module preprocesses the captured data packets, and then the deep analysis engine is used to parse the data packets. The application layer information extraction module extracts the application layer information, and the content recognition algorithm is used to identify the data content of the analysis result; the source recognition algorithm is used to identify the data source of the analysis result; the application layer information includes URL, file type and user identity. The content recognition algorithm includes keyword matching and pattern recognition; the source recognition algorithm includes IP address analysis and port analysis.

[0053] The analysis results are then stored in the data storage module, and a user interface module is developed. A web interface or desktop application is developed to provide a user interaction interface and realize data display functions, such as charts, reports, etc., to intuitively display the analysis results. The user interface module is developed to provide query and statistical functions to facilitate users to analyze the data.

[0054] Perform unit testing, integration testing, and performance testing on the system to ensure system stability and performance. Perform system optimization based on the test results to improve parsing efficiency and accuracy. The process is: Let Test be the test function and Optimize be the optimization function, then: System_performance = Optimize(Test(S)).

[0055] S2: using big data storage technology to store the information extracted in step S1, and then using big data analysis technology to analyze the information extracted in step S1 to identify abnormal behavior and inappropriate content;

[0056] Hadoop Distributed File System (HDFS) is used as the main storage system, combined with columnar database to store and manage data; data mining technology is used to identify patterns and associations in data packets of network traffic, build a real-time data processing pipeline, perform real-time analysis on data packets of network traffic, use machine learning algorithms to analyze data packets, predict and identify abnormal behaviors and bad content; encryption measures are taken during the storage and analysis process to ensure data security.

[0057] S3: Build intelligent detection algorithms to extract features of abnormal behaviors and inappropriate content, and use machine learning and deep learning technologies to automatically identify abnormal behaviors and inappropriate content;

[0058] S31: Data collection and preprocessing, extracting key features of abnormal behavior and bad content;

[0059] Build a data collection system to collect network traffic data, including HTTP requests, DNS queries, IP packet header information, and application layer data; remove irrelevant data, process missing values ​​and outliers, and ensure data quality; manually annotate a portion of the data set to distinguish between normal and abnormal behaviors, as well as the types of inappropriate content; extract key features of abnormal behaviors and inappropriate content, including traffic size, access frequency, access time, IP address distribution, and request type.

[0060] The process is as follows:

[0061] Let D be the collected raw network traffic dataset. Let L be the labeling function, label a part of the data in dataset D as normal and abnormal, as well as the type of bad content, and obtain the labeled dataset D L =L(D).

[0062] Let C be the cleaning function, remove irrelevant data, process missing values ​​and outliers, and get the cleaned data set D C =C(D L ).

[0063] Feature Engineering: Let F be the feature extraction function, extract key features from the cleaned data set, and obtain the feature data set X = F (D C ). The feature dataset X can be represented as a matrix, where each row represents a data sample and each column represents a feature.

[0064] S32: Establish machine learning models and deep learning models, and combine machine learning models and deep learning models through integrated learning methods;

[0065] The machine learning model can use classification algorithms such as random forest, support vector machine, gradient boosting decision tree, etc. as the baseline model. This embodiment takes random forest as an example. rf is a random forest model. The prediction of the model can be expressed as y pred =M rf (X test ), where X test is the feature data of the test set, y pred is the prediction result.

[0066] The deep learning model can use convolutional neural network (CNN) and recurrent neural network (RNN) models to capture the spatiotemporal characteristics of traffic data. cnn or M rnn is the corresponding deep learning model. The forward propagation process of the model can be expressed as y pred =M cnn / rnn (X test ; θ), where θ is a model parameter.

[0067] Use ensemble learning methods (such as Stacking, Blending) to combine machine learning and deep learning models to obtain an ensemble model M E The prediction of the ensemble model can be a weighted average or other combination of the prediction results of each base model.

[0068] S33: Model training and optimization;

[0069] Divide the feature dataset X into a training set X train , validation set X val and the test set X test .

[0070] Use grid search, random search or Bayesian optimization to tune hyperparameters. Let λ be the hyperparameter set, then the tuning process can be expressed as finding the hyperparameter combination that makes the validation set performance optimal, that is, λ*=argmax λ A(ME(X train ;λ),X val ).

[0071] K-fold cross validation is used to ensure the stability and reliability of the model. K is the number of cross validation folds, and the cross validation process can be expressed as train Perform K splits and training, and calculate the average performance.

[0072] The model performance is evaluated using indicators such as accuracy, recall, F1 score, ROC-AUC, etc. The evaluation function can be expressed as A(M E (X test) ,Y test ), where Y test is the true label of the test set.

[0073] S34: Establish a feedback mechanism to allow manual review of the model's detection results and feed the review results back into model training to form a closed-loop learning;

[0074] Realize real-time processing of data streams and quickly respond to changes in network traffic. The real-time detection function can be expressed as R(S(M E ),D new ), where D new It is the new network traffic data.

[0075] A feedback mechanism is established to allow manual review of the model's detection results and to feed the results back into the model training to form a closed-loop learning. The feedback function can be expressed as B(M E , R(S(M E ),D new )) to update model parameters or retrain the model.

[0076] S35: Monitor algorithm performance, regularly update the model with new data, and collect user feedback to improve model design, adjust hyperparameters, or optimize system architecture.

[0077] The model update can be expressed as M Enew =I(M E ,D new ,A,B), where I is the iterative function used to update the model based on new data and feedback results.

[0078] During the entire algorithm recognition process, it is necessary to ensure the security of data during transmission and storage, and use data encryption technology to protect data. Comply with relevant laws and regulations, and anonymize user data to protect user privacy.

[0079] S4: Visualize the detection results.

[0080] Embodiment 2:

[0081] The Internet content intelligent detection system based on big data DPI technology is applicable to the above-mentioned Internet intelligent detection method based on big data DPI technology, including: a deep message analysis system, a big data storage and analysis system and an intelligent detection system; the deep message analysis system performs deep analysis on data packets in network traffic based on DPI technology, the big data storage and analysis system is used for efficient storage and in-depth analysis of data packets, and the intelligent detection system is used for automatically identifying bad content and abnormal behavior in the network.

[0082] It should be noted that each system architecture should consider integration with the existing network architecture and ensure that the system has good scalability so that storage capacity and computing power can be easily increased in the future.

[0083] The deep message parsing system comprises:

[0084] Data capture module: responsible for capturing the network traffic passing through the network interface in real time;

[0085] Let Capture(t) be the data capture function at time t, using libpcap or similar libraries, then: Capture(t) = libpcap_capture(t);

[0086] Configure capture rule R, then: Capture(t,R)=libpcap_capture_with_rule(t,R);

[0087] Implement multi-threaded or asynchronous IO mechanism, let Thread_or_Async_IO be the mechanism, then: Capture_Efficient(t,R)=Thread_or_Async_IO(Capture(t,R)).

[0088] Traffic preprocessing module: performs preliminary processing on captured data packets, such as reassembling TCP streams, removing duplicate data packets, etc.

[0089] Let Preprocess(P) be the preprocessing function, where P is the set of captured packets, then:

[0090] Preprocess(P)={TCP_reassemble(P),Remove_duplicates(P),Timestamp_sync(P)}.

[0091] Deep parsing engine: responsible for deep parsing of pre-processed data packets;

[0092] Let Parse(P′) be the deep parsing function, where P′ is the set of preprocessed data packets, then:

[0093] Parse(P′)=Regex_or_Statemachine_parse(P′);

[0094] Supports multiple protocol parsing. Let Protocol_set be the protocol set:

[0095] Parse(P′,Protocol_set)=Multi_protocol_parse(P′,Protocol_set);

[0096] Implement the protocol identification mechanism, assuming Identify_protocol is the identification function, then:

[0097] Parsed_data=Identify_protocol(Parse(P′)).

[0098] Application layer information extraction module: extracts application layer information from the parsed data;

[0099] Let Extract_app_info(D) be the information extraction function, where D is the parsed data, then: App_info = Extract_app_info(D);

[0100] Implement the content recognition algorithm and the source recognition algorithm. Let Content_recognize and Source_recognize be the two algorithms respectively. Then:

[0101] App_info_detailed={Content_recognize(D), Source_recognize(D)}.

[0102] Data storage module: stores parsed and extracted information for subsequent analysis;

[0103] Assume Store(D′) is the storage function, where D′ is the extracted information, then: Store(D′)=Database_store(D′,Database_system).

[0104] Design data model M, then storage optimization is:

[0105] Store_optimized(D′)=Database_store_with_model(D′,Database_system,M)

[0106] Implement the data backup and recovery mechanism. Assume Backup and Restore are the backup and recovery functions respectively, then: Store_secure(D′)={Store_optimized(D′), Backup(D′), Restore(D′)}.

[0107] User interface module: provides a user interaction interface to display analysis results and statistical information;

[0108] Assume UI_display(D″) is the user interface display function, where D″ is the stored data, then: UI_display(D″)=Web_or_Desktop_app_display(D″)

[0109] To realize data display and query statistics functions, let Visualize and Query_stat be these two functions respectively, then: UI_full(D″)={UI_display(D″),Visualize(D″),Query_stat(D″)}.

[0110] Encrypt the data transmission and storage to realize data encryption transmission and storage and protect data security. Let Encrypt be the encryption function, then the data encryption transmission and storage is: Data_secure(D″′)=Encrypt(D″′)

[0111] Comply with laws and regulations to ensure privacy protection. Let Comply_law be the compliance function, then:

[0112] System_secure(D″′)={Data_secure(D″′),Comply_law(D″′)}.

[0113] The big data storage and analysis system includes:

[0114] Distributed file system: Hadoop Distributed File System (HDFS) is used as the main storage solution;

[0115] Assume HDFS is the Hadoop distributed file system, then the storage solution S can be expressed as: S = HDFS (D);

[0116] Among them, HDFS(D) means storing the data set D in the Hadoop distributed file system.

[0117] Column-oriented databases: used to store and manage data, such as Apache Cassandra or HBase;

[0118] Assume that ColumnStore is a column storage database (such as Apache Cassandra or HBase), then the column storage can be expressed as: D′=ColumnStore(D);

[0119] Wherein, D′ represents the data set after column storage processing.

[0120] Big data analysis module: used to identify patterns and associations in network traffic, analyze network traffic data in real time, and identify abnormal behavior and bad content;

[0121] Assume DataMining is a set of data mining algorithms (such as Apriori, FP-Growth, etc.), then the data mining process can be expressed as: Patterns = DataMining (D')

[0122] Among them, Patterns represents the patterns and associations mined from the dataset D′.

[0123] Let MachineLearning be a set of machine learning algorithms (such as SVM, random forest, cluster analysis, etc.), then machine learning analysis can be expressed as: Models = MachineLearning (D′)

[0124] Among them, Models represents the machine learning model trained based on the dataset D′.

[0125] Let StreamProcessing be a stream processing technology (such as Apache Storm or Apache Flink), then real-time analysis can be expressed as: RealTimeAnalysis = StreamProcessing (D)

[0126] Among them, RealTimeAnalysis represents the real-time analysis of the network traffic data set D.

[0127] Visualization module: used to display the analysis results in the form of charts and graphs;

[0128] Let Visualization be the data visualization tool, then the data visualization process can be expressed as: Visuals = Visualization (Patterns, Models, Real Time Analysis)

[0129] Among them, Visuals means visualizing the mined patterns, trained models, and real-time analysis results as charts and graphs.

[0130] When building a big data storage and analysis system, it is necessary to consider system integration and scalability. Assuming that Integration and Scalability are functions of system integration and scalability respectively, this process can be abstractly expressed as:

[0131] IntegratedSystem=Integration(S,A,Visualization,SecureData)

[0132] ScalableSystem=Scalability(IntegratedSystem)

[0133] Among them, IntegratedSystem refers to the integrated system, and ScalableSystem refers to the scalable system.

[0134] The intelligent detection system comprises:

[0135] Data collection system: used to collect network traffic data;

[0136] Data preprocessing module: used to implement manual labeling, data cleaning and feature extraction;

[0137] Algorithm module: Combine machine learning and deep learning models to improve the generalization ability of the model through integrated learning methods (such as stacking and blending);

[0138] Real-time monitoring and feedback system: Realize real-time processing of data streams, quickly respond to changes in network traffic, manually review the detection results of the model, and feed the results back to the algorithm module to form a closed-loop learning.

[0139] Performance monitoring system: used to monitor the performance of the intelligent detection system and ensure the stable operation of the detection algorithm.

[0140] When building an intelligent detection system, design a scalable system architecture that supports real-time or batch processing of network traffic data. Deploy the trained model to a server or cloud platform to ensure efficient data processing capabilities. Develop an API interface to facilitate other systems or services to call the detection algorithm.

[0141] Embodiment 3:

[0142] A computer device stores a computer program, which, when executed by a processor, implements the above-mentioned Internet intelligent detection method based on big data DPI technology.

[0143] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. An Internet intelligent detection method based on big data DPI technology, characterized in that: The following steps are involved: S1: Based on DPI technology, the data packets in the network traffic are deeply analyzed to extract the application layer information; S2: using big data storage technology to store the information extracted in step S1, and then using big data analysis technology to analyze the information extracted in step S1 to identify abnormal behavior and inappropriate content; S3: Build intelligent detection algorithms to extract key features of abnormal behaviors and inappropriate content, and use machine learning and deep learning technologies to automatically identify abnormal behaviors and inappropriate content; S4: Visualize the detection results.

2. The Internet intelligent detection method based on big data DPI technology according to claim 1 is characterized in that: Specifically, step S1 uses a data capture module to capture data packets of network traffic passing through the network interface in real time, a traffic preprocessing module preprocesses the captured data packets, and then uses a deep analysis engine to parse the data packets. Based on the analysis results, an application layer information extraction module extracts application layer information, and then stores the analysis results in a data storage module to encrypt data transmission and storage.

3. The Internet intelligent detection method based on big data DPI technology according to claim 2 is characterized in that: The application layer information extraction module extracts application layer information by using a content identification algorithm to identify the data content of the parsing result; using a source identification algorithm to identify the data source of the parsing result; the application layer information includes URL, file type and user identity.

4. The Internet intelligent detection method based on big data DPI technology according to claim 3 is characterized in that: The content identification algorithm includes keyword matching and pattern identification; the source identification algorithm includes IP address analysis and port analysis.

5. The Internet intelligent detection method based on big data DPI technology according to claim 1 is characterized in that: Step S2 specifically includes: using Hadoop distributed file system as the main storage system, combining with columnar database to store and manage data; using data mining technology to identify patterns and associations in data packets of network traffic, building a real-time data processing pipeline, performing real-time analysis on data packets of network traffic, using machine learning algorithms to analyze data packets, predicting and identifying abnormal behaviors and bad content; taking encryption measures to ensure data security during storage and analysis.

6. The Internet intelligent detection method based on big data DPI technology according to claim 1 is characterized in that: Step S3 specifically includes the following steps: S31: Data collection and preprocessing, extracting key features of abnormal behavior and bad content; S32: Establish machine learning models and deep learning models, and combine machine learning models and deep learning models through integrated learning methods; S33: Model training and optimization; S34: Establish a feedback mechanism to allow manual review of the model's detection results and feed the review results back into model training to form a closed-loop learning; S35: Monitor algorithm performance, regularly update the model with new data, and collect user feedback to improve model design, adjust hyperparameters, or optimize system architecture.

7. The Internet intelligent detection method based on big data DPI technology according to claim 1 is characterized in that: Step S31 specifically includes building a data collection system to collect network traffic data, wherein the network traffic data includes HTTP requests, DNS queries, IP packet header information, and application layer data; Remove irrelevant data from network traffic data and handle missing values ​​and outliers; Manually annotate a portion of the data set to distinguish between normal and abnormal behaviors, as well as the types of inappropriate content; extract key features of abnormal behaviors and inappropriate content; The key features include traffic size, access frequency, access time, IP address distribution, and request type.

8. An intelligent Internet content detection system based on big data DPI technology, applicable to the intelligent Internet content detection method based on big data DPI technology according to any one of claims 1 to 7, characterized in that: include: Deep message analysis system, big data storage and analysis system and intelligent detection system; The deep message analysis system performs deep analysis on data packets in network traffic based on DPI technology, the big data storage and analysis system is used to efficiently store and deeply analyze data packets, and the intelligent detection system is used to automatically identify bad content and abnormal behavior in the network; When designing each system architecture, we should consider integration with the existing network architecture and ensure that the system has good scalability.

9. The intelligent Internet content detection system based on big data DPI technology according to claim 8 is characterized in that: The deep message parsing system comprises: Data capture module: responsible for capturing the network traffic passing through the network interface in real time; Traffic preprocessing module: performs preliminary processing on captured data packets; Deep parsing engine: responsible for deep parsing of pre-processed data packets; Application layer information extraction module: extracts application layer information from the parsed data; Data storage module: stores parsed and extracted information for subsequent analysis; User interface module: provides a user interaction interface to display analysis results and statistical information; The big data storage and analysis system includes: Distributed file system: Hadoop distributed file system is used as the main storage solution; Column-based database: used to store and manage data; Big data analysis module: used to identify patterns and associations in network traffic, analyze network traffic data, and identify abnormal behavior and inappropriate content; Visualization module: used to display the analysis results in the form of charts and graphs; The intelligent detection system comprises: Data collection system: used to collect network traffic data; Data preprocessing module: used to implement manual labeling, data cleaning and feature extraction; Algorithm module: Combining machine learning and deep learning models, improving the generalization ability of the model through integrated learning methods; Real-time monitoring and feedback system: Realize real-time processing of data streams, quickly respond to changes in network traffic, manually review the detection results of the model, and feed the results back to the algorithm module to form a closed-loop learning; Performance monitoring system: used to monitor the performance of the intelligent detection system.

10. A computer device, characterized in that: The computer device stores a computer program, which, when executed by a processor, implements the Internet intelligent detection method based on big data DPI technology as described in any one of claims 1 to 7.

Images