Webpage mining detection method and system based on machine learning
Through the machine learning-based web mining detection method, the machine learning model is used to detect web features in real time, which solves the problems of difficulty in identifying mining detection and low accuracy in the existing technology, and achieves efficient and accurate mining behavior Identification and early warning improve the security and browsing experience of user equipment.
Patent Information
- Application Number
- CN202410970699.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-15
AI Technical Summary
The existing mining detection methods have problems such as difficulty in maintaining feature databases, complex rules settings and high false alarm rates, and cannot effectively identify and warn of mining behavior.
The web mining detection method based on machine learning is adopted to collect historical and real-time web page data, extract web page features, train machine learning models, conduct real-time detection, and conduct early warnings based on the detection results. The specific steps include data collection, feature extraction, model training and early warning processing, and use WebSocket connection, Web Worker, PostMessage/onMessage event frequency, CPU usage and other features for detection.
It realizes efficient, accurate and real-time detection of web mining behavior, reduces false alarms and missed reports, protects the performance and security of user equipment, improves browsing experience, and maintains detection of the latest mining technologies through continuous learning. ability.
Smart Images

Figure CN118972103A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of mining detection, and in particular to a webpage mining detection method and system based on machine learning. Background Art
[0002] On November 1, 2008, Satoshi Nakamoto released the Bitcoin white paper "Bitcoin: A Peer-to-Peer Electronic Cash System", proposing a virtual encrypted digital currency in the form of P2P. Bitcoin is essentially generated by the calculation of network nodes, which is a clear difference between it and legal tender. It is a distributed, anonymous, and completely decentralized payment system that uses digital signatures, hash algorithms, consensus mechanisms, incentive mechanisms and other technical means as support.
[0003] The acquisition of Bitcoin mainly relies on computing power, and mining rewards are obtained by performing random hash operations and obtaining random numbers within a certain period of time. This also means that as long as you have a computer, you can participate in mining. Due to the convenience of mining and the value of general equivalents, a large number of people have been attracted to the trend of mining. However, with the development of technology, many security issues have also been brought about. A large number of hackers use certain attack methods to illegally use the victim's system resources and network resources to mine currency, causing a lot of losses.
[0004] With the improvement of Web technology WebAssembly, the emergence of algorithms such as Cryptonight specifically used for proof of work, and the popularization of cloud services, hackers' attack methods have become more diverse. Attacking cloud servers to infect users for mining will also become a new trend. This will cause a large amount of resources to be illegally occupied, the number of victims will double, and the losses will be huge. Therefore, the detection of malicious mining is also particularly important. According to the way users often surf the Internet, it is of great significance to strengthen the detection of mining using web pages.
[0005] Existing methods for mining detection include: detection methods based on feature matching and protection methods based on rule detection; these methods have problems such as difficulty in maintaining feature libraries, complex rule settings, and high false alarm rates when used for mining detection, and cannot effectively perform mining detection. Therefore, it is still necessary to propose a mining detection method that can efficiently and accurately detect and warn of mining behavior. Summary of the invention
[0006] The purpose of the present invention is to provide a web page mining detection method and system based on machine learning to solve the problems of difficulty in identifying mining behavior and low recognition accuracy.
[0007] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a webpage mining detection method based on machine learning, the method comprising:
[0008] Collecting historical web page data and real-time web page data, extracting web page features from the historical web page data, and obtaining historical web page feature data;
[0009] Using the historical web page feature data to train a machine learning model to obtain a web page mining detection model based on machine learning;
[0010] Extracting web page feature data from the real-time web page data to obtain the real-time web page feature data, and detecting the real-time web page feature data using the web page mining detection model based on machine learning to obtain a web page mining detection result;
[0011] A webpage mining behavior warning is performed based on the webpage mining detection result.
[0012] As a further improvement of an embodiment of the present invention, the method also includes that the collection of historical web page data and real-time web page data includes collecting historical mining web page data, historical normal web page data and real-time web page data; the historical web page data and real-time web page data at least include using a web crawler to automatically capture web page data content, using traffic monitoring data to capture and record network traffic data, and using a system monitoring tool to obtain system resource usage data.
[0013] As a further improvement of an embodiment of the present invention, the method further includes that the web page feature data includes:
[0014] WebSocket connection establishment status, Web Worker usage, PostMessage / onMessage event frequency, MessageLoop event frequency, CPU usage, preset keyword usage, preset calculation function usage, and preset dynamic function usage.
[0015] As a further improvement of an embodiment of the present invention, the method further includes: extracting web page feature data includes:
[0016] Extracting the WebSocket connection establishment status: monitoring the WebSocket connection establishment status, counting the number of WebSocket connections established within a preset first time period and counting the connection duration;
[0017] Extracting the usage of the Web Worker: monitoring the creation and destruction of the Web Worker, and counting the usage frequency and duration of the Web Worker;
[0018] Extract the PostMessage / onMessage event frequency: monitor the triggering of PostMessage and onMessage events, and count the frequencies of PostMessage and onMessage events in a preset second time period;
[0019] Extracting the MessageLoop event frequency: monitoring the MessageLoop event of the browser, and counting the frequency of the MessageLoop event within a preset third time period;
[0020] Extracting the CPU usage rate: monitoring the CPU usage rate of the webpage, and calculating the average and peak values of the CPU usage rate in a preset fourth time period;
[0021] Extracting the usage of the preset keywords: statically analyzing the webpage source code, using regular expressions to match the preset keywords, and counting the frequency and position of the keywords;
[0022] Extract the usage of the preset calculation function: perform static analysis on the webpage source code, use regular expressions to match the preset calculation function, and count the calling frequency and location of the calculation function; the preset calculation function includes at least Cryptonight function, SHA256 function, hash function, wasmwrapper function, and WebAssemblyscrypt function;
[0023] Extract the usage of the preset dynamic function: perform static analysis on the webpage source code, and count the frequency and location of the preset dynamic function calls; the preset dynamic function at least includes setTimeout function and setInterval function.
[0024] As a further improvement of an embodiment of the present invention, the method further includes: the data preprocessing includes:
[0025] The extracted preset web page feature data is subjected to data cleaning, feature standardization, feature coding and feature selection; the data cleaning includes removing duplicates, filling missing values and processing outliers; the feature standardization at least includes standardizing the numerical features with a mean of 0 and a standard deviation of 1; the feature coding at least includes one-hot coding or label coding of categorical features; the feature selection at least includes model training feature selection using principal component analysis or Lasso regression method.
[0026] As a further improvement of an embodiment of the present invention, the method further includes: the machine learning model includes,
[0027] Select any one from the random forest model, support vector machine model, gradient boosted tree model, and neural network model as the machine learning model.
[0028] As a further improvement of an embodiment of the present invention, the method further includes that the webpage mining behavior warning based on the webpage mining detection result includes:
[0029] When the web mining detection result shows a normal web page, no warning will be issued; when the web mining detection result shows a high-risk web page, the administrator or user will be notified via email or text message that there is web mining behavior and the web page will be blocked from loading to protect the web page, thereby implementing a warning for web mining behavior; when the web mining detection result shows a medium-risk web page, the administrator or user will be notified via email or text message that there is a risk on the web page.
[0030] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides a webpage mining detection system based on machine learning, the system comprising:
[0031] Data collection module, used to collect historical web page data and real-time web page data;
[0032] The feature extraction module is used to extract web page feature data from historical web page data to obtain web page feature data; extract web page features from real-time web page data to obtain real-time web page feature data;
[0033] A model training module is used to train a machine learning model using historical web page feature data to obtain a web page mining detection model based on machine learning;
[0034] A real-time detection module is used to detect real-time web page feature data using a web page mining detection model based on machine learning;
[0035] The alarm processing module is used to issue early warning of web mining behavior based on the web mining detection results.
[0036] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, characterized in that the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the web page mining detection method based on machine learning as described in any one of the above items are implemented.
[0037] In order to achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps in the web page mining detection method based on machine learning as described in any one of the above items are implemented.
[0038] Compared with the prior art, the webpage mining detection method based on machine learning provided by the present invention realizes efficient, accurate and real-time detection and protection of webpage mining behavior by automatically learning and extracting features from a large amount of data through the use of machine learning algorithm training model; compared with the traditional method, the method can adapt to different types of mining scripts, reduce false positives and false negatives, and timely detect and prevent mining behaviors by real-time monitoring and analysis of webpage traffic, thereby protecting the performance and security of user devices; it also optimizes resource usage, reduces system performance degradation and power waste caused by mining, and improves the user's browsing experience; in addition, the crawler system based on Node.js and Chrome remote debugging protocol realizes automatic feature extraction, reduces manual intervention, and improves the efficiency and stability of the system; the machine learning model can continuously learn and update according to the emerging mining behavior, maintain the detection capability of the latest mining technology, and has good scalability and adaptability; by comprehensively monitoring webpage traffic and system resource usage, the security of webpage browsing is enhanced, and an alarm is issued in time when mining behavior is detected, and the detection results are displayed through the user interface, so that users can understand the security status of the device and improve their security awareness. The technical solution of the present invention not only improves system performance and user experience, but also has good scalability and security, and has broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is an overall flow chart of a webpage mining detection method based on machine learning described in the present invention.
[0040] Figure 2 It is a flow chart of constructing a web page mining detection model based on machine learning in a web page mining detection method based on machine learning described in the present invention.
[0041] Figure 3 This is a system architecture diagram of a web page mining detection system based on machine learning described in the present invention. DETAILED DESCRIPTION
[0042] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.
[0043] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0044] In the first embodiment of the present invention, the present invention provides a webpage mining detection method based on machine learning, such as Figures 1-2 As shown, the method includes,
[0045] S1: Collect historical web page data and real-time web page data;
[0046] In a specific embodiment of the present invention, "historical web page data" includes historical mining web page data and historical normal web page data. Historical mining web page data refers to web page data records of mining scripts or codes embedded in past web pages; historical normal web page data refers to web page data records without mining scripts embedded.
[0047] In a specific embodiment of the present invention, "web page data" includes web page data content, network traffic data and system resource usage data.
[0048] In a specific embodiment of the present invention, the "system resource usage data" includes at least CPU usage, number of CPU cores and occupancy, CPU cache usage, and memory occupancy.
[0049] In a specific embodiment of the present invention, collecting web page data includes using web crawler technology to automatically capture web page content on the Internet, using traffic monitoring tools to capture and record network traffic data, and using system monitoring data to obtain system resource usage data.
[0050] In a specific embodiment of the present invention, using web crawler technology to automatically crawl web page content on the Internet includes building a crawler system based on the Chrome remote debugging protocol through a Node.js module, the Node.js module includes a chrome-launcher unit, a chrome-remote-interface unit and a request unit, using the chrome-launcher unit to start the Chrome browser, using the chrome-remote-interface unit to use the Google Chrome debugging protocol to interact with the web page, and using the request unit to send an HTTP request.
[0051] In a specific embodiment of the present invention, the chrome-remote-interface unit includes a Page domain, a Network domain, a Profiler domain and a Debugger domain; the Page domain is used to load and navigate pages, the Network domain is used to monitor and control network requests, the Profiler domain is used for performance analysis, and the Debugger domain is used to set breakpoints and control script execution.
[0052] In a specific implementation scenario of the present invention, collecting historical web page data and real-time web page data includes: using a crawler system based on the Chrome remote debugging protocol to automatically crawl the web page content of historical web pages and real-time web pages; using a Wireshark traffic monitoring tool to capture and record network traffic data; using top or htop system monitoring tools to record CPU usage, CPU core number and occupancy, CPU cache usage, memory occupancy and other system resource usage.
[0053] S2: extracting web page feature data from historical web page data and performing data preprocessing to obtain historical web page feature data;
[0054] In a specific embodiment of the present invention, "web page feature data" includes WebSocket connection establishment status, Web Worker usage, PostMessage / onMessage event frequency, MessageLoop event frequency, CPU usage, preset keyword usage, preset calculation function usage, and preset dynamic function usage.
[0055] In a specific embodiment of the present invention, "extracting web page feature data" includes:
[0056] Extracting the WebSocket connection establishment status: monitoring the WebSocket connection establishment status, counting the number of WebSocket connections established in the first time period and counting the connection duration;
[0057] Extracting the usage of the Web Worker: monitoring the creation and destruction of the Web Worker, and counting the usage frequency and duration of the Web Worker;
[0058] Extract the PostMessage / onMessage event frequency: monitor the triggering of PostMessage and onMessage events, and count the frequencies of PostMessage and onMessage events within a preset second time period;
[0059] Extracting the MessageLoop event frequency: monitoring the MessageLoop event of the browser, and counting the frequency of the MessageLoop event within a preset third time period;
[0060] Extracting the CPU usage rate: monitoring the CPU usage rate of the webpage, and calculating the average and peak values of the CPU usage rate in a preset fourth time period;
[0061] Extracting the usage of the preset keywords: statically analyzing the webpage source code, using regular expressions to match the preset keywords, and counting the frequency and position of the keywords;
[0062] Extract the usage of the preset calculation function: perform static analysis on the webpage source code, use regular expressions to match the preset calculation function, and count the calling frequency and location of the calculation function; the preset calculation function includes at least Cryptonight function, SHA256 function, hash function, wasmwrapper function, and WebAssemblyscrypt function;
[0063] Extract the usage of the preset dynamic function: perform static analysis on the webpage source code, and count the frequency and location of the preset dynamic function calls; the preset dynamic function at least includes setTimeout function and setInterval function.
[0064] In a specific embodiment of the present invention, the criteria for determining webpage mining using the characteristic data of each webpage are:
[0065] The criterion for determining the WebSocket connection establishment is: if a webpage establishes multiple WebSocket connections within a preset first time period, and the duration of these connections exceeds the preset first connection time, it is determined that the webpage may be a mining webpage;
[0066] The criterion for judging the usage of Web Workers is: if the usage frequency and duration of Web Workers used by a web page are both greater than the preset first usage frequency and the preset second duration, it means that a large number of computing tasks may be performed.
[0067] Determine that the webpage may be a mining webpage;
[0068] The judgment standard of the frequency of PostMessage / onMessage events is: if the usage frequency of the PostMessage / onMessage event in the preset second time period is higher than the preset second usage frequency, it is determined that the webpage may be a mining webpage;
[0069] The judgment standard of the MessageLoop event frequency is: if the usage frequency of the browser's MessageLoop event in the preset third time period is higher than the third preset usage frequency, it is determined that the webpage may be a mining webpage;
[0070] The CPU usage rate judgment standard is: if the CPU usage rate is continuously higher than the first preset usage rate within the preset fourth time period, it is determined that the webpage may be a mining webpage;
[0071] The judgment criteria for the usage of the preset keywords are: if the number of times the preset keywords are used in the webpage code is higher than the preset keyword usage threshold, it is determined that the webpage may be a mining webpage;
[0072] The judgment standard of the usage of the preset calculation function is: if the number of times the preset calculation function is used in the webpage code is higher than the preset calculation function usage threshold, it is determined that the webpage may be a mining webpage;
[0073] The judgment standard of the usage of the preset dynamic function is: if the number of times the preset dynamic function is used in the webpage code is higher than the preset dynamic function usage threshold, it is determined that the webpage may be a mining webpage.
[0074] In a specific embodiment of the present invention, the criterion for comprehensively determining whether a web page is a mining web page using web page feature data is:
[0075] If all the characteristic data of a web page are lower than the preset value, it is judged as a normal web page;
[0076] If the number of features exceeding the preset threshold in the webpage feature data exceeds the preset upper limit, it indicates that there is a high suspicion of mining, and the webpage is judged as a high-risk webpage;
[0077] If there are features in the web page feature data that exceed the preset threshold and the number of features that exceed the preset threshold is lower than the preset upper limit, it is determined to be a medium-risk web page.
[0078] It should be noted that Websocket is a protocol that allows both parties to communicate in full-duplex on a single TCP connection (the two parties can actively exchange data with each other). At the same time, the two parties only need to complete one authentication to establish a persistent connection, making the communication between the server and the client more convenient. Compared with the three-way handshake connection of each request of the HTTP protocol, the persistent connection of the Websocket protocol avoids the redundant header information generated by multiple establishment of request-response message pairs, saves a lot of bandwidth resources, and ensures the real-time nature of communication; and most malicious mining websites use the Websocket protocol to establish a connection between the victim and the attacker to facilitate the transmission of information, so the number of Websocket connections is used as one of the characteristics to identify malicious mining websites.
[0079] It should be noted that JavaScript language uses a single-threaded mode, that is, all tasks are completed on one thread and executed in sequence; when a task takes a long time, no matter how simple the subsequent tasks are, they will be delayed, that is, blocked; Web Worker allows the creation of Worke threads attached to the main thread. These attached threads can accept and process tasks assigned by the main thread. This Worker thread is independent of other scripts and runs in the background; for some tasks with complex calculations and long delays, this thread can be used to run in the background, so that the main JavaScript thread used for interaction will not be blocked or slowed down; most malicious mining websites use Worker threads, because they mainly use the victim's resources to perform large amounts of data calculations. If only the main thread is used for calculations, the web page interface will generally respond slowly, resulting in a poor user experience and giving up browsing or causing user suspicion. Therefore, the use of Web Workers is used as one of the features for identifying malicious mining websites.
[0080] It should be noted that when the main thread creates its subsidiary threads to avoid congestion, it generally exchanges data between threads through Postmessage and onmessage events; the main thread sends information to the Worker thread by calling the worker.postMessage() method, and listens to the Worker thread through worker.onmessage to receive its returned information. Since malicious mining websites need to submit results while performing a large amount of calculations, there will be frequent data exchanges between the child thread and the main thread, that is, more Postmessage and onmessage events will occur, so Postmessage and onmessage events are used as one of the features for identifying malicious mining websites.
[0081] It should be noted that the browser is a multi-process structure design, each process has many threads, and the interaction between multiple threads is carried out through Messageloop events. When the main thread creates multiple child threads to process tasks, the child threads frequently interact through Messageloop events to jointly calculate and obtain results. Therefore, Messageloop events are selected as one of the features for identifying malicious mining websites.
[0082] It should be noted that mining websites usually occupy a large amount of users' CPU resources for mining calculations. Obtaining the CPU usage rate is also discriminative for identifying malicious mining; therefore, CPU usage rate is used as one of the features for identifying malicious mining websites.
[0083] It should be noted that by observing the existing mining scripts, it can be found that scripts belonging to the same provider have similar code keywords; therefore, regular expressions can be used to match whether the web page code segment contains preset keywords for identification and judgment.
[0084] It should be noted that the consensus mechanism in the Bitcoin system enables the system to maintain a unified ledger, which mainly uses hash functions to solve random number operations; with the development of technology, in addition to the SHA-256 algorithm, the CryptoNigh algorithm is also often used to achieve consensus; in addition, mining requires downloading a payload, generally using the wasm module; in combination with the above characteristics, the present invention selects "Cryptonight, SHA256, hash, wasmwrapper, WebAssemblyscrypt" as the five keywords to match the function name to identify whether there is a calculation function with the above characteristics.
[0085] It should also be noted that the existence of dynamic execution functions and generation functions allows attackers to hide the code they want to hide in one or more strings after processing, and restore it to its original state and execute it during operation to evade some detection methods; detection under this method is also particularly important, and mining websites can be identified by counting the number of setTimeout() and setInterval().
[0086] In a specific embodiment of the present invention, "data preprocessing" includes data cleaning, feature standardization, feature encoding and feature selection of the extracted preset web page feature data.
[0087] In a specific embodiment of the present invention, "data cleaning" includes removing duplicates, filling missing values with 0, and processing outliers.
[0088] In a specific embodiment of the present invention, "feature standardization" at least includes standardizing the numerical features to a mean of 0 and a standard deviation of 1.
[0089] In a specific embodiment of the present invention, “feature encoding” at least includes one-hot encoding or label encoding of categorical features.
[0090] In a specific embodiment of the present invention, "feature selection" at least includes using principal component analysis or Lasso regression method to perform model training feature selection.
[0091] In a specific embodiment of the present invention, the extracted web page feature data forms a data set in CSV format, which is expressed as: [unique identifier of the data point, WebSocket connection establishment status, Web Worker usage, PostMessage / onMessage event frequency, MessageLoop event frequency, CPU usage, preset keyword usage, preset calculation function usage, preset dynamic function usage, label data].
[0092] In a specific implementation scenario of the present invention, the unique identifier of the data point is a continuous value, which is used to distinguish web pages with different feature data sources; WebSocket connection establishment, Web Worker usage, PostMessage / onMessage event frequency, MessageLoop event frequency, CPU usage, preset keyword usage, preset calculation function usage, and preset dynamic function usage are all categorical data, which are used to indicate whether a web page is a mining web page based on the evaluation criteria of each web page feature data. The label data is a categorical feature, which is used to indicate whether it is a mining web page. The extracted web page feature data format example is:
[0093] [1,1,0,1,1,0,1,1,1,1,1].
[0094] S3: Use historical web page feature data to train a machine learning model to obtain a web page mining detection model based on machine learning.
[0095] In a specific embodiment of the present invention, the "machine learning model" includes at least one of the four machine learning models: random forest model, support vector machine model, gradient boosting tree model, and neural network model.
[0096] In a specific embodiment of the present invention, "using historical web page feature data to train a machine learning model" includes:
[0097] Step 1: Divide the historical web page feature data into training set, validation set and test set;
[0098] Step 2: Select a suitable machine learning model for model training;
[0099] Step 3: Initialize the machine learning model parameters, train the model using the training set data, and adjust the weights to minimize the loss function;
[0100] Step 4: Use cross-validation and grid search to tune model hyperparameters, such as learning rate, number of trees, tree depth, etc.
[0101] Step 5: Use the validation set to evaluate the model performance, check the generalization ability of the model, and observe the performance difference between the training set and the validation set. If the performance of the training set is much higher than that of the validation set, overfitting may have occurred. In this case, regularization or other strategies to prevent overfitting need to be considered.
[0102] Step 6: Evaluate the final model using the test set.
[0103] In a specific implementation scenario of the present invention, the random forest model is trained using web page feature data, including:
[0104] Step 1: Divide the historical web page feature data into 70% training set, 15% validation set, and 15% test set;
[0105] Step 2: Select the random forest model for training;
[0106] Step 3: Initialize the parameters of the random forest model, train the random forest model using the training set data, and adjust the weights to minimize the loss function; the parameters of the random forest model include setting n_estimators: the number of trees, initially 100; max_depth: the maximum depth of the tree, initially None (i.e., no depth limit); random_state: random seed to ensure the repeatability of the results.
[0107] Step 4: Based on the performance of the validation set, adjust the model parameters to improve performance; common parameter adjustments include: increasing n_estimators, limiting max_depth, adjusting min_samples_split and
[0108] min_samples_leaf; You can use Grid Search or Random Search to automatically adjust the parameters.
[0109] Step 5: Use the test set to evaluate the final performance of the model.
[0110] It should be noted that when selecting a suitable machine learning model for model training, the following main factors and steps can be used to select the machine learning model:
[0111] Factor 1: Question type;
[0112] For classification problems such as detecting whether a web page is mining, applicable models include logistic regression, decision tree, random forest, support vector machine (SVM), K-nearest neighbor (KNN), neural network, etc.
[0113] Factor 2: Data characteristics, including data volume, data dimension, data sparsity and other data characteristics;
[0114] When the amount of data is large, random forest and neural network usually perform better and can be preferred; when the amount of data is small, logistic regression and support vector machine (SVM) will be more appropriate.
[0115] When the data is high-dimensional, support vector machine (SVM) and neural network are preferred; when the data is low-dimensional, decision tree and random forest perform better and can be selected.
[0116] When the data is sparse, logistic regression and support vector machine (SVM) usually perform better and can be preferred.
[0117] Factor 3: Computing resource requirements, including training time, memory requirements, and other computing resource requirements;
[0118] When the training time requirement is high and fast training is needed, choose from logistic regression and decision tree training; when the training time requirement is not high and the training time is long, you can choose random forest and neural network as the machine learning model.
[0119] When the memory requirement is not high, logistic regression and decision tree can be selected as machine learning models; when the memory requirement is high, support vector machine (SVM) and neural network are given priority as machine learning models.
[0120] S4: extracting web page feature data from real-time web page data and performing data preprocessing to obtain real-time web page feature data, detecting the real-time web page feature data using a web page mining detection model based on machine learning, and obtaining web page mining detection results.
[0121] In a specific embodiment of the present invention, Figure 2 As shown, "using a web mining detection model based on machine learning to detect real-time web page feature data" includes obtaining real-time web page data such as web page content, web page traffic data, web page source code data, network requests, CPU usage, memory usage, etc. of the real-time web page; extracting preset web page features from the real-time web page data and performing data cleaning to obtain real-time web page feature data; inputting the real-time web page feature data into a web mining detection model based on machine learning, and outputting the result as the web mining detection result.
[0122] S5: Provide early warning of mining behavior based on web mining detection results.
[0123] In a specific embodiment of the present invention, "mining behavior warning based on webpage mining detection results" includes:
[0124] When the web mining detection result shows a normal web page, no warning will be issued; when the web mining detection result shows a high-risk web page, the administrator or user will be notified via email or text message that there is web mining behavior and the web page will be blocked from loading to protect the web page, thereby implementing a warning for web mining behavior; when the web mining detection result shows a medium-risk web page, the administrator or user will be notified via email or text message that there is a risk on the web page.
[0125] In a specific embodiment of the present invention, "other communication methods" include at least instant messaging platforms, such as WeChat, WhatsApp, Telegram, Slack, Discord, etc.; forums and online communities, such as Reddit, StackOverflow, Quora; social media platforms, such as Twitter, Facebook, Instagram, LinkedIn, etc.; telephone communication; broadcast communication.
[0126] In the second embodiment of the present invention, the present invention provides a webpage mining detection system based on machine learning, such as Figure 3 As shown, the system comprises:
[0127] Data collection module, used to collect historical web page data and real-time web page data;
[0128] The feature extraction module is used to extract web page feature data from historical web page data to obtain historical web page feature data; and is used to extract web page feature data from real-time web page data to obtain real-time web page feature data;
[0129] A model training module is used to train a machine learning model using historical web page feature data to obtain a web page mining detection model based on machine learning;
[0130] A real-time detection module is used to detect real-time web page feature data using a web page mining detection model based on machine learning;
[0131] The alarm processing module is used to issue mining behavior warnings based on web mining detection results.
[0132] In a specific embodiment of the present invention, the "data collection module" includes a web crawler unit, a traffic monitoring unit, and a system monitoring unit; the web crawler unit is used to automatically crawl web page content; the traffic monitoring unit is used to capture and record network traffic data; the system monitoring unit is used to record system resource usage such as CPU usage and memory occupancy.
[0133] In a specific embodiment of the present invention, the "feature extraction module" includes a data preprocessing unit and a feature extraction unit; the data preprocessing unit is used to perform data cleaning on the original data; and the feature extraction unit is used to extract web page feature data to generate a feature vector.
[0134] In a specific embodiment of the present invention, the "model training module" includes a data set partitioning unit, a machine learning algorithm unit, and a model evaluation unit; the data set partitioning unit is used to divide the data set into a training set, a validation set, and a test set in proportion; the machine learning algorithm unit is used to train a web mining detection model based on machine learning; and the model evaluation unit is used to evaluate the model performance of the web mining detection model based on machine learning.
[0135] In a specific embodiment of the present invention, the "real-time detection module" includes a traffic monitoring unit, a model prediction unit, and an abnormality blocking unit; the traffic monitoring unit is used to monitor network traffic in real time and extract features; the model prediction unit is used to use a trained machine learning-based web mining detection model to predict real-time data and determine whether there is mining behavior; the abnormality blocking unit is used to prevent web page loading or take other protective measures when mining behavior is detected.
[0136] In a specific embodiment of the present invention, the "alarm processing module" includes a log recording unit, an alarm unit, and a user-visible unit; the log recording unit is used to record all detected mining behaviors and their detailed information; the alarm unit is used to notify the administrator or user via email, SMS or other means; the user-visible unit is used to provide a graphical interface to display the detection results and system status.
[0137] In a third embodiment of the present invention, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the web page mining detection method based on machine learning as described in any one of the above items are implemented.
[0138] In a fourth embodiment of the present invention, the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the web page mining detection method based on machine learning as described in any one of the above items.
[0139] In summary, the web mining detection method based on machine learning provided by the present invention realizes efficient, accurate and real-time detection and protection of web mining behavior by using machine learning algorithm training model to automatically learn and extract features from a large amount of data; compared with traditional methods, this method can adapt to different types of mining scripts, reduce false positives and false negatives, and timely detect and prevent mining behaviors by real-time monitoring and analysis of web traffic, thereby protecting the performance and security of user devices; it also optimizes resource usage, reduces system performance degradation and power waste caused by mining, and improves user browsing experience; in addition, the crawler system based on Node.js and Chrome remote debugging protocol realizes automatic feature extraction, reduces manual intervention, and improves system efficiency and stability; the machine learning model can continuously learn and update according to the emerging mining behavior, maintain the ability to detect the latest mining technology, and has good scalability and adaptability; by comprehensively monitoring web traffic and system resource usage, the security of web browsing is enhanced, and an alarm is issued in time when mining behavior is detected, and the detection results are displayed through the user interface, so that users can understand the security status of the device and improve their safety awareness. The technical solution of the present invention not only improves system performance and user experience, but also has good scalability and security, and has broad application prospects and market value.
[0140] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the modules described above can refer to the corresponding process in the aforementioned method implementation, and will not be repeated here.
[0142] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present implementation scheme.
[0143] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0144] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0145] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned implementation modes, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned implementation modes, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various implementation modes of the present application.
Claims
1. A webpage mining detection method based on machine learning, characterized in that: include, Collecting historical web page data and real-time web page data, extracting web page feature data from the historical web page data to obtain historical web page feature data; Using the historical web page feature data to train a machine learning model to obtain a web page mining detection model based on machine learning; Extracting web page feature data from the real-time web page data to obtain the real-time web page feature data, and using the web page mining detection model based on machine learning to detect the real-time web page feature data to obtain a web page mining detection result; A webpage mining behavior warning is performed based on the webpage mining detection result.
2. The webpage mining detection method based on machine learning according to claim 1, characterized in that: The collection of historical web page data and real-time web page data includes collecting historical mining web page data, historical normal web page data and real-time web page data; the historical web page data and real-time web page data at least include web page data content automatically captured by web crawlers, network traffic data captured and recorded by traffic monitoring data, and system resource usage data obtained by system monitoring tools.
3. The webpage mining detection method based on machine learning according to claim 1, characterized in that: The web page feature data includes: WebSocket connection establishment status, Web Worker usage, PostMessage / onMessage event frequency, MessageLoop event frequency, CPU usage, preset keyword usage, preset calculation function usage, and preset dynamic function usage.
4. The webpage mining detection method based on machine learning according to claim 3 is characterized in that: The extracting of web page feature data comprises: Extracting the WebSocket connection establishment status: monitoring the WebSocket connection establishment status, counting the number of WebSocket connections established within a preset first time period and counting the connection duration; Extract the usage of the Web Worker: monitor the creation and destruction of the Web Worker, and count the usage frequency and duration of the Web Worker; Extract the PostMessage / onMessage event frequency: monitor the triggering of PostMessage and onMessage events, and count the frequencies of PostMessage and onMessage events within a preset second time period; Extracting the MessageLoop event frequency: monitoring the MessageLoop event of the browser, and counting the frequency of the MessageLoop event in a preset third time period; Extracting the CPU usage rate: monitoring the CPU usage rate of the webpage, and counting the average and peak values of the CPU usage rate in a preset fourth time period; Extracting the usage of the preset keywords: statically analyzing the webpage source code, using regular expressions to match the preset keywords, and counting the frequency and position of the keywords; Extract the usage of the preset calculation function: perform static analysis on the webpage source code, use regular expressions to match the preset calculation function, and count the calling frequency and location of the calculation function; the preset calculation function includes at least Cryptonight function, SHA256 function, hash function, wasmwrapper function, and WebAssemblyscrypt function; Extract the usage of the preset dynamic function: perform static analysis on the webpage source code, and count the frequency and location of the preset dynamic function calls; the preset dynamic function at least includes setTimeout function and setInterval function.
5. The webpage mining detection method based on machine learning according to claim 1, characterized in that: The data preprocessing includes: The extracted preset web page feature data is subjected to data cleaning, feature standardization, feature coding and feature selection; the data cleaning includes removing duplicates, filling missing values and processing outliers; the feature standardization at least includes standardizing the numerical features with a mean of 0 and a standard deviation of 1; the feature coding at least includes one-hot coding or label coding of categorical features; the feature selection at least includes model training feature selection using principal component analysis or Lasso regression method.
6. The webpage mining detection method based on machine learning according to claim 1, characterized in that: The machine learning model is: Any of the random forest model, support vector machine model, gradient boosted tree model, and neural network model.
7. The webpage mining detection method based on machine learning according to claim 1, characterized in that: The webpage mining behavior early warning based on the webpage mining detection result includes: When the webpage mining detection result shows that the webpage is normal, no warning will be issued; When the webpage mining detection result shows a high-risk webpage, the administrator or user is notified of the webpage mining behavior through email or SMS and the webpage is blocked from loading to protect the webpage, thus providing an early warning of webpage mining behavior. When the webpage mining detection result shows that the webpage is medium-risk, the administrator or user will be notified via email or SMS that the webpage is risky.
8. A webpage mining detection system based on machine learning, characterized in that: include, Data collection module, used to collect historical web page data and real-time web page data; The feature extraction module is used to extract web page feature data from historical web page data to obtain web page feature data; extract web page feature data from real-time web page data to obtain real-time web page feature data; A model training module is used to train a machine learning model using historical web page feature data to obtain a web page mining detection model based on machine learning; A real-time detection module is used to detect real-time web page feature data using a web page mining detection model based on machine learning; The alarm processing module is used to issue early warning of web mining behavior based on the web mining detection results.
9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the web page mining detection method based on machine learning as described in any one of claims 1-7 are implemented.
10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps in the web page mining detection method based on machine learning as described in any one of claims 1 to 7 are implemented.
