Data cleaning system
By designing a data cleaning system that combines machine learning algorithms and log analysis, the problem that existing anti-swiping measures cannot effectively distinguish between normal requests and malicious requests is solved, and more efficient malicious request identification and filtering is achieved, and the normal operation of the medical registration process is protected.
Patent Information
- Application Number
- CN202411801336.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-11
AI Technical Summary
The existing anti-swiping measures are insufficient in identifying and filtering malicious requests, and they cannot effectively distinguish between normal requests and malicious requests, resulting in accidentally hurting normal users and unable to effectively protect the registration process.
A data cleaning system is designed to receive requests through the Internet request access module, and uses the request feature analysis module, behavior pattern recognition module and real-time monitoring and response module to identify and filter malicious requests in combination with machine learning algorithms and log analysis.
The system can more efficiently identify and filter malicious requests, reduce misjudgment of normal users, improve the usability and user experience of the system, and ensure the fair distribution of medical resources.
Smart Images
Figure CN119807610A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data cleaning, and in particular to a data cleaning system. Background Art
[0002] With the development of Internet technology, more and more medical services are provided through online platforms, among which medical registration is one of the most common services. Through the Internet platform, patients can make appointments with doctors conveniently and quickly, which greatly improves the efficiency and convenience of medical services. However, due to the scarcity of medical resources, malicious requests (such as scalpers) have become a serious problem. Scalpers use automated tools to request registration in batches, making it difficult for normal patients to make appointments, thus affecting the fair distribution of medical resources.
[0003] The scarcity of medical resources will bring some challenges: 1. Scarcity: Medical resources, especially high-quality medical resources, are very scarce, resulting in a much greater demand for registration than supply. 2. Fairness: Scalpers use automated tools to grab appointments in batches, undermining the fairness of registration and making it difficult for ordinary patients to make appointments for the medical services they need.
[0004] However, the existing anti-swiping measures have some shortcomings: 1. Verification code: Although the verification code can play a certain protective role, it is easy to be cracked and the user experience is poor. 2. Frequency limit: Overly simple frequency limit may accidentally hurt normal users and reduce the availability of the system. 3. Lack of intelligent identification: The existing measures lack in-depth analysis of request behavior and cannot accurately distinguish between normal requests and malicious requests. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a data cleaning system to more efficiently identify and filter malicious requests and protect the normal registration process from interference.
[0006] In a first aspect, an embodiment of the present invention provides a data cleaning system, which includes: an Internet request access module, a request feature analysis module, a behavior pattern recognition module, a log recording and analysis module, and a real-time monitoring and response module; the Internet request access module, the request feature analysis module, and the real-time monitoring and response module are connected in sequence, the Internet request access module, the behavior pattern recognition module, and the real-time monitoring and response module are connected in sequence, and the Internet request access module, the log recording and analysis module, and the real-time monitoring and response module are connected in sequence; the Internet request access module is used to receive multiple requests from the Internet, classify the multiple requests, and send the registration request to the request feature analysis module and the behavior pattern recognition module; wherein the request includes at least one of the following: a registration request, an inquiry request, or an appointment request; the request The feature analysis module is used to extract the features of the registration request, analyze the features of the registration request, and determine whether the registration request is a normal request or a malicious request; wherein the features of the registration request include at least one of the following: IP address, user agent or access frequency; the behavior pattern recognition module is used to determine whether the registration request is a normal request or a malicious request through a pre-trained machine learning model; the log recording and analysis module is used to record the log information of each request, and optimize the algorithms of the behavior pattern recognition module and the log recording and analysis module based on the log information; the real-time monitoring and response module is used to execute specified response measures when the request feature analysis module or the behavior pattern recognition module determines that the registration request is a malicious request; wherein the response measures include at least one of the following: restricting IP access, adding verification code, recording logs or notifying administrators.
[0007] In an optional embodiment of the present application, the above-mentioned Internet request access module includes: a request receiving submodule, a request parsing submodule, a request classification submodule, and a request forwarding submodule; the request receiving submodule is used to listen to the designated port and receive multiple requests from the Internet; the request parsing submodule is used to parse multiple requests and determine key information of multiple requests; wherein the key information includes at least one of the following: request type or request parameter; the request classification submodule is used to classify multiple requests based on the key information; the request forwarding submodule is used to send the registration request to the request feature analysis module and the behavior pattern recognition module.
[0008] In an optional embodiment of the present application, the above-mentioned Internet request access module also includes: a log recording submodule; the log recording submodule is used to record multiple requests and processing results of multiple requests, and send multiple requests and processing results of multiple requests to the log recording and analysis module.
[0009] In an optional embodiment of the present application, the above-mentioned request feature analysis module includes: a feature extraction submodule, a feature analysis submodule, and a request classification submodule; the feature extraction submodule is used to extract the features of the registration request; the feature analysis submodule is used to analyze the features of the registration request to determine whether the features of the registration request meet the features of a normal request; the request classification submodule is used to determine the registration request as a normal request when the features of the registration request are all features that meet the features of a normal request; when the features of the registration request include features that do not meet the features of a normal request, determine the registration request as a malicious request.
[0010] In an optional embodiment of the present application, the request feature analysis module further includes: a feature library maintenance submodule; the feature library maintenance submodule is used to maintain a feature library of normal requests and optimize feature extraction and feature analysis algorithms.
[0011] In an optional embodiment of the present application, the above-mentioned behavior pattern recognition module includes: a data acquisition sub-module, a feature engineering sub-module, a model training sub-module, and a model application sub-module; the data acquisition sub-module is used to obtain historical request data; wherein the historical requests include: normal requests and malicious requests; the feature engineering sub-module is used to extract the features of the historical request data; the features of the historical request data include at least one of the following: IP address, user agent, access frequency or request mode; the model training sub-module is used to: train a machine learning model based on the features of the historical request data; the model application sub-module is used to determine whether the registration request is a normal request or a malicious request through a machine learning model.
[0012] In an optional embodiment of the present application, the above-mentioned behavior pattern recognition module also includes: a model optimization submodule; the model optimization submodule is used to optimize the parameters of the machine learning model.
[0013] In an optional embodiment of the present application, the above-mentioned log recording and analysis module includes: a log recording sub-module, a data cleaning sub-module, a data analysis sub-module, a report generation sub-module, and a feedback optimization sub-module; the log recording sub-module is used to receive each request and the processing result of each request as the original data of the log information; the data cleaning sub-module is used to clean and pre-process the original data; the data analysis sub-module is used to perform statistical analysis on the cleaned original data; the report generation sub-module is used to generate an analysis report based on the results of the statistical analysis; the feedback optimization sub-module is used to analyze the report optimization behavior pattern recognition module and the algorithm of the log recording and analysis module.
[0014] In an optional embodiment of the present application, the above-mentioned real-time monitoring and response module includes: a real-time monitoring sub-module, an anomaly detection sub-module, a response measure sub-module, and a dynamic adjustment sub-module; the real-time monitoring sub-module is used to use a network monitoring tool to continuously monitor the request traffic of the data cleaning system; the anomaly detection sub-module is used to detect malicious requests based on preset rules or machine learning models; wherein the rules include at least one of the following: frequency analysis, behavior pattern matching or threshold detection; the response measure sub-module is used to: when a malicious request is detected, execute a specified response measure; the dynamic adjustment sub-module is used to adjust the rules based on the detection results of malicious requests.
[0015] In an optional embodiment of the present application, the above-mentioned real-time monitoring and response module also includes: an alarm notification submodule; the alarm notification submodule is used to send an alarm notification when the number of malicious requests exceeds a preset threshold.
[0016] The embodiments of the present invention bring the following beneficial effects:
[0017] An embodiment of the present invention provides a data cleaning system, which receives requests from the Internet through an Internet request access module and passes them to a subsequent processing module. The feature analysis module identifies malicious requests by analyzing features such as the IP address, user agent, and access frequency of the request. The behavior pattern recognition module is based on a machine learning algorithm and uses a training model to identify and classify the behavior patterns of normal requests and malicious requests. The real-time monitoring and response module monitors the request traffic in real time, and immediately takes filtering measures when a malicious request is detected, such as restricting IP access, adding verification codes, etc., and forwards the filtered normal request data to the back-end server. The data cleaning system in this embodiment can more efficiently identify and filter malicious requests and protect the normal registration process from interference.
[0018] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.
[0019] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A structural diagram of a data cleaning system provided by an embodiment of the present invention;
[0022] Figure 2 A schematic diagram of the structure of an Internet request access module provided by an embodiment of the present invention;
[0023] Figure 3 A schematic diagram of the structure of a request feature analysis module provided by an embodiment of the present invention;
[0024] Figure 4 A schematic diagram of the structure of a behavior pattern recognition module provided by an embodiment of the present invention;
[0025] Figure 5 A schematic diagram of the structure of a log recording and analysis module provided by an embodiment of the present invention;
[0026] Figure 6 A schematic diagram of the structure of a real-time monitoring and response module provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Currently, existing anti-fraud measures often rely on simple verification codes or frequency limits. For example, some systems require users to enter verification codes to prove that they are real people, not automated programs; some systems limit the number of requests sent from the same IP address in a short period of time. However, these measures have obvious shortcomings:
[0029] 1. Captchas are easily bypassed: Although captchas can prevent automated programs to a certain extent, the risk of captchas being cracked is increasing with the advancement of technology. For example, some automated tools can use image recognition technology to crack captchas.
[0030] 2. Frequency limiting is not accurate: Although frequency limiting can limit the request frequency, this method is not smart enough and can easily hurt normal users. For example, if a hospital suddenly has a large number of real users trying to register at the same time, the frequency limit may be triggered, causing the requests of normal users to be rejected.
[0031] 3. Lack of intelligent identification: Existing measures lack in-depth analysis and intelligent identification of request behaviors and cannot effectively distinguish between normal users and malicious requests.
[0032] In order to solve the above problems, a more efficient data cleaning device is needed to identify and filter malicious requests and protect the normal registration process from being disturbed. The ideal solution should have the following characteristics:
[0033] Intelligent identification: Accurately identify malicious requests by analyzing the behavioral characteristics of requests and combining them with machine learning algorithms.
[0034] Dynamic response: Based on the results of real-time monitoring, filter strategies are dynamically adjusted to respond to new malicious behaviors in a timely manner.
[0035] User experience: Reduce misjudgment of normal users and improve system availability and user experience.
[0036] Log analysis: record and analyze request logs, continuously optimize identification algorithms, and improve the system's defense capabilities.
[0037] In summary, with the development of Internet technology, medical registration services are facing severe security challenges. Existing simple anti-number-swiping measures are not enough to deal with increasingly complex malicious requests. Therefore, it is particularly important to develop an intelligent and efficient malicious request identification and filtering system.
[0038] Based on this, a data cleaning system provided in an embodiment of the present invention specifically provides a data cleaning system that can identify and filter malicious requests. By combining advanced data analysis technology and machine learning algorithms, the security of the system can be significantly improved and the fair allocation of medical resources can be ensured.
[0039] To facilitate understanding of this embodiment, a data cleaning system disclosed in an embodiment of the present invention is first introduced in detail.
[0040] Embodiment 1:
[0041] The present invention provides a data cleaning system. Figure 1 A structural diagram of a data cleaning system shown, the data cleaning system includes: an Internet request access module, a request feature analysis module, a behavior pattern recognition module, a log recording and analysis module, and a real-time monitoring and response module;
[0042] The Internet request access module, the request feature analysis module, and the real-time monitoring and response module are connected in sequence; the Internet request access module, the behavior pattern recognition module, and the real-time monitoring and response module are connected in sequence; the Internet request access module, the log recording and analysis module, and the real-time monitoring and response module are connected in sequence;
[0043] The Internet request access module is used to receive multiple requests from the Internet, classify the multiple requests, and send the registration request to the request feature analysis module and the behavior pattern recognition module; wherein the request includes at least one of the following: a registration request, an inquiry request, or an appointment request;
[0044] The request feature analysis module is used to extract the features of the registration request, analyze the features of the registration request, and determine whether the registration request is a normal request or a malicious request; wherein the features of the registration request include at least one of the following: IP address, user agent or access frequency;
[0045] The behavior pattern recognition module is used to determine whether a registration request is a normal request or a malicious request through a pre-trained machine learning model;
[0046] The log recording and analysis module is used to record the log information of each request and optimize the algorithms of the behavior pattern recognition module and the log recording and analysis module based on the log information;
[0047] The real-time monitoring and response module is used to execute specified response measures when the request feature analysis module or the behavior pattern recognition module determines that the registration request is a malicious request; wherein the response measures include at least one of the following: restricting IP access, adding verification codes, recording logs or notifying administrators.
[0048] like Figure 1 As shown, the Internet request access module in this embodiment is used to receive multiple requests from the Internet and pass these requests to subsequent processing modules. The request feature analysis module identifies malicious requests by analyzing the characteristics of the request such as the IP address, user agent, and access frequency. The behavior pattern recognition module is based on a machine learning algorithm and uses a training model to identify and classify the behavior patterns of normal requests and malicious requests. The real-time monitoring and response module monitors the request traffic in real time and immediately takes filtering measures when a malicious request is detected, such as restricting IP access, adding verification codes, etc. It can also record the logs of malicious requests or notify the administrator to process them; and send normal requests to the Internet medical platform for subsequent medical procedures. The log recording and analysis module records the log information of all requests and analyzes them to continuously optimize the recognition algorithm.
[0049] An embodiment of the present invention provides a data cleaning system, which receives requests from the Internet through an Internet request access module and passes them to a subsequent processing module. The feature analysis module identifies malicious requests by analyzing features such as the IP address, user agent, and access frequency of the request. The behavior pattern recognition module is based on a machine learning algorithm and uses a training model to identify and classify the behavior patterns of normal requests and malicious requests. The real-time monitoring and response module monitors the request traffic in real time, and immediately takes filtering measures when a malicious request is detected, such as restricting IP access, adding verification codes, etc., and forwards the filtered normal request data to the back-end server. The data cleaning system in this embodiment can more efficiently identify and filter malicious requests and protect the normal registration process from interference.
[0050] Embodiment 2:
[0051] This embodiment provides another data cleaning system, which is implemented on the basis of the above embodiment, focusing on the description of the Internet request access module. The Internet request access module is one of the core components of the entire system, responsible for receiving multiple requests from the Internet and passing these requests to subsequent processing modules. The Internet request access module in this embodiment is specially optimized for requests from the medical registration system to ensure that only legitimate registration requests can pass smoothly and enter the subsequent processing links. The main functions of the Internet request access module in this embodiment include:
[0052] 1. Request reception: Receive multiple requests from the Internet, including but not limited to registration requests, inquiry requests, appointment requests, etc.
[0053] 2. Initial Verification: Perform initial verification on the received request to ensure that the request is in the correct format and contains the necessary information.
[0054] 3. Request classification: Classify requests according to request type and distinguish between registered requests and other types of requests.
[0055] 4. Request forwarding: forward the classified requests to the corresponding processing module for further processing.
[0056] 5. Logging: Record all received requests and their processing results for subsequent analysis and auditing.
[0057] See also Figure 2A structural schematic diagram of an Internet request access module is shown, and the Internet request access module includes: a request receiving submodule, a request parsing submodule, a request classification submodule, and a request forwarding submodule; the request receiving submodule is used to listen to a designated port and receive multiple requests from the Internet; the request parsing submodule is used to parse multiple requests and determine key information of the multiple requests; wherein the key information includes at least one of the following: request type or request parameter; the request classification submodule is used to classify multiple requests based on the key information; the request forwarding submodule is used to send the registration request to the request feature analysis module and the behavior pattern recognition module.
[0058] (1) The request receiving submodule can listen to the specified port and receive HTTP / HTTPS requests from the Internet. Standard network programming techniques (such as Socket programming or HTTP library-based API) are used to receive requests.
[0059] The specific steps may be: Step 1, listening to a designated port (such as 80 or 443) to receive HTTP / HTTPS requests from the Internet. Step 2, using network programming technology to receive request messages.
[0060] (2) The request parsing submodule can parse the received request and extract key information in the request, such as request type, request parameters, etc. Use HTTP parsing library (such as libcurl, libxml2, etc.) to parse the request and extract key fields in the request.
[0061] The specific steps may be: Step 1, use the HTTP parsing library to parse the request message and extract key information in the request header and request body. Step 2, extract information such as request type and request parameters.
[0062] (3) The request classification submodule can classify requests according to the request type and distinguish between registration requests and other types of requests. The request type is determined by analyzing the request URL path and request parameters. For example, if the URL path contains " / register" or " / appointment", it is considered a registration request.
[0063] The specific steps may be: Step 1, judging the request type according to the URL path and request parameters of the request. Step 2, if it is a registered request, marking it as a registered request; otherwise, marking it as a request of other types.
[0064] (4) The request forwarding submodule can forward the classified requests to the corresponding processing module for further processing. It uses a message queue (such as RabbitMQ, Kafka, etc.) or directly calls the corresponding processing function to forward the request to the request feature analysis module and the behavior pattern recognition module.
[0065] The specific steps may be: Step 1, putting the classified registration request into the message queue. Step 2, forwarding other types of requests to the corresponding processing module.
[0066] like Figure 2 As shown, the Internet request access module also includes: a log recording submodule; the log recording submodule is used to record multiple requests and processing results of the multiple requests, and send the multiple requests and processing results of the multiple requests to the log recording and analysis module.
[0067] (5) The logging submodule can record all received requests and their processing results for subsequent analysis and auditing. Use a logging framework (such as Log4j, syslog, etc.) to record request information, including request time, request type, request parameters, response results, etc.
[0068] The specific steps may be: Step 1, recording the timestamp, request type, request parameters and other information of the request. Step 2, recording the processing result of the request, including the response code and response time.
[0069] The Internet request access module provided in this embodiment is an important part of the entire system, responsible for receiving and preliminarily processing multiple requests from the Internet. Through this module, the system can effectively classify and forward the received requests, ensuring that the registration request can be correctly identified and passed to the subsequent processing module. At the same time, the log recording function provides an important basis for the operation and maintenance and auditing of the system. Through the above detailed description, the function and implementation method of the Internet request access module can be better understood, ensuring that it can effectively prevent malicious requests in practical applications and protect the normal operation of the medical registration system.
[0070] Embodiment three:
[0071] This embodiment provides another data cleaning system, which is implemented on the basis of the above embodiment, focusing on the request feature analysis module. The request feature analysis module is one of the core components of the entire system, responsible for extracting key features from the received requests and identifying and classifying requests through these features. The goal of this module is to accurately distinguish between normal requests and malicious requests, thereby protecting the normal operation of the medical registration system. The main functions of the request feature analysis module in this embodiment include:
[0072] 1. Feature extraction: Extract various features from the request, such as IP address, user agent, access frequency, etc.
[0073] 2. Feature analysis: Analyze the extracted features to determine whether the request meets the characteristics of normal requests.
[0074] 3. Request classification: Based on the results of feature analysis, the request is classified as normal request or malicious request.
[0075] 4. Feature library maintenance: Maintain the feature library, and continuously update and optimize feature extraction and analysis algorithms.
[0076] See also Figure 3 A structural schematic diagram of a request feature analysis module is shown, and the request feature analysis module includes: a feature extraction submodule, a feature analysis submodule, and a request classification submodule; the feature extraction submodule is used to extract the features of the registration request; the feature analysis submodule is used to analyze the features of the registration request to determine whether the features of the registration request meet the features of a normal request; the request classification submodule is used to determine the registration request as a normal request when the features of the registration request are all features that meet the features of a normal request; when the features of the registration request include features that do not meet the features of a normal request, determine the registration request as a malicious request.
[0077] (1) Feature extraction submodule: responsible for extracting various features from requests. Various features can be extracted from requests. Standard network programming techniques (such as Python's http.server module) are used to extract key information from requests.
[0078] The specific steps may be: Step 1, receiving a request and parsing key information in the request, such as an IP address, a user agent, access frequency, etc. Step 2, extracting other features in the request, such as a request header, request parameters, etc.
[0079] (2) Feature analysis submodule: responsible for analyzing the extracted features. The extracted features can be analyzed to determine whether the request meets the characteristics of a normal request. The features are analyzed using pre-set rules or machine learning models. The above rules can be frequency analysis, behavior pattern matching, or threshold detection.
[0080] The specific steps may be: Step 1, analyzing the extracted features according to a pre-set rule or machine learning model. Step 2, determining whether the request meets the characteristics of a normal request.
[0081] (3) Request classification submodule: Based on the results of feature analysis, the requests are classified as normal requests or malicious requests. Based on the results of feature analysis, the requests are classified into two categories and marked with corresponding labels.
[0082] The specific steps may be: Step 1, classifying the request as a normal request or a malicious request according to the result of feature analysis. Step 2, marking the category of the request and recording the classification result.
[0083] like Figure 3As shown, the request feature analysis module also includes: a feature library maintenance submodule; the feature library maintenance submodule is used to maintain the feature library of normal requests and optimize the feature extraction and feature analysis algorithms.
[0084] (4) Feature library maintenance submodule: responsible for maintaining the feature library, updating and optimizing the feature extraction and analysis algorithms. The feature library can be maintained, and the feature extraction and analysis algorithms can be continuously updated and optimized. The feature library is updated regularly, and the feature extraction and analysis algorithms are optimized.
[0085] The specific steps may be: Step 1, regularly update the feature library and add new feature rules. Step 2, optimize the feature extraction and analysis algorithm to improve classification accuracy.
[0086] The request feature analysis module in this embodiment is the key part of the entire system responsible for identifying and filtering malicious requests. Through this module, the system can effectively extract various features in the request, and accurately distinguish normal requests from malicious requests through feature analysis and request classification functions. This module also includes a feature library maintenance function to ensure that the system can continuously optimize and update feature extraction and analysis algorithms to improve the system's defense capabilities. Through the above detailed description, we can better understand the functions and implementation methods of the request feature analysis module, ensuring that it can effectively prevent malicious requests in practical applications and protect the normal operation of the medical registration system.
[0087] Embodiment 4:
[0088] This embodiment provides another data cleaning system, which is implemented on the basis of the above embodiment, focusing on the description of the behavior pattern recognition module. The behavior pattern recognition module is an important part of the entire system, responsible for identifying and classifying the behavior patterns of requests through machine learning and data analysis technology. This module aims to distinguish normal requests from malicious requests by analyzing historical data and real-time data, thereby improving the security of the system. The main functions of the behavior pattern recognition module in this embodiment include:
[0089] 1. Data collection: Collect and organize historical request data, including normal requests and malicious requests.
[0090] 2. Feature Engineering: Extract features from request data for training and testing machine learning models.
[0091] 3. Model training: Use machine learning algorithms to train the model to identify the behavior patterns of normal and malicious requests.
[0092] 4. Model application: Apply the trained model to real-time request data for real-time recognition and classification.
[0093] 5. Model optimization: Continuously optimize model performance and improve recognition accuracy.
[0094] See also Figure 4 A structural schematic diagram of a behavior pattern recognition module is shown, and the behavior pattern recognition module includes: a data acquisition submodule, a feature engineering submodule, a model training submodule, and a model application submodule; the data acquisition submodule is used to obtain historical request data; wherein the historical requests include: normal requests and malicious requests; the feature engineering submodule is used to extract features of the historical request data; the features of the historical request data include at least one of the following: IP address, user agent, access frequency or request mode; the model training submodule is used to; train a machine learning model based on the features of the historical request data; the model application submodule is used to determine whether the registration request is a normal request or a malicious request through a machine learning model.
[0095] (1) Data collection submodule: responsible for collecting and organizing historical request data. It can collect and organize historical request data, including normal requests and malicious requests. It records request data through the logging system and database, and performs data cleaning and preprocessing.
[0096] The specific steps may be: Step 1, record the log information of all requests, including IP address, user agent, request time, request path, etc. Step 2, clean and pre-process the log information to ensure data quality.
[0097] (2) Feature Engineering Submodule: Responsible for extracting features from request data. Features in request data can be extracted for training and testing machine learning models. Multiple features are extracted from request data, such as: IP address features: historical request frequency of IP address, geographic location information, etc. User agent features: browser type, operating system, device type, etc. Access frequency features: number of requests per unit time. Request mode features: request path, request parameters, request method, etc.
[0098] The specific steps may be: Step 1, extracting multiple features from the cleaned data, such as the historical request frequency of the IP address, the user agent type, etc. Step 2, combining the extracted features into a feature vector for model training.
[0099] (3) Model training submodule: responsible for training the model using machine learning algorithms. The model can be trained using machine learning algorithms to identify the behavior patterns of normal and malicious requests. Select appropriate machine learning algorithms (such as random forests, support vector machines, neural networks, etc.) for training, and use cross-validation and other techniques to evaluate model performance.
[0100] The specific steps may be: Step 1, select a suitable machine learning algorithm (such as random forest) for training. Step 2, use cross-validation and other techniques to evaluate model performance and select the best model.
[0101] (4) Model application submodule: responsible for applying the trained model to real-time request data. The trained model can be applied to real-time request data for real-time recognition and classification. The features of the request data are extracted in real time, and the trained model is used for prediction, outputting the classification results of the request.
[0102] The specific steps may be: Step 1, extracting the features of the request data in real time. Step 2, using the trained model to make predictions and output the classification results of the request.
[0103] like Figure 4 As shown, the behavior pattern recognition module also includes: a model optimization submodule; the model optimization submodule is used to optimize the parameters of the machine learning model.
[0104] (5) Model optimization submodule: responsible for continuously optimizing model performance. It can continuously optimize model performance and improve recognition accuracy. Through continuous data collection and model training, it continuously optimizes model parameters and uses the latest data to update the model.
[0105] The specific steps may be: Step 1, continuously collect the latest request data. Step 2, use the latest data to update the model and optimize the model performance.
[0106] The behavior pattern recognition module in this embodiment is the key part of the entire system responsible for identifying and classifying request behavior patterns through machine learning technology. Through this module, the system can effectively learn the behavior patterns of normal requests and malicious requests from historical data, and apply them to real-time request data, thereby improving the system's defense capabilities. This module also includes functions such as data collection, feature engineering, model training, model application, and model optimization to ensure that the system can continuously optimize and update the model and improve recognition accuracy. Through the above detailed description, the functions and implementation methods of the behavior pattern recognition module can be better understood, ensuring that it can effectively prevent malicious requests in practical applications and protect the normal operation of the medical registration system.
[0107] Embodiment five:
[0108] This embodiment provides another data cleaning system, which is implemented on the basis of the above embodiment, focusing on the log recording and analysis module. The log recording and analysis module is an important part of the entire system, responsible for recording all request data and processing results, and continuously optimizing the system's identification and filtering algorithms by analyzing these log information. This module aims to improve the security and reliability of the system through continuous data collection and analysis. The main functions of the log recording and analysis module in this embodiment include:
[0109] 1. Logging: Record all received requests and their processing results.
[0110] 2. Data cleaning: Clean and preprocess the recorded log data.
[0111] 3. Data analysis: Analyze log data and extract useful information.
[0112] 4. Report generation: Generate analysis reports for reference by system administrators and developers.
[0113] 5. Feedback optimization: Optimize other modules in the system (such as request feature analysis module, behavior pattern recognition module, etc.) based on the analysis results.
[0114] See also Figure 5 A structural schematic diagram of a log recording and analysis module is shown, and the log recording and analysis module includes: a log recording submodule, a data cleaning submodule, a data analysis submodule, a report generation submodule, and a feedback optimization submodule; the log recording submodule is used to receive each request and the processing result of each request as the original data of the log information; the data cleaning submodule is used to clean and preprocess the original data; the data analysis submodule is used to perform statistical analysis on the cleaned original data; the report generation submodule is used to generate an analysis report based on the results of the statistical analysis; the feedback optimization submodule is used to analyze the report optimization behavior pattern recognition module and the algorithm of the log recording and analysis module.
[0115] (1) Logging submodule: responsible for recording all requests and their processing results. It can record all received requests and their processing results. Use a logging framework (such as Log4j, syslog, etc.) to record request information, including request time, request type, request parameters, response results, etc.
[0116] The specific steps may be: Step 1, recording all received requests and their processing results. Step 2, storing the recorded log information in a log file or a database.
[0117] (2) Data cleaning submodule: responsible for cleaning and preprocessing the recorded log data. The recorded log data can be cleaned and preprocessed. Use data cleaning tools (such as Python's Pandas library) to clean the data, including removing duplicate records, filling missing values, converting data formats, etc.
[0118] The specific steps may be: Step 1, reading the original log data in the log file or database. Step 2, cleaning and preprocessing the original data, including removing duplicate records, filling missing values, converting data formats, etc.
[0119] (3) Data analysis submodule: responsible for analyzing log data and extracting useful information. It can analyze log data and extract useful information. It uses data analysis tools (such as Python's Pandas and NumPy libraries) to perform data analysis, including statistical analysis, trend analysis, etc.
[0120] The specific steps may be: Step 1, use data analysis tools to perform statistical analysis on the cleaned data. Step 2, extract useful features and information, such as request frequency, request type distribution, etc.
[0121] (4) Report generation submodule: responsible for generating analysis reports. Analysis reports can be generated for reference by system administrators and developers. Charts and reports are generated using report generation tools (such as Python's Matplotlib and Seaborn libraries).
[0122] The specific steps may be: Step 1, generating charts and reports based on the analysis results. Step 2, summarizing the analysis results into a report for reference by system administrators and developers.
[0123] (5) Feedback optimization submodule: responsible for optimizing other modules in the system based on the analysis results. It can optimize other modules in the system based on the analysis results. It feeds back the analysis results to other modules in the system, such as the request feature analysis module and the behavior pattern recognition module, to optimize their algorithms and strategies.
[0124] The specific steps may be: Step 1, feeding back the analysis results to other modules in the system. Step 2, optimizing the algorithms and strategies of other modules according to the feedback results.
[0125] The log recording and analysis module in this embodiment is the key part of the entire system responsible for recording and analyzing request data. Through this module, the system can effectively record all requests and their processing results, extract useful information through data analysis, and generate reports for reference by system administrators and developers. This module also includes functions such as data cleaning, data analysis, report generation, and feedback optimization to ensure that the system can continuously optimize and update the algorithms and strategies of other modules to improve the security and reliability of the system. Through the above detailed description, the functions and implementation methods of the log recording and analysis module can be better understood to ensure that it can effectively prevent malicious requests in practical applications and protect the normal operation of the medical registration system.
[0126] Embodiment six:
[0127] This embodiment provides another data cleaning system, which is implemented on the basis of the above embodiment, focusing on the real-time monitoring and response module. The real-time monitoring and response module is a key component in the entire system responsible for real-time monitoring of request traffic and taking immediate response measures when malicious requests are detected. This module aims to ensure the security and reliability of the medical registration system through real-time monitoring and rapid response mechanisms, and prevent malicious number brushing and scalper number brushing. The main functions of the real-time monitoring and response module in this embodiment include:
[0128] 1. Real-time monitoring: Continuously monitor the request flow in the system.
[0129] 2. Anomaly detection: Identify and detect requests that do not conform to normal behavior patterns.
[0130] 3. Response measures: Take immediate response measures when malicious requests are detected.
[0131] 4. Dynamic adjustment: Dynamically adjust the filtering strategy based on real-time monitoring results.
[0132] 5. Alarm notification: Send an alarm notification when a serious anomaly is detected.
[0133] See also Figure 6 A structural schematic diagram of a real-time monitoring and response module is shown, and the real-time monitoring and response module includes: a real-time monitoring sub-module, an anomaly detection sub-module, a response measure sub-module, and a dynamic adjustment sub-module; the real-time monitoring sub-module is used to use a network monitoring tool to continuously monitor the request traffic of the data cleaning system; the anomaly detection sub-module is used to detect malicious requests based on preset rules or machine learning models; wherein the rules include at least one of the following: frequency analysis, behavior pattern matching or threshold detection; the response measure sub-module is used to; when a malicious request is detected, execute a specified response measure; the dynamic adjustment sub-module is used to adjust the rules based on the detection results of malicious requests.
[0134] (1) Real-time monitoring submodule: responsible for continuously monitoring the request traffic in the system. It can continuously monitor the request traffic in the system. Use network monitoring tools (such as Wireshark, Nginx, Apache, etc.) to monitor the request traffic in real time and record the relevant information of each request.
[0135] The specific implementation methods are: Logging: record the timestamp, IP address, user agent, request path and other information of each request. Traffic monitoring: monitor the change trend of request traffic in real time.
[0136] (2) Anomaly detection submodule: responsible for identifying and detecting malicious requests. It can identify and detect malicious requests. It uses preset rules or machine learning models to analyze request data and identify requests that do not conform to normal behavior patterns.
[0137] The specific implementation methods are as follows: Frequency analysis: Detect IP addresses with frequent requests in a short period of time. Behavior pattern matching: Use the output results of the behavior pattern recognition module. Threshold detection: Set a threshold, and when the request frequency exceeds the threshold, it is considered abnormal. Machine learning model: Use the trained model to analyze the request data in real time to identify malicious requests.
[0138] (3) Response measures submodule: responsible for taking response measures when malicious requests are detected. Response measures can be taken immediately when malicious requests are detected. Appropriate response measures are taken according to the detected abnormal situation.
[0139] The specific implementation methods are: Restrict access: temporarily or permanently ban access from a certain IP address. Add verification: require users to verify their identity through a verification code. Record logs: record relevant information about malicious requests for subsequent analysis. Notify administrators: notify system administrators via email or SMS.
[0140] (4) Dynamic adjustment submodule: responsible for dynamically adjusting the filtering strategy according to the real-time monitoring results. The filtering strategy can be dynamically adjusted according to the real-time monitoring results. The filtering strategy can be dynamically adjusted according to the results of anomaly detection.
[0141] The specific implementation methods are as follows: Adjust the frequency threshold: adjust the threshold of the request frequency according to the actual situation. Update the behavior pattern library: update the behavior pattern library according to the new malicious request data. Optimize the algorithm: continuously optimize the anomaly detection algorithm to improve the detection accuracy.
[0142] like Figure 6 As shown, the real-time monitoring and response module also includes: an alarm notification submodule; the alarm notification submodule is used to send an alarm notification when the number of malicious requests exceeds a preset threshold.
[0143] (5) Alarm notification submodule: responsible for sending alarm notifications when serious anomalies are detected. Alarm notifications can be sent when serious anomalies are detected. Alarm notifications are sent via email, SMS, system logs, etc.
[0144] The specific implementation method is as follows: Alarm condition: trigger an alarm when a large number of malicious requests are detected. Alarm method: send alarm information via email or SMS. Log record: record alarm events and their processing results.
[0145] The real-time monitoring and response module in this embodiment is a key part of the entire system responsible for real-time monitoring of request traffic and taking immediate response measures when malicious requests are detected. Through this module, the system can effectively monitor request traffic, identify malicious requests, and take corresponding measures to ensure the security and reliability of the medical registration system. This module also includes functions such as dynamic adjustment and alarm notification to ensure that the system can continuously optimize and update filtering strategies and improve the system's defense capabilities. Through the above detailed description, the functions and implementation methods of the real-time monitoring and response module can be better understood to ensure that it can effectively prevent malicious requests in actual applications and protect the normal operation of the medical registration system.
[0146] In summary, the data cleaning system proposed in the present invention is a malicious request identification and filtering system based on feature analysis and machine learning. Through the collaborative work of multiple modules, it can effectively identify and filter malicious requests to ensure the normal operation of the medical registration system. The system has high security and accuracy, and continuously optimizes the identification algorithm through log analysis to maintain the long-term effectiveness of the system.
[0147] The data cleaning system proposed in the present invention can be implemented based on software: software-based implementation is the most common implementation scheme of the data cleaning system. This scheme can be implemented using a variety of programming languages, such as Python, Java, C++, etc. Software-based implementation has the advantages of high flexibility and strong customization, but also has the disadvantages of insufficient performance and stability.
[0148] The data cleaning system proposed in the present invention can be implemented based on a hardware all-in-one machine: the hardware-based implementation can improve the efficiency and stability of data cleaning. This solution is usually implemented using a combination of a server and software. The hardware-based implementation has the advantages of high efficiency and strong stability, but also has the disadvantages of high cost and poor customization.
[0149] The data cleaning system proposed by the present invention mainly discloses:
[0150] 1. Request feature analysis:
[0151] Key technical points: Identify possible malicious requests by extracting and analyzing multiple features in the request (such as IP address, user agent, access frequency, etc.).
[0152] Implementation method: Use a variety of feature extraction technologies, combined with rule engines or machine learning models, to conduct a comprehensive analysis of request data to determine whether the request is normal.
[0153] Technical details: Extract multiple features from the request, such as the historical request frequency of the IP address, user agent string, request frequency, etc., and perform comprehensive analysis through the rule engine or machine learning model.
[0154] Technical advantages: Ability to identify malicious requests from multiple angles, improving the accuracy and robustness of identification.
[0155] 2. Behavioral pattern recognition:
[0156] Key technical points: Use machine learning algorithms to train models to identify and classify behavioral patterns of normal requests and malicious requests.
[0157] Implementation method: By collecting and analyzing a large amount of historical request data, a model that can distinguish between normal and malicious requests is trained, and the model is applied to real-time requests.
[0158] Technical details: The model is trained using machine learning algorithms (such as random forest, support vector machine, neural network, etc.) to learn the behavior patterns of normal and malicious requests through a large amount of historical data.
[0159] Technical advantages: Ability to automatically learn and adapt to new attack patterns, improving the system's self-protection capabilities.
[0160] 3. Real-time monitoring and response:
[0161] Key technical points: Monitor request traffic in real time and take immediate response measures when malicious requests are detected.
[0162] Implementation method: Use real-time monitoring technology combined with anomaly detection algorithms to promptly identify and respond to malicious requests, and take measures such as restricting access and adding verification codes.
[0163] Technical details: High-performance real-time monitoring technology combined with anomaly detection algorithms can quickly identify and respond to malicious requests.
[0164] Technical advantages: It can protect the system from attacks in real time and improve the system's response speed and security.
[0165] 4. Log recording and analysis:
[0166] Key technical points: Record log information of all requests, perform data analysis, and continuously optimize the recognition algorithm.
[0167] Implementation method: Collect request data through the logging system, and use data analysis tools (such as Python's Pandas, NumPy, etc.) to perform statistical analysis, generate reports, and provide feedback to optimize other modules.
[0168] Technical details: Collect detailed request data through the logging system, use data analysis tools to perform statistical analysis, generate reports, and provide feedback to optimize other modules.
[0169] Technical advantages: Ability to continuously optimize recognition algorithms and improve system accuracy and reliability.
[0170] 5. Intelligent dynamic adjustment:
[0171] Key technical points: Dynamically adjust filtering strategies based on real-time monitoring results to improve the adaptability and flexibility of the system.
[0172] Implementation method: Through the feedback of the real-time monitoring module, dynamically adjust the request frequency threshold, update the behavior pattern library, etc., to optimize the filtering strategy.
[0173] Technical details: Dynamically adjust filtering strategies based on real-time monitoring results, such as adjusting request frequency thresholds, updating behavior pattern libraries, etc.
[0174] Technical advantages: Ability to adapt to ever-changing attack methods and improve system flexibility and adaptability.
[0175] 6. Multi-layer protection mechanism:
[0176] Key technical points: Through a multi-level protection mechanism, the system security is fully protected from request access to final response.
[0177] Implementation method: Combine multiple technical means such as request feature analysis, behavior pattern recognition, real-time monitoring and response to build a multi-layer protection system to improve the overall security of the system.
[0178] Technical details: Through a multi-level protection mechanism, from request access to final response, the system security is fully protected.
[0179] Technical advantages: It can form a multi-level security barrier and improve the overall security of the system.
[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned embodiment and will not be repeated here.
[0181] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0182] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0183] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0184] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A data cleaning system, characterized in that: The data cleaning system includes: an Internet request access module, a request feature analysis module, a behavior pattern recognition module, a log recording and analysis module, and a real-time monitoring and response module; The Internet request access module, the request feature analysis module, and the real-time monitoring and response module are connected in sequence, the Internet request access module, the behavior pattern recognition module, and the real-time monitoring and response module are connected in sequence, and the Internet request access module, the log recording and analysis module, and the real-time monitoring and response module are connected in sequence; The Internet request access module is used to receive multiple requests from the Internet, classify the multiple requests, and send the registration request to the request feature analysis module and the behavior pattern recognition module; wherein the request includes at least one of the following: a registration request, an inquiry request, or an appointment request; The request feature analysis module is used to extract the features of the registration request, analyze the features of the registration request, and determine whether the registration request is a normal request or a malicious request; wherein the features of the registration request include at least one of the following: IP address, user agent or access frequency; The behavior pattern recognition module is used to determine whether the registration request is a normal request or a malicious request through a pre-trained machine learning model; The log recording and analysis module is used to record the log information of each request, and optimize the algorithms of the behavior pattern recognition module and the log recording and analysis module based on the log information; The real-time monitoring and response module is used to execute specified response measures when the request feature analysis module or the behavior pattern recognition module determines that the registration request is a malicious request; wherein the response measures include at least one of the following: restricting IP access, adding verification codes, recording logs or notifying administrators.
2. The data cleaning system according to claim 1, characterized in that: The Internet request access module includes: a request receiving submodule, a request parsing submodule, a request classification submodule, and a request forwarding submodule; The request receiving submodule is used to monitor a designated port and receive multiple requests from the Internet; The request parsing submodule is used to parse multiple requests and determine key information of the multiple requests; wherein the key information includes at least one of the following: request type or request parameter; The request classification submodule is used to classify multiple requests based on the key information; The request forwarding submodule is used to send the registration request to the request feature analysis module and the behavior pattern recognition module.
3. The data cleaning system according to claim 2, characterized in that: The Internet request access module also includes: a log recording submodule; The log recording submodule is used to record multiple requests and processing results of the multiple requests, and send the multiple requests and the processing results of the multiple requests to the log recording and analysis module.
4. The data cleaning system according to claim 1, characterized in that: The request feature analysis module includes: a feature extraction submodule, a feature analysis submodule, and a request classification submodule; The feature extraction submodule is used to extract the features of the registration request; The feature analysis submodule is used to analyze the features of the registration request to determine whether the features of the registration request meet the features of a normal request; The request classification submodule is used to determine the registration request as a normal request when the features of the registration request are all consistent with the features of a normal request; when the features of the registration request include features that do not meet the features of a normal request, determine the registration request as a malicious request.
5. The data cleaning system according to claim 4, characterized in that: The request feature analysis module also includes: a feature library maintenance submodule; The feature library maintenance submodule is used to maintain the feature library of normal requests and optimize the algorithms of feature extraction and feature analysis.
6. The data cleaning system according to claim 1, characterized in that: The behavior pattern recognition module includes: a data acquisition submodule, a feature engineering submodule, a model training submodule, and a model application submodule; The data collection submodule is used to obtain historical request data; wherein the historical requests include: normal requests and malicious requests; The feature engineering submodule is used to extract features of the historical request data; the features of the historical request data include at least one of the following: IP address, user agent, access frequency or request mode; The model training submodule is used to train a machine learning model based on the features of the historical request data; The model application submodule is used to determine whether the registration request is a normal request or a malicious request through the machine learning model.
7. The data cleaning system according to claim 6, characterized in that: The behavior pattern recognition module also includes: a model optimization submodule; The model optimization submodule is used to optimize the parameters of the machine learning model.
8. The data cleaning system according to claim 1, characterized in that: The log recording and analysis module includes: a log recording submodule, a data cleaning submodule, a data analysis submodule, a report generation submodule, and a feedback optimization submodule; The log recording submodule is used to receive each request and the processing result of each request as the original data of the log information; The data cleaning submodule is used to clean and pre-process the original data; The data analysis submodule is used to perform statistical analysis on the cleaned raw data; The report generation submodule is used to generate an analysis report based on the results of the statistical analysis; The feedback optimization submodule is used for the analysis report to optimize the algorithms of the behavior pattern recognition module and the log recording and analysis module.
9. The data cleaning system according to claim 1, characterized in that: The real-time monitoring and response module includes: a real-time monitoring submodule, an abnormality detection submodule, a response measure submodule, and a dynamic adjustment submodule; The real-time monitoring submodule is used to continuously monitor the request flow of the data cleaning system using a network monitoring tool; The anomaly detection submodule is used to detect malicious requests based on preset rules or machine learning models; wherein the rules include at least one of the following: frequency analysis, behavior pattern matching or threshold detection; The response measure submodule is used to: execute a specified response measure when the malicious request is detected; The dynamic adjustment submodule is used to adjust the rule based on the detection result of the malicious request.
10. The data cleaning system according to claim 9, characterized in that: The real-time monitoring and response module also includes: an alarm notification submodule; The alarm notification submodule is used to send an alarm notification when the number of the malicious requests exceeds a preset threshold.