Data detection method and system

Abstract

The invention relates to a data detection method and system, and belongs to the technical field of data detection, and the method comprises the following steps: S1, collecting original data; s2, preprocessing the collected data to ensure data quality and consistency; s3, extracting meaningful features from the preprocessed data; s4, separating and processing the extracted features; s5, substituting the feature information into the detection model; s6, training the detection model by using the marked data set; s7, evaluating the performance of the detection model by using the test data set, and optimizing and verifying the detection model; s8, deploying the trained detection model; and S9, in actual use, detecting the data, substituting a detection result into the detection model, and outputting a corresponding processing result. According to the invention, the detection model can be trained, so that the detection accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of data detection technology, and in particular to a data detection method and system. Background Technology

[0002] In the field of computer local area networks (LANs), fault detection of various network elements is mainly achieved through traditional Network Management Systems (NMS). NMS is a network management tool based on the Simple Network Management Protocol (SNMP). Each network element (such as servers, switches, routers, firewalls, etc.) sends Trap information to a designated log server. This Trap information contains status information of each network element, such as hardware load, traffic volume, and interface status. The log server then collects and displays this information to the network administrator.

[0003] Chinese Patent Publication No. CN106446021A discloses a method and system for detecting and processing abnormal data. The method includes: collecting specific behavioral data of a user within a preset time period; extracting feature information from the specific behavioral data; determining a data range for the user's specific behavioral data based on the feature information; and performing a predetermined operation when the user's specific behavioral data exceeds its corresponding data range. This invention can detect and process abnormal transaction data promptly and accurately, improving the efficiency of abnormal data monitoring and reducing corresponding harms.

[0004] However, in existing technologies, it is impossible to train the detection model, resulting in low detection accuracy. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the background art by proposing a data detection method and system that can train a detection model, thereby improving the accuracy of detection.

[0006] The technical solution of the present invention: a data detection method, comprising the following steps:

[0007] S1. Collect raw data;

[0008] S2. Preprocess the collected data to ensure data quality and consistency;

[0009] S3. Extract meaningful features from the preprocessed data;

[0010] S4. Separate and process the extracted features;

[0011] S5. Substitute the feature information into the detection model;

[0012] S6. Train the detection model using the labeled dataset;

[0013] S7. Use the test dataset to evaluate the performance of the detection model, and optimize and validate the detection model;

[0014] S8. Deploy the trained detection model;

[0015] S9. In actual use, the data is detected, and the detection results are substituted into the detection model to output the corresponding processing results.

[0016] Preferably, raw data collection includes databases, networks, sensors, and manual input.

[0017] Preferably, the collected raw data lines are cleaned, noise-reduced, and then normalized.

[0018] Preferably, the detection model includes machine learning models and deep learning neural networks.

[0019] Preferably, the detection model is trained using labeled data.

[0020] A data detection system, applied to the aforementioned data detection method, includes a data acquisition module for collecting data;

[0021] A data preprocessing module that preprocesses the collected data;

[0022] A feature extraction module that extracts features from the processed data;

[0023] A model training module for training the detection model;

[0024] A model evaluation module for assessing the performance of detection data models;

[0025] A decision-making and processing module that makes decisions and processes the results output by the detection model;

[0026] Provides a user interface that displays test results and related statistical information, and receives user feedback, along with a feedback module.

[0027] Preferably, it also includes a log and monitoring module for recording system information logs and performance monitoring information.

[0028] Compared with the prior art, the present invention has the following beneficial technical effects:

[0029] In this invention, the system determines whether a website is malicious based on the model's output. If a website is malicious, corresponding security measures are triggered, such as blocking user access and generating alerts. A user interface is provided to display the detection results and related statistical information. User feedback is received, such as marking false positives or false negatives, to improve the system. System operation logs are recorded, including the detection results and processing records for each website, and system performance is monitored regularly to ensure stable system operation. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the structure of an embodiment of the present invention. Detailed Implementation

[0031] Example 1

[0032] The present invention proposes a data detection method, comprising:

[0033] S1. Collect raw data;

[0034] S2. Preprocess the collected data to ensure data quality and consistency;

[0035] S3. Extract meaningful features from the preprocessed data;

[0036] S4. Separate and process the extracted features;

[0037] S5. Substitute the feature information into the detection model;

[0038] S6. Train the detection model using the labeled dataset;

[0039] S7. Use the test dataset to evaluate the performance of the detection model, and optimize and validate the detection model;

[0040] S8. Deploy the trained detection model;

[0041] S9. In actual use, the data is detected, and the detection results are substituted into the detection model to output the corresponding processing results.

[0042] Raw data collection includes databases, networks, sensors, and manual input.

[0043] The collected raw data lines are cleaned, noise-reduced, and then normalized.

[0044] The detection models include machine learning models and deep learning neural networks.

[0045] The detection model is trained using labeled data.

[0046] In this embodiment, when it is necessary to automatically identify whether a website is malicious, the system crawls URLs and webpage content from the internet to obtain users' historical access data. The collected webpage content is parsed using HTML to extract text content. Natural language processing operations such as word segmentation, stop word removal, and stemming are then performed on the text content to convert the data into feature vectors usable by the detection model. Text features are extracted, allowing the bag-of-words model or word embeddings to convert the text into vector representations. The extracted features are combined into a single feature vector, which serves as the training dataset for labeled malicious and normal websites. Machine learning algorithms (such as random forests) are used to train the model. The model performance is evaluated using a test dataset, including metrics such as accuracy, recall, and precision. Based on the model's output, it is determined whether the website is malicious. If it is malicious, corresponding security measures are triggered, such as blocking user access and generating alerts. A user interface is provided to display the detection results and related statistical information. User feedback, such as marking false positives or false negatives, is received to improve the system. System operation logs are recorded, including the detection results and processing records for each website. System performance is monitored regularly to ensure stable system operation.

[0047] Use databases such as MySQL and MongoDB to store collected data and training sets. Use Python for data processing, training machine learning models, and prediction. Use machine learning frameworks such as Scikit-Learn or TensorFlow. Use web frameworks such as Django and Flask to build the user interface, displaying detection results and providing user feedback. You can choose a cloud service provider for system deployment to ensure high availability and scalability.

[0048] Example 2

[0049] like Figure 1 As shown, the data detection system proposed in this invention, compared to Embodiment 1, is applied to the aforementioned data detection method. It includes a data acquisition module for collecting data; a data preprocessing module for preprocessing the collected data; a feature extraction module for extracting features from the processed data; a model training module for training a detection model; a model evaluation module for evaluating the performance of the detection model; a decision-making and processing module for making decisions and processing the results output by the detection model; and an interface and feedback module providing a user interface to display detection results and related statistical information and receive user feedback. It also includes a log and monitoring module for recording system information logs and performance monitoring information.

[0050] In this embodiment, the data acquisition module is responsible for acquiring raw data from various data sources such as sensors, databases, and APIs. The data preprocessing module performs preprocessing operations such as cleaning, normalization, and noise reduction on the raw data to ensure data quality. The feature extraction module extracts relevant features from the preprocessed data according to task requirements. The model training module trains machine learning or deep learning models using labeled datasets. The model evaluation module evaluates the model's performance using test datasets, including accuracy, recall, and F1 score. The decision-making and processing module makes corresponding decisions and performs processing based on the model's output, which may include alarms, notifications, and logging. The interface and feedback module provides a user interface to display detection results and related statistical information, while also receiving user feedback. The logging and monitoring module records system operation logs and performance monitoring information for system maintenance and optimization.

[0051] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A data detection method, characterized in that, Includes the following steps: S1. Collect raw data; S2. Preprocess the collected data to ensure data quality and consistency; S3. Extract meaningful features from the preprocessed data; S4. Separate and process the extracted features; S5. Substitute the feature information into the detection model; S6. Train the detection model using the labeled dataset; S7. Use the test dataset to evaluate the performance of the detection model, and optimize and validate the detection model; S8. Deploy the trained detection model; S9. In actual use, the data is detected, and the detection results are substituted into the detection model to output the corresponding processing results.

2. The data detection method according to claim 1, characterized in that, Raw data collection includes databases, networks, sensors, and manual input.

3. The data detection method according to claim 2, characterized in that, The collected raw data lines are cleaned, noise-reduced, and then normalized.

4. The data detection method according to claim 3, characterized in that, The detection models include machine learning models and deep learning neural networks.

5. The data detection method according to claim 4, characterized in that, The detection model is trained using labeled data.

6. A data detection system, applied to a data detection method according to any one of claims 1-5, characterized in that, Includes a data acquisition module for collecting data; A data preprocessing module that preprocesses the collected data; A feature extraction module that extracts features from the processed data; A model training module for training the detection model; A model evaluation module for assessing the performance of detection data models; A decision-making and processing module that makes decisions and processes the results output by the detection model; Provides a user interface that displays test results and related statistical information, and receives user feedback, along with a feedback module.

7. A data detection system according to claim 6, characterized in that, It also includes a logging and monitoring module for recording system information logs and performance monitoring information.

Images