System, method and device for realizing structured key data verification based on fusion of active scanning and passive monitoring, processor and medium
By adopting a system of active scanning and passive monitoring fusion mechanisms in large-scale institutions and public networks, combined with deep learning technology, the problem of inefficient data discovery and responsibility identification is solved. It has achieved solid support for data security and network security and reduced legal risks.
Patent Information
- Application Number
- CN202411064313.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
AI Technical Summary
The existing technology has problems such as low efficiency, insufficient linkage verification and lagging laws and regulations in data discovery, screening and responsibility identification in large-scale institutions and public networks, resulting in challenges in data security and network security.
Adopt a system based on the fusion mechanism of active scanning and passive monitoring, through the data discovery module, data analysis module and continuous verification module, combined with active scanning and passive monitoring technology, the coupled checksum continuous verification of structured key data is realized, and the two-way cycle is used. Neural networks, deep convolutional networks and integrated SVMs perform deep feature extraction and classification hierarchy.
It realizes comprehensive and accurate verification of structured key data, improves data security supervision efficiency, ensures data security and integrity, reduces the risks of data leakage and abuse, enhances the trust in data processing and storage, and helps enterprises Complying with data protection regulations reduces legal risks.
Smart Images

Figure CN119026117A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information security, and in particular to the field of data verification and network monitoring, and specifically refers to a system, method, device, processor and computer-readable storage medium thereof for implementing verification processing on structured key data based on a fusion mechanism of active scanning and passive monitoring. Background Art
[0002] With the rapid development of technologies such as big data, cloud computing, and artificial intelligence, the scale, variety, and complexity of data are increasing, making data discovery, identification, and responsibility identification more difficult and complex. Therefore, research on the discovery, identification, and responsibility of important data in large-scale institutions and public networks is particularly urgent and is an important research topic in the current field of information technology and network security.
[0003] At present, the research status in this field is mainly reflected in the following aspects:
[0004] First, data discovery technology continues to improve: by using data mining, machine learning and other technologies, researchers can more accurately extract valuable information from massive amounts of data. These technologies can not only identify patterns, trends and associations in data, but also help organizations discover potential security risks and threats.
[0005] Second, data screening methods are becoming increasingly sophisticated: the screening of data authenticity, integrity and reliability is the focus of research. Researchers use technical means such as data tracing, verification and validation to conduct in-depth analysis and verification of data to ensure the accuracy and credibility of the data.
[0006] Third, the responsibility identification mechanism is gradually established: With the frequent occurrence of security incidents such as data leakage and abuse, data responsibility identification has become an urgent problem to be solved. Currently, researchers are exploring the establishment of a responsibility identification mechanism based on legal, technical and ethical aspects to clarify the responsibilities of all parties involved in data use, processing and protection.
[0007] However, research in this field still faces some challenges and difficulties. Data discovery methods are still imperfect, the efficiency of identifying data in circulation is low, and there is a lack of linkage verification between the two. At the same time, data governance and related laws and regulations lag behind. These problems require the joint efforts of researchers, policymakers and the industry to solve them through technological innovation, policy formulation and the establishment of cooperation mechanisms.
[0008] In the future, with the continuous development of technology and the deepening of its application, the research on the discovery, identification and responsibility of important data in large-scale institutions and public networks will become more in-depth and extensive. We look forward to more breakthroughs and progress in this field, providing more solid technical and theoretical support for data security and network security. Summary of the invention
[0009] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide a system, method, device, processor and computer-readable storage medium thereof that meet the requirements of good accuracy, comprehensiveness and wide application range for verifying and processing structured key data based on the fusion mechanism of active scanning and passive monitoring.
[0010] In order to achieve the above objectives, the system, method, device, processor and computer-readable storage medium thereof for verifying structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention are as follows:
[0011] The system realizes verification and processing of structured key data based on the fusion mechanism of active scanning and passive monitoring, and its main characteristics are that the system includes a data discovery module, a data analysis module and a continuous verification module, the data analysis module is connected to the data discovery module, and the continuous verification module is connected to the data analysis module; the data discovery module is used to acquire data through active scanning and passive monitoring, the data analysis module is used to perform coupling verification and real-time analysis on captured data packets, and the continuous verification module is used to perform coupling verification and continuous verification on active scanning data and passive monitoring data.
[0012] Preferably, the data discovery module includes:
[0013] An active scanning detection unit, used to discover key data through active scanning technology;
[0014] Passive flow monitoring unit, used for passive monitoring and real-time monitoring;
[0015] The data verification unit is connected to the active scanning detection unit and the passive flow monitoring unit, and is used to verify the data of active scanning and passive monitoring.
[0016] Preferably, the data analysis module includes:
[0017] A key information extraction unit, used for extracting key information;
[0018] A second data verification unit, used for performing coupling verification of data;
[0019] A result output unit, used for outputting results;
[0020] A data cleaning unit, connected to the key information extraction unit, the second data verification unit and the result output unit, for cleaning the data;
[0021] The data integration unit is connected to the key information extraction unit, the second data verification unit and the result output unit, and is used to integrate the cleaned data.
[0022] Preferably, the continuous verification module includes a continuous detection unit, a continuous monitoring unit, a third data verification unit and a report output unit. The continuous detection unit, the continuous monitoring unit, the third data verification unit and the report output unit are all connected to the data analysis module. The third data verification unit is used to compare and analyze active scanning and passive monitoring data to evaluate data consistency and potential risks. The continuous detection unit is used to continuously detect key data and network activities. The continuous monitoring unit is used to continuously monitor key data and network activities. The report output unit is used to output verification results and take response measures.
[0023] Preferably, the active scanning detection unit includes a Web service subsystem and a scanning service subsystem. The Web service subsystem is connected to the scanning service subsystem. The Web service subsystem is used to process requests and add scanning tasks to the redis task queue module. The scanning service subsystem is used to perform cyberspace asset mapping and identification according to queue tasks, and send the scanning service to the corresponding submodule by identifying the type of task.
[0024] Preferably, the scanning service subsystem includes an IP activity detection module, a domain name identification module, an operating system identification module, a port identification module and a protocol and component identification module. The IP activity detection module and the domain name identification module are respectively connected to the operating system identification module and the port identification module, and the protocol and component identification module is connected to the port identification module. The IP activity detection module performs IP scanning, the domain name identification module performs domain name identification, the port identification module performs port identification on the acquired IP activity detection results, the operating system identification module performs operating system identification on the acquired IP activity detection results, and the protocol and component identification module performs protocol and component scanning on the port identification results.
[0025] Preferably, the passive flow monitoring unit performs passive monitoring including the following steps:
[0026] Deploy passive monitoring devices or software at key locations on a network or system to capture and analyze network traffic or system activity;
[0027] Monitoring devices silently capture passing network traffic or system activity packets;
[0028] Parse and analyze captured data packets to extract real-time information during data transmission, processing and use;
[0029] The monitoring results are recorded and saved for subsequent comparison and analysis with active scanning results.
[0030] Preferably, the data analysis module performs structured key data verification based on the fusion of active scanning and passive monitoring, specifically including word segmentation based on a bidirectional recurrent neural network, text deep feature extraction based on a deep convolutional network, and deep feature classification and grading based on an integrated SVM.
[0031] The method for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism by using the above system is mainly characterized in that the method comprises the following steps:
[0032] (1) Prepare data and configure parameters;
[0033] (2) Conduct active scanning;
[0034] (3) Conduct passive monitoring;
[0035] (4) Coupling verification of data;
[0036] (5) Output and respond to results.
[0037] Preferably, the step (2) specifically comprises the following steps:
[0038] (2.1) Use active scanning tools to send probe requests to the target system to obtain information about the source, storage location, and access path of key data;
[0039] (2.2) Receive the response data packet returned by the target system and perform in-depth analysis to extract detailed information of key data.
[0040] Preferably, the step (3) specifically comprises the following steps:
[0041] (3.1) Deploy passive monitoring devices on key data transmission paths to capture network traffic or system activity data packets;
[0042] (3.2) Capture data packets in real time and conduct in-depth analysis to extract real-time information about the data transmission, processing, and use.
[0043] Preferably, the step (4) specifically comprises the following steps:
[0044] (4.1) Design a structured verification process based on the verification objectives;
[0045] (4.2) Perform verification tasks at each stage in sequence according to the predetermined verification process;
[0046] (4.3) Verify the data source;
[0047] (4.4) Perform data transmission verification;
[0048] (4.5) Conduct data processing verification;
[0049] (4.6) Perform data usage verification.
[0050] Preferably, the step (5) specifically comprises the following steps:
[0051] (5.1) Generate a detailed verification report based on the verification results;
[0052] (5.2) Take appropriate countermeasures based on the recommended measures in the verification report.
[0053] The device for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism has the following main features: the device comprises:
[0054] a processor configured to execute computer executable instructions;
[0055] A memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for verifying and processing structured key data based on the active scanning and passive monitoring fusion mechanism are implemented.
[0056] The processor for implementing verification processing for structured key data based on the fusion mechanism of active scanning and passive monitoring has the main feature that the processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the above-mentioned method for implementing verification processing for structured key data based on the fusion mechanism of active scanning and passive monitoring are implemented.
[0057] The main feature of the computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for verifying and processing structured key data based on the fusion mechanism of active scanning and passive monitoring.
[0058] The system, method, device, processor and computer-readable storage medium thereof for verifying and processing structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention can timely detect and prevent the occurrence of data leakage and abuse, thereby reducing potential economic losses for enterprises. By real-time monitoring and verifying the integrity and accuracy of data, the present invention can help enterprises improve operational efficiency. The present invention will greatly enhance the public's confidence in data security and enhance the public's trust in data processing and storage. All walks of life have regulatory requirements regarding data protection and privacy. The implementation of the present invention will help enterprises, governments and institutions better comply with these regulations, ensure the legal and compliant use of data, and reduce the legal risks caused by illegal operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 The present invention is a flowchart of an implementation of active scanning of a system for verifying structured key data based on a fusion mechanism of active scanning and passive monitoring.
[0060] Figure 2 The present invention is a flowchart of a system for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism and utilizing a deep convolutional network to extract text.
[0061] Figure 3 It is a structural schematic diagram of the convolution filtering of the deep convolutional neural network of the system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention.
[0062] Figure 4 This is a schematic diagram of the ResNet structure used in this project in the deep network of the system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention.
[0063] Figure 5 It is a schematic diagram of an integrated SVM classifier of the system for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention.
[0064] Figure 6 It is a schematic diagram of the main flow of the system for implementing verification processing on structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention. DETAILED DESCRIPTION
[0065] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.
[0066] The system of the present invention implements verification processing for structured key data based on the fusion mechanism of active scanning and passive monitoring, which includes a data discovery module, a data analysis module and a continuous verification module. The data analysis module is connected to the data discovery module, and the continuous verification module is connected to the data analysis module; the data discovery module is used to acquire data through active scanning and passive monitoring, the data analysis module is used to perform coupling verification and real-time analysis on captured data packets, and the continuous verification module is used to perform coupling verification and continuous verification on active scanning data and passive monitoring data.
[0067] As a preferred embodiment of the present invention, the data discovery module includes:
[0068] An active scanning detection unit, used to discover key data through active scanning technology;
[0069] Passive flow monitoring unit, used for passive monitoring and real-time monitoring;
[0070] The data verification unit is connected to the active scanning detection unit and the passive flow monitoring unit, and is used to verify the data of active scanning and passive monitoring.
[0071] As a preferred embodiment of the present invention, the data analysis module includes:
[0072] A key information extraction unit, used for extracting key information;
[0073] A second data verification unit, used for performing coupling verification of data;
[0074] A result output unit, used for outputting results;
[0075] A data cleaning unit, connected to the key information extraction unit, the second data verification unit and the result output unit, for cleaning the data;
[0076] The data integration unit is connected to the key information extraction unit, the second data verification unit and the result output unit, and is used to integrate the cleaned data.
[0077] As a preferred embodiment of the present invention, the continuous verification module includes a continuous detection unit, a continuous monitoring unit, a third data verification unit and a report output unit. The continuous detection unit, the continuous monitoring unit, the third data verification unit and the report output unit are all connected to the data analysis module. The third data verification unit is used to compare and analyze active scanning and passive monitoring data to evaluate data consistency and potential risks. The continuous detection unit is used to continuously detect key data and network activities. The continuous monitoring unit is used to continuously monitor key data and network activities. The report output unit is used to output verification results and take response measures.
[0078] As a preferred embodiment of the present invention, the active scanning detection unit includes a Web service subsystem and a scanning service subsystem. The Web service subsystem is connected to the scanning service subsystem. The Web service subsystem is used to process requests and add scanning tasks to the redis task queue module. The scanning service subsystem is used to perform cyberspace asset mapping and identification according to queue tasks, and send the scanning service to the corresponding submodule by identifying the type of task.
[0079] As a preferred embodiment of the present invention, the scanning service subsystem includes an IP activity detection module, a domain name identification module, an operating system identification module, a port identification module and a protocol and component identification module. The IP activity detection module and the domain name identification module are respectively connected to the operating system identification module and the port identification module, and the protocol and component identification module is connected to the port identification module. The IP activity detection module performs IP scanning, the domain name identification module performs domain name identification, the port identification module performs port identification on the obtained IP activity detection result, the operating system identification module performs operating system identification on the obtained IP activity detection result, and the protocol and component identification module performs protocol and component scanning on the port identification result.
[0080] As a preferred embodiment of the present invention, the passive flow monitoring unit performs passive monitoring including the following steps:
[0081] Deploy passive monitoring devices or software at key locations on a network or system to capture and analyze network traffic or system activity;
[0082] Monitoring devices silently capture passing network traffic or system activity packets;
[0083] Parse and analyze captured data packets to extract real-time information during data transmission, processing and use;
[0084] The monitoring results are recorded and saved for subsequent comparison and analysis with active scanning results.
[0085] As a preferred embodiment of the present invention, the data analysis module performs structured key data verification based on the fusion of active scanning and passive monitoring, specifically including word segmentation based on a bidirectional recurrent neural network, text deep feature extraction based on a deep convolutional network, and deep feature classification and grading based on an integrated SVM.
[0086] The present invention uses the above system to implement a method for verifying and processing structured key data based on an active scanning and passive monitoring fusion mechanism, wherein the method comprises the following steps:
[0087] (1) Prepare data and configure parameters;
[0088] (2) Conduct active scanning;
[0089] (3) Conduct passive monitoring;
[0090] (4) Coupling verification of data;
[0091] (5) Output and respond to results.
[0092] As a preferred embodiment of the present invention, the step (2) specifically comprises the following steps:
[0093] (2.1) Use active scanning tools to send probe requests to the target system to obtain information about the source, storage location, and access path of key data;
[0094] (2.2) Receive the response data packet returned by the target system and perform in-depth analysis to extract detailed information of key data.
[0095] As a preferred embodiment of the present invention, the step (3) specifically comprises the following steps:
[0096] (3.1) Deploy passive monitoring devices on key data transmission paths to capture network traffic or system activity data packets;
[0097] (3.2) Capture data packets in real time and conduct in-depth analysis to extract real-time information about the data transmission, processing, and use.
[0098] As a preferred embodiment of the present invention, the step (4) specifically comprises the following steps:
[0099] (4.1) Design a structured verification process based on the verification objectives;
[0100] (4.2) Perform verification tasks at each stage in sequence according to the predetermined verification process;
[0101] (4.3) Verify the data source;
[0102] (4.4) Perform data transmission verification;
[0103] (4.5) Conduct data processing verification;
[0104] (4.6) Perform data usage verification.
[0105] As a preferred embodiment of the present invention, the step (5) specifically comprises the following steps:
[0106] (5.1) Generate a detailed verification report based on the verification results;
[0107] (5.2) Take appropriate countermeasures based on the recommended measures in the verification report.
[0108] The device for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention, wherein the device comprises:
[0109] a processor configured to execute computer executable instructions;
[0110] A memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for verifying and processing structured key data based on the active scanning and passive monitoring fusion mechanism are implemented.
[0111] The processor of the present invention is used to implement verification processing for structured key data based on the fusion mechanism of active scanning and passive monitoring, wherein the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the various steps of the above-mentioned method for implementing verification processing for structured key data based on the fusion mechanism of active scanning and passive monitoring are implemented.
[0112] The computer-readable storage medium of the present invention stores a computer program thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for verifying and processing structured key data based on the fusion mechanism of active scanning and passive monitoring.
[0113] In a specific embodiment of the present invention, a structured key data verification technology based on the integration of active scanning and passive monitoring is provided, which aims to solve the accuracy and comprehensiveness problems of important data verification in large-scale institutions and public networks.
[0114] By combining active scanning and passive monitoring technologies, coupled verification of key data is achieved. Active scanning technology can actively detect the target system, obtain key information such as the source, storage location and access path of the data, and provide basic data for subsequent verification. Passive monitoring technology can capture and analyze network traffic or system activities, obtain real-time information during data transmission, processing and use, and ensure that there are no important omissions in the data source.
[0115] The present invention can achieve comprehensive coverage and in-depth verification of key data. The data collected by active scanning and passive monitoring can complement and verify each other, improving the accuracy and reliability of the verification results. At the same time, the present invention also adopts a structured verification method to decompose the verification process into multiple stages and links, each stage and link has clear goals and outputs, making the verification process clearer and controllable.
[0116] The structured key data verification technology based on the integration of active scanning and passive monitoring is the main content protected by this invention. The innovation of this system lies in the in-depth exploration and practice of multi-dimensional verification technology for data security supervision needs. By combining the two advanced technologies of active scanning and passive monitoring, and introducing a coupling verification mechanism, the system significantly improves the supervision efficiency of data security and ensures the security and integrity of the supervision data.
[0117] The first is to adopt a comprehensive verification strategy. The present invention combines active scanning and passive monitoring technologies to form a comprehensive verification strategy. Active scanning can actively discover potential data security risks, while passive monitoring can capture and analyze network traffic or system activities in real time, thereby achieving real-time monitoring of data. This comprehensive strategy makes the verification process more comprehensive and accurate.
[0118] The second is to introduce a coupling verification mechanism. The present invention introduces a coupling verification mechanism to verify the results of active scanning and passive monitoring. This mechanism can ensure the accuracy and reliability of the verification results and reduce false positives and false negatives. By comparing and analyzing the information obtained by the two technologies, the security and integrity of the data can be evaluated more accurately.
[0119] The third is to adopt a structured verification method. The present invention adopts a structured verification method, which decomposes the verification process into multiple stages and links, and each stage has clear goals and outputs. This structured verification method makes the verification work clearer and more orderly, and improves the efficiency and accuracy of the verification. At the same time, it also makes the verification process easier to manage and control.
[0120] This invention proposes a comprehensive technical solution to achieve comprehensive and accurate verification of structured key data to ensure the security and integrity of the data. This technical solution combines active scanning and passive monitoring technologies, and achieves real-time and efficient monitoring and verification of data through a coupled verification mechanism and a structured verification method.
[0121] The first is to study IP detection technology based on active scanning:
[0122] Active scanning technology is one of the key technologies of the present invention, which is used to actively detect the target system to obtain relevant information of key data. This technology sends specific detection requests to the target system, collects and analyzes the returned data packets, and obtains key information such as the source, storage location, and access path of the data. Active scanning technology can be performed regularly or irregularly to ensure continuous monitoring of the target system status and timely data discovery.
[0123] The active scanning module will be composed of a Web service subsystem and a scanning service subsystem, in which the mapping and identification of cyberspace assets are mainly completed by the scanning service subsystem, which mainly includes an IP detection module, a port identification module, a protocol and component identification module, an operating system identification module, and a domain name identification module. When a user adds a task through the Web front end, the Web service subsystem first processes the request, and adds the scanning task to the redis task queue module in the data operation module. The scanning service subsystem then maps and identifies cyberspace assets according to the queue task. The scanning service identifies the type of task and sends it to the corresponding submodule, such as IP scanning or domain name identification. The obtained IP detection results will be sent to the port identification module and the operating system identification module for port identification and operating system identification respectively. The port identification results will continue to be sent to the protocol and component identification module for protocol and component scanning.
[0124] Scan target: The target of the IP scan can be an accurate IP address, IP network segment or IP address range. When entering the IP, the first step is to perform a preliminary check through regular expressions, and then the final check is performed through the system network toolkit.
[0125] IP live detection technology: There are many IP live detection technologies, and the following ones are mainly used here:
[0126] 1) TCP SYN Ping: Set an empty TCP packet with the SYN flag and send it to port 80 of the target host by default. This can also be modified according to special needs. If the target host is alive, it responds to the SYN / ACK TCP packet and performs the second step of the TCP three-way handshake.
[0127] 2) TCP ACK Ping: Set a TCP data packet with only the ACK flag and send it to a port of the target host. If the target host is not alive, it will not respond to the request. If the target host is online, it will return a RST packet.
[0128] 3) UDP Ping: Set an empty UDP packet and send it to the target host port 40125 by default. It can also be modified according to special needs. If the target host responds, an ICMP port unreachable error is returned. If the target host is not alive, various ICMP error messages are returned.
[0129] 4) ARP Ping: The host broadcasts the ARP request to all hosts on the network, receives the return message, determines the physical address of the target IP address, and stores the IP address and hardware address in the local ARP cache. The next request will directly query the ARP cache. If the target host is found on the local area network where it is located, an ARP scan will be performed.
[0130] 5) ICMP Echo Scan: Send an ICMP type 8 (echo request) message to the target IP address. If the target host is available, it will receive a type 0 (echo response) message.
[0131] 6) ICMP Sweep Scan: Concurrently use ICMP Echo Scan to detect multiple target hosts at a time to improve detection efficiency, suitable for large-scale assessments.
[0132] 7) Broadcast ICMP scanning: Broadcast ICMP scanning takes advantage of the differences in ICMP implementation of some hosts. By setting the target address of the ICMP request packet to the broadcast address or network address, it can detect hosts within the broadcast domain or the entire network, and all surviving hosts in the subnet will respond. However, this is only suitable for UNIX / Linux systems.
[0133] Fine-grained block technology: If a large segment of IP is scanned directly, there are two disadvantages: 1. The task takes too long to return, and the task progress is not clear enough; 2. Scanning too large a set of tasks will affect the accuracy of the final result. The structured key data verification technology based on the fusion of active scanning and passive monitoring will divide the large segment of IP into blocks, and then scan them concurrently, and finally recycle and organize the results of concurrent processing into the warehouse. After an IP task is divided into segments and concurrently, after each subtask completes the scan, the progress of the task execution can be known more accurately through the returned results. And the utilization rate of the network is also greatly improved.
[0134] IP scanning rate and performance guarantee technology: In a single IP large task, the algorithm is used to split the single task into multiple subtasks, and 100 threads are started by default to perform concurrent tasks. For example, a ' / 16' network segment covers 65536 IP addresses, which is divided into 255 ' / 24' network segments for concurrent processing. In other words, this technology will support the scanning of 65,536 hosts in a single task.
[0135] Port precision scanning technology: The port identification module is used to determine the open ports on a specific IP address, preparing for the next step of detection and penetration. The port precision scanning technologies used in this invention mainly include the following:
[0136] 1) TCP-SYN scan (half-open scan): A better choice in most cases and will be used as the default scan strategy. It has low scanning noise, high speed, good compatibility with the TCP protocol stack, and clearer and more reliable identification of port status.
[0137] 2) TCP complete connection scan: Establish a complete link with the port on the target IP. The detection speed is average, which will generate more network traffic and is easily monitored and recorded by the target machine.
[0138] 3) UDP port scanning: Sends a UDP message of increasing length to the target IP, and evaluates the status of the target port based on the response. It is more difficult than TCP detection, so the speed is also slower. It can be activated at the same time as TCP scanning.
[0139] 4) Non-SYN / RST / ACK and Maimon scans: According to RFC-793, scans initiated in RFC-compliant environments are more concealed and can evade stateless firewalls and packet filtering routers. However, they cannot distinguish between the open and filtered states of ports.
[0140] 5) ACK scan: It is mainly used to determine the rules of the firewall (whether there is a state or not). It can find out whether the scan of the port is filtered by the rules, but it cannot determine the open or closed status of the port.
[0141] The second is to study the passive monitoring technology based on traffic:
[0142] Passive monitoring technology is another important component of the present invention. It obtains information about data transmission, processing and use in real time by capturing and analyzing network traffic or system activities. This technology does not need to actively send detection requests, but silently monitors network or system activities to ensure that there are no important omissions in the data source. Passive monitoring technology can monitor network traffic and system activities in real time, and promptly detect risks such as abnormal behavior and data leakage. The implementation of the passive traffic monitoring technology in the present invention will be mainly divided into the following steps:
[0143] 1) Deploy passive monitoring devices or software at key locations on the network or system to capture and analyze network traffic or system activity;
[0144] 2) Monitoring devices silently capture passing network traffic or system activity packets;
[0145] 3) Parse and analyze the captured data packets to extract real-time information during data transmission, processing and use;
[0146] 4) The monitoring results are recorded and saved for subsequent comparison and analysis with active scanning results.
[0147] The third is to study the implementation of data coupling verification based on active detection and passive monitoring modes:
[0148] Compare and analyze the data information obtained by active scanning with the real-time information obtained by passive monitoring. Check whether there are any anomalies in data consistency and behavior patterns. Based on the comparison results, discover potential security risks and threats, such as data leakage and tampering. Record and report the potential risks found so that corresponding security measures can be taken in a timely manner.
[0149] The patent of this invention aims to focus on the discovery, identification, classification and grading of sensitive data during the research process. These technologies provide strong support for efficient and accurate data security management. To overcome the core key technologies of data asset graph database technology, specifically, to design and implement an efficient data asset graph database for storing and managing metadata of sensitive data. The database uses graph database technology (such as Neo4j) to efficiently store and query complex relationships between data. A set of data coupling verification algorithms has been developed that can automatically classify and grade data according to the sensitivity and importance of the data. The algorithm combines a rule engine and machine learning technology to dynamically adjust classification strategies. A breakthrough has been made in the compatibility of multiple data systems, and a structured key data verification technology based on the fusion of active scanning and passive monitoring has been developed.
[0150] The present invention uses the following three core methods:
[0151] ① Word segmentation method based on bidirectional recurrent neural network
[0152] A bidirectional gated recurrent unit (GRU) network is used for Chinese word segmentation. The basic steps are as follows:
[0153] First, the Embedding method is used to convert words into word vectors, and then the word vectors are input into a bidirectional gated recurrent unit (GRU) network for word segmentation.
[0154] The bidirectional gated recurrent unit network structure is shown in the following figure:
[0155] The input layer at the bottom of the figure assumes that a sample (sentence) has 10 words of input x1, x2, ..., x 10 There are now two separate GRUCells:
[0156] For the forward fw_cell, the samples are in the order of x1,x2,...,x 10 The first set of state outputs {h1,h2,...,h 10};
[0157] For the reverse bw_cell, the samples are in the order of x 10 ,x9,...,x1 are input into the cell in reverse order to get the second set of state outputs
[0158]
[0159] Each element of the two sets of state outputs is a vector of length hidden_size (usually, h1 and Now put the two sets of state variables together in the following form
[0160] Finally, for each timestep input x t , and get a state output of length 2*hidden_size In this way, using H t The model can be converted into a traditional unidirectional network and processed like this.
[0161] ②Text deep feature extraction method based on deep convolutional network
[0162] This method calculates the length of sample sentences by word, maps the words to the embedding layer to become word vectors using the Embedding method, calculates the weights, and obtains the feature matrix; uses CNNs' filters to scan the embedding layer, uses max-pool for sampling, and generates deep features; and inputs the generated new features into a multi-class classifier for classification.
[0163] The deep convolutional network adopts the ResNet structure in the learning process, which can effectively avoid the gradient disappearance. Compared with the traditional convolutional network, it can effectively reduce the parameters and reduce the computational complexity, and improve the spatial aggregation of deep features.
[0164] The basic steps for text extraction using deep convolutional networks are as follows:
[0165] First, the sample sentence is separated by word to calculate the length, and the word is mapped to the embedding layer to become a word vector using the embedding method, and the weight is calculated to obtain the feature matrix. Then the filter of the deep convolutional network is used to scan the embedding layer to generate deep features, and then the generated new features are input into the multi-class classifier for classification. Figure 2 shown.
[0166] The convolution filter of deep convolutional neural networks (CNNs) is usually structured as follows: Figure 3 shown.
[0167] In the deep network, this project uses the ResNet structure, such as Figure 4 shown.
[0168] The use of the ResNet structure can effectively avoid gradient vanishing in the learning process. Compared with traditional convolutional networks, it can effectively reduce parameters and computational complexity and improve the spatial aggregation of deep features.
[0169] ③ Deep feature classification and grading based on integrated SVM:
[0170] The integrated support vector machine (SVM) is used to classify the deep features of the text extracted by the deep convolutional network to complete the classification and grading of the text and the discovery of sensitive data. The integrated method uses character and word embedding to form different feature spaces, and uses a combination of multi-dimensionality reduction and multi-kernel functions to form multiple SVM classifiers. The output of the classifier uses a weighted combination voting method, and the result with the highest score is the text classification result. The integrated SVM classifier is as follows: Figure 5 shown.
[0171] For characters and words, the deep features extracted by deep convolutional networks are reduced in different dimensions using PCA to form a multi-dimensional deep feature matrix, and different kernel functions are used for SVM. In this way, a classifier SVM is formed by using the reduced dimension features of a specific dimension and a kernel function SVM for the deep features of characters or words. i , as shown in the figure above. The output of each classifier is input into the combined voting module for weighted combined voting. The winning result of the combined voting is the final classification result, thereby achieving the classification and grading of texts, and then realizing the discovery of sensitive data. The use of integrated SVM effectively improves the accuracy of classification.
[0172] Figure 6 The main process of structured key data verification technology based on the fusion of active scanning and passive monitoring is demonstrated.
[0173] Data discovery and initialization: The process of discovering key data using active scanning technology.
[0174] Passive monitoring deployment: The deployment of passive monitoring equipment represents the beginning of real-time monitoring.
[0175] Data analysis: The process of analyzing captured data packets in real time.
[0176] Coupling verification: refers to the process of comparing and analyzing active scanning and passive monitoring data to assess data consistency and potential risks.
[0177] Structured Verification: represents a structured verification process, including verification tasks in multiple stages.
[0178] Result output and response: represents the output of verification results and corresponding response measures.
[0179] Continuous monitoring and improvement: The system continuously monitors key data and network activities, and continuously optimizes and improves verification technologies and methods.
[0180] The overall implementation process of this system includes several stages, including demand analysis and research, key technology research, system outline design, system detailed design, function development, function testing, project acceptance, etc., and finally completes the overall research of structured key data verification technology based on the integration of active scanning and passive monitoring. The specific implementation will be divided into the following steps:
[0181] 1) Technical preparation stage:
[0182] Clarify verification objectives: determine the scope, source, transmission path, etc. of the structured key data that needs to be verified.
[0183] Select tools and configure parameters: According to the verification objectives, select appropriate active scanning and passive monitoring tools, and configure corresponding parameters, such as scanning frequency, monitoring port, etc.
[0184] 2) Active scanning phase
[0185] Sending a probe request: Using active scanning tools, send a probe request to the target system to obtain information such as the source, storage location, and access path of key data;
[0186] Receive and analyze data packets: Receive response data packets returned by the target system and perform in-depth analysis to extract detailed information on key data.
[0187] 3) Passive monitoring stage
[0188] Deploy monitoring equipment: Deploy passive monitoring equipment on key data transmission paths to capture network traffic or system activity data packets;
[0189] Real-time capture and analysis: Capture data packets in real time and conduct in-depth analysis to extract real-time information during data transmission, processing, and use.
[0190] 4) Data coupling verification phase
[0191] Verification process design: Design a structured verification process based on the verification objectives, and clarify the verification tasks, objectives, and outputs of each stage;
[0192] Perform verification tasks in sequence: According to the predetermined verification process, perform verification tasks at each stage in sequence to ensure the accuracy and completeness of the data:
[0193] Data source verification: confirm whether the source of the data is reliable and legal.
[0194] Data transmission verification: Check whether the data has been tampered with or leaked during transmission.
[0195] Data processing validation: ensuring that data complies with predetermined rules and logic during processing.
[0196] Data usage verification: Monitor data usage to prevent unauthorized access or misuse.
[0197] 5) Result output and response
[0198] Generate verification report: Generate a detailed verification report based on the verification results, including data status, potential risks, recommended measures, etc.
[0199] Response processing: Take appropriate countermeasures based on the recommended measures in the verification report.
[0200] The specific implementation scheme of this embodiment can refer to the relevant description in the above embodiment, which will not be repeated here.
[0201] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.
[0202] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.
[0203] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0204] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0205] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.
[0206] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0207] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0208] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0209] The system, method, device, processor and computer-readable storage medium thereof for verifying and processing structured key data based on the active scanning and passive monitoring fusion mechanism of the present invention can timely detect and prevent the occurrence of data leakage and abuse, thereby reducing potential economic losses for enterprises. By real-time monitoring and verifying the integrity and accuracy of data, the present invention can help enterprises improve operational efficiency. The present invention will greatly enhance the public's confidence in data security and enhance the public's trust in data processing and storage. All walks of life have regulatory requirements regarding data protection and privacy. The implementation of the present invention will help enterprises, governments and institutions better comply with these regulations, ensure the legal and compliant use of data, and reduce the legal risks caused by illegal operations.
[0210] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.
Claims
1. A system for verifying and processing structured key data based on the fusion mechanism of active scanning and passive monitoring. It is characterized in that The system includes a data discovery module, a data analysis module and a continuous verification module, wherein the data analysis module is connected to the data discovery module, and the continuous verification module is connected to the data analysis module; the data discovery module is used to acquire data through active scanning and passive monitoring, the data analysis module is used to perform coupling verification and real-time analysis on captured data packets, and the continuous verification module is used to perform coupling verification and continuous verification on active scanning data and passive monitoring data.
2. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 1, It is characterized in that The data discovery module includes: An active scanning detection unit, used to discover key data through active scanning technology; Passive flow monitoring unit, used for passive monitoring and real-time monitoring; The data verification unit is connected to the active scanning detection unit and the passive flow monitoring unit, and is used to verify the data of active scanning and passive monitoring.
3. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 1, It is characterized in that The data analysis module includes: A key information extraction unit, used for extracting key information; A second data verification unit, used for performing coupling verification of data; A result output unit, used for outputting results; A data cleaning unit, connected to the key information extraction unit, the second data verification unit and the result output unit, for cleaning the data; The data integration unit is connected to the key information extraction unit, the second data verification unit and the result output unit, and is used to integrate the cleaned data.
4. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 1, It is characterized in that The continuous verification module includes a continuous detection unit, a continuous monitoring unit, a third data verification unit and a report output unit. The continuous detection unit, the continuous monitoring unit, the third data verification unit and the report output unit are all connected to the data analysis module. The third data verification unit is used to compare and analyze active scanning and passive monitoring data to evaluate data consistency and potential risks. The continuous detection unit is used to continuously detect key data and network activities. The continuous monitoring unit is used to continuously monitor key data and network activities. The report output unit is used to output verification results and take response measures.
5. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 2, It is characterized in that The active scanning detection unit includes a Web service subsystem and a scanning service subsystem. The Web service subsystem is connected to the scanning service subsystem. The Web service subsystem is used to process requests and add scanning tasks to the redis task queue module. The scanning service subsystem is used to perform cyberspace asset mapping and identification according to queue tasks, and send the scanning service to the corresponding submodule by identifying the type of task.
6. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 5, It is characterized in that The scanning service subsystem includes an IP detection module, a domain name identification module, an operating system identification module, a port identification module and a protocol and component identification module. The IP detection module and the domain name identification module are respectively connected to the operating system identification module and the port identification module. The protocol and component identification module is connected to the port identification module. The IP detection module performs IP scanning, the domain name identification module performs domain name identification, the port identification module performs port identification on the obtained IP detection result, the operating system identification module performs operating system identification on the obtained IP detection result, and the protocol and component identification module performs protocol and component scanning on the port identification result.
7. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 2, It is characterized in that The passive flow monitoring unit performs passive monitoring including the following steps: Deploy passive monitoring devices or software at key locations on a network or system to capture and analyze network traffic or system activity; Monitoring devices silently capture passing network traffic or system activity packets; Parse and analyze captured data packets to extract real-time information during data transmission, processing and use; The monitoring results are recorded and saved for subsequent comparison and analysis with active scanning results.
8. The system for verifying structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 2, It is characterized in that The data analysis module verifies structured key data based on the fusion of active scanning and passive monitoring, specifically including word segmentation based on a bidirectional recurrent neural network, text deep feature extraction based on a deep convolutional network, and deep feature classification and grading based on an integrated SVM.
9. A method for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism using the system of claim 1, It is characterized in that The method comprises the following steps: (1) Prepare data and configure parameters; (2) Conduct active scanning; (3) Conduct passive monitoring; (4) Coupling verification of data; (5) Output and respond to results.
10. The method for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 9, It is characterized in that The step (2) specifically comprises the following steps: (2.1) Use active scanning tools to send probe requests to the target system to obtain information about the source, storage location, and access path of key data; (2.2) Receive the response data packet returned by the target system and perform in-depth analysis to extract detailed information of key data.
11. The method for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 9, It is characterized in that The step (3) specifically comprises the following steps: (3.1) Deploy passive monitoring devices on key data transmission paths to capture network traffic or system activity data packets; (3.2) Capture data packets in real time and conduct in-depth analysis to extract real-time information about the data transmission, processing, and use.
12. The method for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 9, It is characterized in that The step (4) specifically comprises the following steps: (4.1) Design a structured verification process based on the verification objectives; (4.2) Perform verification tasks at each stage in sequence according to the predetermined verification process; (4.3) Verify the data source; (4.4) Perform data transmission verification; (4.5) Conduct data processing verification; (4.6) Perform data usage verification.
13. The method for implementing verification processing for structured key data based on the active scanning and passive monitoring fusion mechanism according to claim 9, It is characterized in that The step (5) specifically comprises the following steps: (5.1) Generate a detailed verification report based on the verification results; (5.2) Take appropriate countermeasures based on the recommended measures in the verification report.
14. A device for verifying and processing structured key data based on a fusion mechanism of active scanning and passive monitoring, It is characterized in that The device comprises: a processor configured to execute computer executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the method for verifying structured key data based on the fusion mechanism of active scanning and passive monitoring as described in any one of claims 9 to 13 are implemented.
15. A processor for implementing verification processing for structured key data based on the fusion mechanism of active scanning and passive monitoring, It is characterized in that The processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for verifying structured key data based on the fusion mechanism of active scanning and passive monitoring as described in any one of claims 9 to 13 are implemented.
16. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the method for verifying structured key data based on the active scanning and passive monitoring fusion mechanism as described in any one of claims 9 to 13.
