XDR data quality statistical system on multiple probe nodes

Abstract

The invention discloses an XDR data quality statistical system on multiple probe nodes. The system comprises a probe node data acquisition module, a probe node state acquisition module, a probe node evaluation module, a data processing module and an information sending module. The probe node data acquisition module is used for acquiring data information monitored and acquired by a probe node; the probe node state acquisition module is used for acquiring a probe node state; the probe node evaluation module is used for carrying out evaluation on the probe nodes after the probe nodes are set, and obtaining the evaluation information of the probe nodes; the data processing module is used for processing data information obtained by monitoring of the probe nodes to obtain data quality evaluation information; the probe node state is processed, and probe node setting management and control information is obtained; and processing the probe node evaluation information to obtain probe node management and control information. According to the method, XDR data quality statistical analysis with higher accuracy can be carried out.

Description

Technical Field

[0001] The present invention relates to the field of data quality analysis, and in particular to an XDR data quality statistics system on multiple probe nodes. Background Art

[0002] Probe Node XDR is a concept related to the field of network security. It combines multiple security technologies and components to achieve rapid detection, correlation analysis and response to network threats.

[0003] In the XDR architecture, probe nodes play a vital role. They are usually deployed at key locations in the network to capture and collect key information such as network traffic, system logs, application data, etc. This information is then transmitted to the XDR platform for analysis and correlation to identify potential network threats;

[0004] The data quality statistics system is used during the statistical analysis of XDR data quality on the probe node.

[0005] In actual use, the existing data quality statistics system mostly only performs data quality statistical analysis, without considering the impact of probe node anomalies on data quality statistical analysis, which has a certain impact on the use of the data quality statistics system. Therefore, an XDR data quality statistics system on multiple probe nodes is proposed. Summary of the invention

[0006] The technical problem to be solved by the present invention is: how to solve the problem that in the existing data quality statistics system, only data quality statistical analysis is performed during actual use, and the impact of the abnormality of the probe node on the data quality statistical analysis is not considered, which has a certain impact on the use of the data quality statistical system. An XDR data quality statistics system on multiple probe nodes is provided.

[0007] The present invention solves the above technical problems through the following technical solutions. The present invention includes a probe node data acquisition module, a probe node status acquisition module, a probe node evaluation module, a data processing module and an information sending module;

[0008] The probe node data collection module is used to collect data information acquired by the probe node monitoring;

[0009] The probe node status collection module is used to collect the probe node status;

[0010] The probe node evaluation module is used to collect and evaluate the probe node after setting the probe node, and obtain the probe node evaluation information;

[0011] The data processing module is used to process the data information acquired by the probe node monitoring to obtain data quality assessment information;

[0012] Process the probe node status and obtain the probe node setting management and control information;

[0013] Process the probe node evaluation information and obtain the probe node management and control information;

[0014] The information sending module is used to send data quality assessment information, probe node setting management information and probe node management information to a preset receiving terminal.

[0015] Furthermore, the data quality assessment information includes a first quality assessment, a second quality assessment and a third quality assessment, and the process of obtaining the data quality assessment information is as follows:

[0016] Extracting the data information collected by the probe node monitoring, preprocessing the data information collected by the probe node monitoring, and obtaining the preprocessed data collected by the probe node;

[0017] Perform quality analysis on the data collected by the preprocessed probe nodes to obtain the first score, the second score, and the third score;

[0018] The first score, the second score, and the third score are processed to obtain a comprehensive evaluation. When the comprehensive score is greater than the preset value a1, a first quality evaluation is generated. When the comprehensive score is between the preset values ​​a1 and a2, a second quality evaluation is generated. When the comprehensive score is less than the preset value a1, a third quality evaluation is generated.

[0019] Furthermore, the specific contents of preprocessing the data information acquired by the probe node monitoring are as follows:

[0020] Perform data cleaning, clean the collected raw data, remove duplicate, invalid or abnormal data, check the integrity and consistency of the data, and then unify the data format to unify the data format from different probe nodes.

[0021] Furthermore, the specific process of obtaining the first score, the second score and the third score is as follows:

[0022] Extract the data collected by the preprocessed probe nodes and perform integrity assessment on them to check whether the data is complete, including whether key fields are missing and whether the data is collected at the expected time interval;

[0023] Record the number of key fields missing and the number of times data was not collected at the expected time intervals;

[0024] Calculate the ratio of the number of missing key fields to the total number of key fields to obtain the first evaluation parameter, calculate the ratio of the number of missing key fields to the total number of collection times to obtain the second evaluation parameter;

[0025] Analyze the first evaluation parameter and the second evaluation parameter to obtain a first score. When the first evaluation parameter and the second evaluation parameter are both less than a preset value, the first score is a preset value b1.

[0026] When any one of the first evaluation parameter and the second evaluation parameter is greater than or equal to a preset value, the first score is the preset value b2;

[0027] When any of the first evaluation parameter and the second evaluation parameter is greater than or equal to the preset value, the first score is the preset value b3;

[0028] b1>b2>b3;

[0029] Extract the data collected by the preprocessed probe node, evaluate its accuracy, verify the accuracy of the data, including checking errors, outliers or inconsistencies in the data, verify the acquired data accuracy, analyze the data accuracy to obtain a second score, when the data accuracy is greater than a preset value f1, the second score is a preset value e1, when the data accuracy is between the preset values ​​f1 and f2, the second score is the preset value e1, when the data accuracy is less than the preset value f2, the second score is the preset value e3;

[0030] f1>f2, e1>e2>e3;

[0031] Extract the data collected by the preprocessed probe nodes and conduct a normative evaluation on them: check whether the data meets the predetermined specifications or standards, extract the number of data that does not meet the predetermined specifications or standards, calculate the ratio of the number of data that does not meet the predetermined specifications or standards to the total amount of data, and obtain the evaluation ratio;

[0032] The evaluation ratio is analyzed to obtain a third score. When the evaluation ratio is greater than the preset y1, the third score is the preset value c1. When the evaluation ratio is between the preset y1 and y2, the third score is the preset value c2. When the evaluation ratio is less than the preset y2, the third score is the preset value c3.

[0033] c1>c2>c3, y1>y2.

[0034] Furthermore, the comprehensive score is obtained as follows: extracting a first score, a second score, and a third score;

[0035] The first score, the second score and the third score are marked as G1, G2 and G3;

[0036] Assign a correction value T1 to the first score G1, a correction value T2 to the second score G2, and a correction value T3 to the third score G3, T1+T2+T3=1, T1=T2<T3;

[0037] The comprehensive score can be obtained through the formula G1*T1+G2*T2+G3*T3=Gg.

[0038] Furthermore, the specific acquisition process of the probe node setting management and control information is as follows: extracting the collected probe node status;

[0039] The probe node status includes the number of probe nodes set, the bandwidth usage of a single probe node, the size of the monitored network area covered by a single probe node, and the number of monitored traffic paths covered by a single probe node;

[0040] Then collect the size of the current network area to be monitored, the number of monitored traffic paths, and the current bandwidth information;

[0041] The set number of probe nodes is marked as H, the bandwidth occupancy of a single probe node is marked as U1, and the current bandwidth information is marked as U2;

[0042] The bandwidth evaluation parameter Uu is obtained by the formula U2-H*U1*α=Uu, where α is the correction value, 1.1≤α≤1.2;

[0043] When the bandwidth evaluation parameter Uu is less than the preset value, the probe node setting control information is generated;

[0044] When the bandwidth evaluation parameter Uu is greater than or equal to the preset value, the current network area size to be monitored is marked as X1, and the size of the monitored network area covered by the single probe node is marked as X2;

[0045] The number of monitored traffic paths is marked as P1, and the number of monitored traffic paths covered by a single probe node is marked as P2;

[0046] The path evaluation parameter Pp is obtained through the formula H1*P1-P2=Pp. If the path evaluation parameter Pp is less than the preset value, the probe node setting control information is generated.

[0047] The coverage evaluation parameter Xx is obtained through the formula H1*X1-X2=Xx. If the coverage evaluation parameter Xx is less than the preset value, the probe node setting management and control information is generated.

[0048] Furthermore, the specific processing process of the probe node control information is as follows: extracting the collected probe node evaluation information, the probe node evaluation information is to perform access test on the probe node, record the number of disconnections of the probe node within a preset time and the number of vulnerabilities obtained by scanning the probe node for vulnerabilities within the preset time;

[0049] When the number of disconnections of the probe node within a preset time period is greater than a preset number or the number of vulnerabilities obtained by scanning the vulnerability of the probe node within a preset time period is greater than a preset value, the probe node management and control information is generated.

[0050] Compared with the prior art, the present invention has the following advantages: the XDR data quality statistics system on multiple probe nodes can intuitively understand the quality of XDR data on the probe nodes through the generated data quality evaluation information, and then adjust the relevant equipment of the corresponding probe nodes according to the actual data quality status; through the set probe node setting management and control information, the relevant information statistics of the probe nodes are collected, and then it is timely discovered whether the set verification nodes meet the actual needs, and it is evaluated whether they will occupy excessive bandwidth and affect the normal use of the monitoring system; when an abnormality is found, the management and control information is generated to prompt the probe node type to change, thereby ensuring that while better realizing data monitoring, the impact of the probe node setting on the monitored system is reduced; the probe nodes are tested through the generated probe node management and control information, the stability of the probe stage is understood, and more stable data quality statistics are guaranteed, making the system more worthy of popularization and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0052] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0053] like Figure 1 As shown, this embodiment provides a technical solution: an XDR data quality statistics system on multiple probe nodes, including a probe node data collection module, a probe node status collection module, a probe node evaluation module, a data processing module and an information sending module;

[0054] The probe node data collection module is used to collect data information acquired by the probe node monitoring;

[0055] The probe node status collection module is used to collect the probe node status;

[0056] The probe node evaluation module is used to collect and evaluate the probe node after setting the probe node, and obtain the probe node evaluation information;

[0057] The data processing module is used to process the data information acquired by the probe node monitoring to obtain data quality assessment information;

[0058] Process the probe node status and obtain the probe node setting management and control information;

[0059] Process the probe node evaluation information and obtain the probe node management and control information;

[0060] The information sending module is used to send data quality assessment information, probe node setting management information and probe node management information to a preset receiving terminal.

[0061] The data quality assessment information includes a first quality assessment, a second quality assessment and a third quality assessment. The process of obtaining the data quality assessment information is as follows:

[0062] Extracting the data information collected by the probe node monitoring, preprocessing the data information collected by the probe node monitoring, and obtaining the preprocessed data collected by the probe node;

[0063] Perform quality analysis on the data collected by the preprocessed probe nodes to obtain the first score, the second score, and the third score;

[0064] The first score, the second score, and the third score are processed to obtain a comprehensive evaluation. When the comprehensive score is greater than a preset value a1, a first quality evaluation is generated. When the comprehensive score is between the preset values ​​a1 and a2, a second quality evaluation is generated. When the comprehensive score is less than a1, a third quality evaluation is generated.

[0065] The first quality assessment refers to the data quality difference of the XDR data quality statistics on the probe node, and operations such as probe node replacement are required to improve data quality;

[0066] The second quality assessment means that there are certain problems with the data quality of the XDR data quality statistics on the probe node, and optimization is required;

[0067] The third quality assessment means that the data quality of the XDR data quality statistics on the probe node is good, and the file quality needs to be monitored regularly.

[0068] The specific contents of preprocessing the data information acquired by the probe node monitoring are as follows:

[0069] Data cleaning: Clean the collected raw data to remove duplicate, invalid or abnormal data.

[0070] Check the integrity and consistency of data to ensure that the data conforms to the predetermined format and specifications.

[0071] Data format unification: Unify the data formats from different probe nodes to facilitate subsequent data analysis and processing.

[0072] A standardized data model or protocol may be employed.

[0073] The specific process of obtaining the first score, the second score and the third score is as follows:

[0074] Extract the data collected by the preprocessed probe nodes and perform integrity assessment on them to check whether the data is complete, including whether key fields are missing and whether the data is collected at the expected time interval;

[0075] Record the number of key fields missing and the number of times data was not collected at the expected time intervals;

[0076] Calculate the ratio of the number of missing key fields to the total number of key fields to obtain the first evaluation parameter, calculate the ratio of the number of missing key fields to the total number of collection times to obtain the second evaluation parameter;

[0077] Analyze the first evaluation parameter and the second evaluation parameter to obtain a first score. When the first evaluation parameter and the second evaluation parameter are both less than a preset value, the first score is a preset value b1.

[0078] When any one of the first evaluation parameter and the second evaluation parameter is greater than or equal to a preset value, the first score is the preset value b2;

[0079] When any of the first evaluation parameter and the second evaluation parameter is greater than or equal to the preset value, the first score is the preset value b3;

[0080] b1>b2>b3;

[0081] Extract the data collected by the preprocessed probe node, evaluate its accuracy, verify the accuracy of the data, including checking errors, outliers or inconsistencies in the data, verify the acquired data accuracy, analyze the data accuracy to obtain a second score, when the data accuracy is greater than a preset value f1, the second score is a preset value e1, when the data accuracy is between the preset values ​​f1 and f2, the second score is the preset value e1, when the data accuracy is less than the preset value f2, the second score is the preset value e3;

[0082] f1>f2, e1>e2>e3;

[0083] Extract the data collected by the preprocessed probe nodes and conduct a normative evaluation on them: check whether the data meets the predetermined specifications or standards, extract the number of data that does not meet the predetermined specifications or standards, calculate the ratio of the number of data that does not meet the predetermined specifications or standards to the total amount of data, and obtain the evaluation ratio;

[0084] The evaluation ratio is analyzed to obtain a third score. When the evaluation ratio is greater than the preset y1, the third score is the preset value c1. When the evaluation ratio is between the preset y1 and y2, the third score is the preset value c2. When the evaluation ratio is less than the preset y2, the third score is the preset value c3.

[0085] c1>c2>c3, y1>y2;

[0086] Through the above process, more accurate first scores, second scores, and third scores are obtained, which can ensure the accuracy of the final generated comprehensive score.

[0087] The process of obtaining the comprehensive score is as follows: extracting the first score, the second score and the third score;

[0088] The first score, the second score and the third score are marked as G1, G2 and G3;

[0089] Assign a correction value T1 to the first score G1, a correction value T2 to the second score G2, and a correction value T3 to the third score G3, T1+T2+T3=1, T1=T2<T3;

[0090] The comprehensive score can be obtained through the formula G1*T1+G2*T2+G3*T3=Gg.

[0091] The specific acquisition process of the probe node setting management and control information is as follows: extracting the collected probe node status;

[0092] The probe node status includes the number of probe nodes set, the bandwidth usage of a single probe node, the size of the monitored network area covered by a single probe node, and the number of monitored traffic paths covered by a single probe node;

[0093] Then collect the size of the current network area to be monitored, the number of monitored traffic paths, and the current bandwidth information;

[0094] The set number of probe nodes is marked as H, the bandwidth occupancy of a single probe node is marked as U1, and the current bandwidth information is marked as U2;

[0095] The bandwidth evaluation parameter Uu is obtained by the formula U2-H*U1*α=Uu, where α is the correction value, 1.1≤α≤1.2;

[0096] When the bandwidth evaluation parameter Uu is less than the preset value, the probe node setting control information is generated;

[0097] When the bandwidth evaluation parameter Uu is greater than or equal to the preset value, the current network area size to be monitored is marked as X1, and the size of the monitored network area covered by the single probe node is marked as X2;

[0098] The number of monitored traffic paths is marked as P1, and the number of monitored traffic paths covered by a single probe node is marked as P2;

[0099] The path evaluation parameter Pp is obtained through the formula H1*P1-P2=Pp. If the path evaluation parameter Pp is less than the preset value, the probe node setting control information is generated.

[0100] The coverage evaluation parameter Xx is obtained through the formula H1*X1-X2=Xx. If the coverage evaluation parameter Xx is less than the preset value, the probe node setting control information is generated.

[0101] Through the above process, we can evaluate the impact of probe deployment on network performance, ensure that the deployment will not cause excessive burden on the network or affect user experience, and select the appropriate probe type and model according to monitoring needs. Different probes may have different functions and performance characteristics, and need to be selected according to actual conditions. This will better ensure the accuracy of XDR data quality statistics on multiple probe nodes.

[0102] The specific processing process of the probe node control information is as follows: extracting the collected probe node evaluation information, the probe node evaluation information is to perform access test on the probe node, record the number of disconnection information of the probe node within a preset time and the number of vulnerabilities obtained by scanning the vulnerability of the probe node within the preset time;

[0103] When the number of disconnections of the probe node within the preset time period is greater than the preset number or the number of vulnerabilities obtained by scanning the vulnerability of the probe node within the preset time period is greater than the preset value, the probe node control information is generated;

[0104] The specific content of the probe node management and control information is that the stability and security of the probe node are poor and need to be replaced to ensure the accuracy of the results of the XDR data quality statistics on multiple probe nodes.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An XDR data quality statistics system on multiple probe nodes, characterized in that: It includes a probe node data collection module, a probe node status collection module, a probe node evaluation module, a data processing module and an information sending module; The probe node data collection module is used to collect data information acquired by the probe node monitoring; The probe node status collection module is used to collect the probe node status; The probe node evaluation module is used to collect and evaluate the probe node after setting the probe node, and obtain the probe node evaluation information; The data processing module is used to process the data information acquired by the probe node monitoring to obtain data quality assessment information; Process the probe node status and obtain the probe node setting management and control information; Process the probe node evaluation information and obtain the probe node management and control information; The information sending module is used to send data quality assessment information, probe node setting management information and probe node management information to a preset receiving terminal.

2. The XDR data quality statistics system on multiple probe nodes according to claim 1, characterized in that: The data quality assessment information includes a first quality assessment, a second quality assessment and a third quality assessment. The process of obtaining the data quality assessment information is as follows: Extracting the data information collected by the probe node monitoring, preprocessing the data information collected by the probe node monitoring, and obtaining the preprocessed data collected by the probe node; Perform quality analysis on the data collected by the preprocessed probe nodes to obtain the first score, the second score, and the third score; The first score, the second score, and the third score are processed to obtain a comprehensive evaluation. When the comprehensive score is greater than the preset value a1, a first quality evaluation is generated. When the comprehensive score is between the preset values ​​a1 and a2, a second quality evaluation is generated. When the comprehensive score is less than the preset value a1, a third quality evaluation is generated.

3. The XDR data quality statistics system on multiple probe nodes according to claim 2, characterized in that: The specific contents of preprocessing the data information acquired by the probe node monitoring are as follows: Perform data cleaning, clean the collected raw data, remove duplicate, invalid or abnormal data, check the integrity and consistency of the data, and then unify the data format to unify the data format from different probe nodes.

4. The XDR data quality statistics system on multiple probe nodes according to claim 2, characterized in that: The specific process of obtaining the first score, the second score and the third score is as follows: Extract the data collected by the preprocessed probe nodes and perform integrity assessment on them to check whether the data is complete, including whether key fields are missing and whether the data is collected at the expected time interval; Record the number of key fields missing and the number of times data was not collected at the expected time intervals; Calculate the ratio of the number of missing key fields to the total number of key fields to obtain the first evaluation parameter, calculate the ratio of the number of missing key fields to the total number of collection times to obtain the second evaluation parameter; Analyze the first evaluation parameter and the second evaluation parameter to obtain a first score. When the first evaluation parameter and the second evaluation parameter are both less than a preset value, the first score is a preset value b1. When any one of the first evaluation parameter and the second evaluation parameter is greater than or equal to a preset value, the first score is the preset value b2; When any of the first evaluation parameter and the second evaluation parameter is greater than or equal to the preset value, the first score is the preset value b3; b1>b2>b3; Extract the data collected by the preprocessed probe node, evaluate its accuracy, verify the accuracy of the data, including checking errors, outliers or inconsistencies in the data, verify the acquired data accuracy, analyze the data accuracy to obtain a second score, when the data accuracy is greater than a preset value f1, the second score is a preset value e1, when the data accuracy is between the preset values ​​f1 and f2, the second score is the preset value e1, when the data accuracy is less than the preset value f2, the second score is the preset value e3; f1>f2, e1>e2>e3; Extract the data collected by the preprocessed probe nodes and conduct a normative evaluation on them: check whether the data meets the predetermined specifications or standards, extract the number of data that does not meet the predetermined specifications or standards, calculate the ratio of the number of data that does not meet the predetermined specifications or standards to the total amount of data, and obtain the evaluation ratio; The evaluation ratio is analyzed to obtain a third score. When the evaluation ratio is greater than the preset y1, the third score is the preset value c1. When the evaluation ratio is between the preset y1 and y2, the third score is the preset value c2. When the evaluation ratio is less than the preset y2, the third score is the preset value c3. c1>c2>c3, y1>y2.

5. The XDR data quality statistics system on multiple probe nodes according to claim 2, characterized in that: The process of obtaining the comprehensive score is as follows: extracting the first score, the second score and the third score; The first score, the second score and the third score are marked as G1, G2 and G3; Assign a correction value T1 to the first score G1, a correction value T2 to the second score G2, and a correction value T3 to the third score G3, T1+T2+T3=1, T1=T2<T3; The comprehensive score can be obtained through the formula G1*T1+G2*T2+G3*T3=Gg.

6. The XDR data quality statistics system on multiple probe nodes according to claim 1, characterized in that: The specific acquisition process of the probe node setting management and control information is as follows: extracting the collected probe node status; The probe node status includes the number of probe nodes set, the bandwidth usage of a single probe node, the size of the monitored network area covered by a single probe node, and the number of monitored traffic paths covered by a single probe node; Then collect the size of the current network area to be monitored, the number of monitored traffic paths, and the current bandwidth information; The set number of probe nodes is marked as H, the bandwidth occupancy of a single probe node is marked as U1, and the current bandwidth information is marked as U2; The bandwidth evaluation parameter Uu is obtained by the formula U2-H*U1*α=Uu, where α is the correction value, 1.1≤α≤1.2; When the bandwidth evaluation parameter Uu is less than the preset value, the probe node setting control information is generated; When the bandwidth evaluation parameter Uu is greater than or equal to the preset value, the current network area size to be monitored is marked as X1, and the size of the monitored network area covered by the single probe node is marked as X2; The number of monitored traffic paths is marked as P1, and the number of monitored traffic paths covered by a single probe node is marked as P2; The path evaluation parameter Pp is obtained through the formula H1*P1-P2=Pp. If the path evaluation parameter Pp is less than the preset value, the probe node setting control information is generated. The coverage evaluation parameter Xx is obtained through the formula H1*X1-X2=Xx. If the coverage evaluation parameter Xx is less than the preset value, the probe node setting management and control information is generated.

7. The XDR data quality statistics system on multiple probe nodes according to claim 1, characterized in that: The specific processing process of the probe node control information is as follows: extracting the collected probe node evaluation information, the probe node evaluation information is to perform access test on the probe node, record the number of disconnection information of the probe node within a preset time and the number of vulnerabilities obtained by scanning the vulnerability of the probe node within the preset time; When the number of disconnections of the probe node within a preset time period is greater than a preset number or the number of vulnerabilities obtained by scanning the vulnerability of the probe node within a preset time period is greater than a preset value, the probe node management and control information is generated.

Images