Network data asset security discovery system based on distributed machine learning

Abstract

The invention provides a network data asset security discovery system based on distributed machine learning, and belongs to the technical field of data processing systems. The system comprises an acquisition module used for acquiring historical network data through a data center node by taking a first preset duration as a period; the establishing module is used for establishing a support vector description model based on historical network data; the first issuing module is used for issuing the support vector description model to the data processing node; the first acquisition module is used for approximately acquiring real-time network data of each network data asset in combination with a Chebyshev polynomial; the second acquisition module is used for acquiring load data of each data processing node; the second issuing module is used for issuing the real-time network data to the data processing node by taking the minimum network data processing delay as a target; and the identification module is used for carrying out security identification on the received real-time network data and outputting abnormal real-time network data. And the anomaly detection rate is improved while the anomaly detection accuracy is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing systems, and in particular relates to a network data asset security discovery system based on distributed machine learning. Background Art

[0002] Network data assets refer to virtual or physical devices and their related resources existing in cyberspace, including but not limited to servers, routers, switches, virtual machines, containers, IP addresses, MAC addresses, domain names, network interfaces, protocol ports, etc. They are the core components of network communication and data transmission, and involve key nodes of data flow; distributed machine learning is a technology that decomposes machine learning tasks and distributes them to multiple computing nodes for processing, which is suitable for large-scale data computing and analysis. By distributing data, models or tasks to multiple nodes, distributed machine learning can improve computing efficiency, processing power and scalability, while supporting collaborative learning in a distributed environment.

[0003] Security discovery of network data assets can help identify abnormal traffic, intrusion behavior or configuration vulnerabilities in a timely manner, preventing data leakage, device loss of control or system interruption. As the scale and complexity of the network increase, security risks are also rising. Using advanced technology to conduct security discovery of network data assets can ensure the normal operation of key assets, enhance network security situation awareness, and reduce the impact of security threats on the business.

[0004] However, the existing network data asset discovery process does not take into account the high concurrent data flow of each asset in the actual process, resulting in long task queues and extremely high delays in data processing nodes, which is extremely detrimental to the security of network data assets. Moreover, the existing network data asset discovery process often performs anomaly detection based on fixed thresholds, which has insufficient anomaly detection accuracy, resulting in frequent false alarms and insufficient practicality. Summary of the invention

[0005] In order to solve the problem that the network data asset discovery process in the prior art does not take into account the high concurrent data flow of each asset in the actual process, resulting in an overly long task queue in the data processing node and extremely high delay, which is extremely detrimental to the security of network data assets. In addition, the existing network data asset discovery process often performs anomaly detection based on a fixed threshold, and the anomaly detection accuracy is insufficient, which leads to frequent false alarms and insufficient practicality. The present invention provides a network data asset security discovery system based on distributed machine learning.

[0006] The present invention provides a network data asset security discovery system based on distributed machine learning, the network data asset security discovery system comprising a data center node and a plurality of data processing nodes all connected to the data center node;

[0007] The network data asset security discovery system based on distributed machine learning also includes:

[0008] An acquisition module, configured to acquire historical network data through a data center node at a first preset time period, wherein the historical network data includes asset interaction times of various network data assets and asset interaction frequencies at the asset interaction times;

[0009] Establish a module for combining the distribution center of historical network data and establishing a support vector description model based on historical network data to determine the minimum hypersphere for describing the normal operation status of network data assets;

[0010] A first sending module is used to send the support vector description model to each data processing node;

[0011] The first acquisition module is used to approximate the real-time network data of each network data asset by combining Chebyshev polynomials;

[0012] The second acquisition module is used to collect load data of each data processing node;

[0013] The second sending module is used to send the real-time network data to each data processing node based on the load data and with the goal of minimizing the network data processing delay;

[0014] The identification module is used to safely identify the received real-time network data through the support vector description model in each data processing node, and output abnormal real-time network data, that is, output the real-time network data outside the minimum hypersphere.

[0015] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0016] In the present invention, the data center node is used as the data allocation center, and the data processing tasks are issued according to the load status of each data processing node, which effectively avoids the problem of too long task queue and too high processing delay, ensures load balancing, improves data processing efficiency and real-time performance, and makes the security discovery of network data assets more accurate and efficient, which is suitable for large-scale high-concurrency scenarios. In the process of security identification of network data assets, combined with the distribution center of historical network data, a support vector description model based on historical network data is established to determine the minimum hypersphere used to describe the normal operation status of network data assets, which can not only adapt to the dynamically changing network environment, but also more accurately distinguish normal and abnormal data, greatly improving the accuracy of abnormal detection. In the process of real-time data collection, the real-time network data of each network data asset is estimated and approximated by combining Chebyshev polynomials, which can effectively deal with the problems of data acquisition difficulties and excessive delay caused by data missing and high concurrent data flow, and can quickly obtain network data in high concurrency and data missing environments, significantly reduce delays and computational complexity, ensure the real-time and reliability of data collection, and improve the efficiency and practicality of network data asset security discovery. Finally, the established support vector description model is used at each data processing node to safely and quickly identify the collected data, with low latency and high accuracy, and can cope with large-scale data streams that conform to actual conditions. Throughout the entire process, from data collection to data identification, the delay of asset identification can be effectively reduced, and abnormal data status can be effectively, accurately and timely detected based on historical data rules, with low false alarm rate and higher practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The preferred implementation modes will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present invention.

[0018] Figure 1 It is a structural diagram of a network data asset security discovery system based on distributed machine learning provided by the present invention;

[0019] Figure 2 It is a structural schematic diagram of the relationship between a data center node and a data processing node provided by the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0021] In order to simplify the drawings, only the parts related to the invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0022] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] In this document, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] In addition, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] In one embodiment, referring to Figure 1 , showing a schematic diagram of the structure of a network data asset security discovery system based on distributed machine learning provided by the present invention. Figure 2 , showing a structural diagram of resource allocation of a medical community provided by the present invention.

[0026] The present invention provides a network data asset security discovery system based on distributed machine learning. The network data asset security discovery system includes a data center node and a plurality of data processing nodes connected to the data center node.

[0027] Among them, the network data asset security discovery system collects network data through the data center nodes and realizes intelligent distribution through the collaborative work of the data center nodes and multiple data processing nodes (1 to n). By processing network data in a distributed manner, it can discover anomalies in real time and realize efficient and secure data asset management.

[0028] The network data asset security discovery system based on distributed machine learning also includes:

[0029] The acquisition module 1 is used to acquire historical network data through a data center node with a first preset time period as a period.

[0030] Among them, the historical network data includes the asset interaction time of each network data asset and the asset interaction frequency at the asset interaction time.

[0031] Among them, network data assets are various virtual devices or physical devices existing in the network space included in the monitoring scope, including IP addresses, routers, etc. The interaction time is the time when the network data asset has data transmission, which can be uploading data or receiving data. The asset interaction frequency is the total number of data transmissions at the interaction time, and each data transmission will be recorded separately as an asset interaction. All historical data collected in step S1 are normal operation data.

[0032] It should be noted that those skilled in the art can set the first preset duration according to actual needs, and the present invention does not limit this. Specifically, the preset duration can be one minute, one hour, or one day, etc.

[0033] Optionally, the first preset duration is determined by first performing Fourier transform on the asset interaction frequency time series in the historical network data to obtain a spectrum. Then, the square of the modulus of the spectrum is used as the contribution intensity of different occurrence times to the original signal, i.e., the spectrum, to obtain a frequency spectrum density including each asset interaction frequency reflecting the signal intensity of each frequency, wherein the peak frequency f peak As the most significant periodic component, and based on this periodic component, the period is calculated Then, the time series is divided based on this period as the basic window length W. Then, a simple k-means clustering algorithm is used to cluster the asset interaction frequency time series to obtain multiple candidate windows T that can reflect the periodic law. 1 ,T 2 ,... Then calculate the asset interaction frequency variance within each base window and candidate window It is used to measure the volatility or regularity of the interaction frequency within the time window. The smaller the variance of the asset interaction frequency, the more significant the periodicity or stability of the interaction frequency within the window. F k represents the asset interaction frequency at the kth moment in the base window or candidate window W, μ W represents the mean of the interaction frequency within the basic window or candidate window W. Finally, the comprehensive T freq And the candidate window extracted from the cluster selects the first preset duration, that is,

[0034]

[0035] It is understandable that by dynamically determining the first preset time length through Fourier transform and K-means clustering, the periodicity of asset interaction frequency can be accurately captured, avoiding misjudgment of abnormal data caused by fixed window length. This method based on periodicity and variance optimization provides stable and regular input data for the subsequent establishment of a support vector description model, thereby improving the model's fitting accuracy for normal data distribution and improving the accuracy and robustness of anomaly detection.

[0036] In one possible implementation, the network data assets include servers, routers, switches, personal terminal devices, virtual machines, containers, IP addresses, MAC addresses, domain names, network interfaces, and protocol ports.

[0037] Module 2 is established to combine the distribution center of historical network data and establish a support vector description model based on historical network data to determine the minimum hypersphere used to describe the normal operation status of network data assets.

[0038] Among them, the Support Vector Data Description (SVDD) model is a machine learning model for anomaly detection. It constructs a minimum hypersphere in high-dimensional space to contain most of the normal data in the sphere, and any data outside the sphere is judged as abnormal data. The minimum hypersphere is a geometric body in high-dimensional space. Its definition is to contain as many data points as possible in the sphere with the smallest radius, while allowing a small number of data points to deviate from the sphere and controlling the deviation through slack variables. This method can describe the normal distribution range of data and is used to detect abnormal data that deviates from the range.

[0039] By combining the distribution center of historical network data, the support vector description model (SVDD) is used to establish a minimum hypersphere to accurately describe the normal operating state range of network data assets, thereby providing a benchmark for subsequent anomaly detection. This step can dynamically adapt to data changes, improve the accuracy and practicality of anomaly detection, and significantly reduce the false alarm rate and missed alarm rate.

[0040] In a possible implementation, the establishment module 2 is specifically used for:

[0041] Based on the asset interaction time and the asset interaction frequency at the asset interaction time, the historical network data state vector of each network data asset is established:

[0042]

[0043] Among them, t k represents the kth asset interaction moment of the network data asset, Indicates that the i-th network data asset is at t k The frequency of asset interaction, xi,t represents the interaction state vector of the ith network data asset at time t, T 1 ,T 2 They respectively represent the start time and end time of the preset duration.

[0044] Among them, the historical network data state vector marks the asset interaction frequency of each network data asset at each asset interaction moment within a recording period. This data is used for subsequent data analysis to capture the periodic laws of each network data asset in time and the upper and lower limits of the interaction frequency in state.

[0045] Establish the smallest hypersphere that describes the normal operation status of network data assets:

[0046]

[0047] ||x i -c|| 2 ≤R 2 +ξ i ,ξ i ≥0

[0048] e≤ε

[0049]

[0050] Where R and c represent the radius and center of the hypersphere, respectively, i represents the slack variable for the ith network data asset, i = 1, 2, ..., N, N represents the total number of network data assets, C represents the penalty coefficient for controlling the degree of hypersphere relaxation, x i represents the interaction state vector of the ith network data asset at any time, f represents the distribution center of historical network data, that is, the data center of mass of all historical network data, e represents the distance from the center of the hypersphere c to the data center of mass f, that is, the eccentricity, λ represents the penalty coefficient when the eccentricity e exceeds the dynamic eccentricity ε, L(R,c,ε,λ) represents the Lagrangian function about R,c,ε,λ, and |||| represents the calculation of the Euclidean distance.

[0051] The dynamic eccentricity is determined according to the optimization condition of Lagrangian duality theory, namely the KKT condition. The penalty coefficient for the eccentricity exceeding the dynamic eccentricity ε is used as a Lagrangian multiplier, which can be solved by the Lagrangian function in the process of solving the minimum hypersphere. The data centroid is the geometric mean of all data points. In the process of solving the minimum hypersphere, f is used as the eccentric point to correct the hypersphere center c of the minimum hypersphere to ensure that the hypersphere center does not deviate too much from the data center f, so as to better fit the actual data distribution of historical network data.

[0052] Using the Gaussian kernel function, the historical network data state vectors of each network data asset are mapped in a pairwise high-dimensional manner:

[0053]

[0054] Among them, x j represents the interaction state vector of the jth network data asset at any time, K(x i ,x j ) represents the description x i and x j Gaussian kernel function value of similarity in high-dimensional space, exp represents the natural exponential function, σ i and σ j Respectively represent the description x i and x j Kernel width of the local distribution properties.

[0055] In a possible implementation, the calculation formula of the core width is specifically:

[0056]

[0057] in, and Respectively represent x i and x j The mth nearest neighbor point of , where M represents the preset neighborhood value.

[0058] Specifically, by selecting the M nearest neighbor points of each point to calculate the kernel width, it is possible to adapt to the local data distribution characteristics, thereby retaining the local structure and similarity of the data during the high-dimensional mapping process. This dynamic kernel width method can effectively reduce the impact of uneven distribution in high-dimensional data, avoid overfitting or underfitting problems caused by fixed kernel width, and improve the model's ability to handle complex data and the accuracy of high-dimensional mapping. By dynamically adjusting the kernel width and adapting to the local data distribution characteristics, a more accurate support vector description model can be established to more accurately identify anomalies in network data assets, effectively reduce false alarm rates and missed alarm rates, and improve the reliability and adaptability of security detection.

[0059] It should be noted that those skilled in the art can set the size of the preset neighborhood value according to actual needs, and the present invention is not limited thereto.

[0060] Solve the interaction state vector weight value through the Lagrangian optimization algorithm:

[0061]

[0062] Among them, α i represents the interaction state vector weight value of the i-th network data asset, α jRepresents the weight value of the interaction state vector of the j-th network data asset.

[0063] Calculate the radius and center of the hypersphere based on the weight value of the interaction state vector obtained:

[0064]

[0065] R 2 =||φ(x s )-c|| 2

[0066] Among them, x s Represents the interaction state vector of any network data asset, |||| 2 Represents calculation of the square of the Euclidean norm.

[0067] Specifically, in the process of establishing the support vector machine description model, first, the historical network data state vector is constructed according to the asset interaction time and interaction frequency, marking the periodic law and frequency upper and lower limit characteristics of the network data assets. Then, the support vector description model is used to map the state vector to a high-dimensional space through the Gaussian kernel function, and the minimum hypersphere is constructed. The center and radius of the hypersphere are used to describe the distribution range of normal data. In this process, the eccentricity e is introduced to correct the center c of the hypersphere to ensure that c is closer to the distribution center f of the historical data. The introduction of eccentricity avoids the deviation of the center of the hypersphere from the actual data centroid, especially in the case of asymmetric data distribution or interference from abnormal points, which can improve the robustness and fitting accuracy of the model. The weight value of the interaction state vector is calculated by the Lagrangian optimization algorithm, and the radius and center of the hypersphere are finally determined, so that the hypersphere can dynamically adapt to the data distribution, comprehensively and accurately describe the normal operation status of network data assets, and provide strong support for subsequent anomaly detection. The application of eccentricity significantly improves the adaptability and detection accuracy of the model to complex data distribution.

[0068] The first sending module 3 is used to send the support vector description model to each data processing node.

[0069] It should be noted that the first sending module distributes the support vector description model to each data processing node, so that it can independently perform real-time data anomaly detection, thereby improving the system's distributed processing capability and detection efficiency.

[0070] The first collection module 4 is used to perform approximate collection of real-time network data of each network data asset in combination with Chebyshev polynomials.

[0071] Among them, Chebyshev polynomials are a special type of orthogonal polynomials, which are widely used in function approximation, numerical optimization and error minimization. It has good numerical stability and fast convergence characteristics, and can approximate the target value in a low-order form in a high-dimensional complex function, thereby reducing the amount of calculation and improving efficiency. The first acquisition module uses Chebyshev polynomials to estimate the real-time network data. By using its efficient approximation and error control characteristics, it can quickly and accurately obtain real-time network data in the case of high concurrent data streams and partial data missing, significantly reducing latency and ensuring the stability and reliability of data acquisition.

[0072] In a possible implementation manner, the first acquisition module 4 is specifically used for:

[0073] Count the total number of network data assets and record the asset interaction time of each network data asset.

[0074] The number of asset interactions of each network data asset is counted through Chebyshev polynomials:

[0075]

[0076] Among them, N(A i ) represents network data asset A i The total number of interactions, S represents the interaction subset containing network data assets, |S| represents the interaction subset containing network data assets A i The number of interaction subsets, k represents the upper limit of interaction subsets related to m, and m represents the total number of network data assets. Represents other network data assets A in S j The total number of interactions, represents the Chebyshev polynomial coefficients used to adjust the influence weight of the interaction subset S, and ! represents the factorial operation.

[0077] The number of asset interactions is taken as the asset interaction frequency, and the real-time network data is obtained by combining the asset interaction moments.

[0078] in, When the number of subsets is large, the weight is reduced by combining the proportions. Control when the data volume is large and fast convergence is required. The symmetry is guaranteed, and the influence ratio of low-dimensional subsets and high-dimensional subsets is strictly distributed based on the number of combinations, which can avoid excessive participation of certain subsets and lead to estimation bias. The combined effect ensures that the influence of high-dimensional subsets is significantly reduced, while having a rapid convergence effect on errors. By adjusting the ratio of k and m, the balance of influence between low-dimensional and high-dimensional subsets can be controlled to adapt to data scenarios of different scales. It maintains symmetry and quickly converges high-dimensional subset weights through exponential decay, which is well suited for large-scale network asset interaction analysis.

[0079] It should be noted that by collecting real-time network data through the polynomial combination and weight adjustment method of Chebyshev polynomials, the information of partial interaction subsets can be effectively used to quickly estimate the total number of interactions of assets, avoiding the scanning of the full amount of data and improving the efficiency of data processing. In addition, this method balances the influence of low-dimensional and high-dimensional subsets by dynamically adjusting the ratio of subset k and the total number of assets m, reduces the interference of high-dimensional subsets on the results, and ensures the symmetry and accuracy of data statistics. Even in scenarios with large data scale, partial data missing or high concurrency, it can still converge quickly, reduce latency and ensure efficient collection of real-time network data.

[0080] The second acquisition module 5 is used to acquire the load data of each data processing node.

[0081] In a possible implementation, the load data includes CPU usage, memory occupancy, network bandwidth utilization, and length of a queue of pending tasks.

[0082] It should be noted that load data comprehensively evaluates the operating status of each node by monitoring CPU usage, memory occupancy, network bandwidth utilization and the length of the pending task queue, providing real-time basis for task allocation and ensuring load balancing and system efficiency.

[0083] The second sending module 6 is used to send the real-time network data to each data processing node based on the load data and with the goal of minimizing the network data processing delay.

[0084] It can be understood that real-time network data is dynamically allocated according to the load data of the data processing node (including CPU usage, memory occupancy, network bandwidth utilization, etc.) to minimize network data processing delays, ensure system load balancing, and improve data processing efficiency and real-time performance.

[0085] In a possible implementation manner, the second sending module 6 is specifically configured to:

[0086] Determine the data delivery constraint function with the goal of minimizing network data processing delay:

[0087]

[0088] Among them, Minimize means taking the minimum value, R q , B q and L q They represent the processing rate, network bandwidth and network delay of data processing node q respectively, q=1,2,…,Q, Q represents the total number of data processing nodes, D q Indicates the amount of network data sent to node q, Represents for any data processing node q.

[0089] Determine the node priority of each data processing node:

[0090]

[0091] Among them, P q Indicates the node priority of data processing node q.

[0092] Under the condition that the data allocation constraint function is met, the real-time network data is sent to each data processing node according to the node priority.

[0093] It should be noted that the second sending module first optimizes the data allocation strategy by constructing a data sending amount constraint function with the goal of minimizing network data processing delay, comprehensively considering the processing rate, network bandwidth and delay of the data processing node. Then, the node priority is calculated according to the processing rate, bandwidth and delay of each node, and the high-priority node receives the task first. Finally, under the premise of meeting the data allocation amount constraint conditions, the real-time network data is allocated to each data processing node according to the priority. This process significantly improves the efficiency and fairness of data processing, ensures system load balancing and delay minimization, and adapts to the complex needs of dynamic network environments.

[0094] The identification module 7 is used to perform security identification on the received real-time network data through the support vector description model in each data processing node, and output abnormal real-time network data, that is, output real-time network data outside the minimum hypersphere.

[0095] In a possible implementation, the identification module is specifically used for:

[0096] Establish a screening formula for abnormal real-time network data:

[0097] ||φ(x)-c|| 2 >R 2

[0098]

[0099] Where x represents the input real-time network data, φ(x) represents the high-dimensional mapping value of x, K(x,x) represents the inner product of the high-dimensional mapping value of x, and K(x,x) represents the inner product of the high-dimensional mapping value of x. i ) represents x and x i The kernel function value in high-dimensional space is the similarity.

[0100] The real-time network data that meets the screening formula is output as abnormal real-time network data.

[0101] Specifically, in this process, the recognition module maps the input real-time network data to a high-dimensional space through a high-dimensional mapping function, and uses the kernel function to calculate the similarity between the data points and the support vectors; then, according to the center and radius of the smallest hypersphere, the data points outside the sphere in the high-dimensional space are screened out as abnormal data. The kernel function is used to capture the nonlinear characteristics of the data, achieving high-precision anomaly detection, while effectively reducing false positives and false negatives, and improving the accuracy and reliability of network data asset security identification.

[0102] In a possible implementation, it further includes:

[0103] The support vector description model is updated at intervals of a second preset time length.

[0104] It can be understood that by updating the support vector description model at intervals of the second preset time, it is possible to dynamically adapt to changes in network data assets, ensure the real-time performance and detection accuracy of the model, and improve the robustness and adaptability of the system.

[0105] It should be noted that those skilled in the art can set the second preset duration according to actual needs, and the present invention is not limited thereto.

[0106] In a possible implementation, it further includes:

[0107] Issue early warnings in the presence of abnormal real-time network data.

[0108] It should be noted that when abnormal real-time network data is detected, the system will issue a warning notification in a timely manner to facilitate rapid response and processing, thereby improving the initiative and practicality of security monitoring.

[0109] In actual application, the network data asset security discovery system based on distributed machine learning obtains historical network data through data center nodes and combines Fourier transform and K-means clustering to determine the first preset duration, and uses the support vector description model (SVDD) to establish a minimum hypersphere to describe the normal asset status. After distributing the model to each data processing node, it combines Chebyshev polynomials to efficiently collect real-time network data and dynamically optimizes distribution according to node load. Finally, each node uses the SVDD model to detect anomalies in real-time data, quickly identify and output abnormal data, and has the characteristics of low latency, high accuracy and strong adaptability.

[0110] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0111] In the present invention, the data center node is used as the data allocation center, and the data processing tasks are issued according to the load status of each data processing node, which effectively avoids the problem of too long task queue and too high processing delay, ensures load balancing, improves data processing efficiency and real-time performance, and makes the security discovery of network data assets more accurate and efficient, which is suitable for large-scale high-concurrency scenarios. In the process of security identification of network data assets, combined with the distribution center of historical network data, a support vector description model based on historical network data is established to determine the minimum hypersphere used to describe the normal operation status of network data assets, which can not only adapt to the dynamically changing network environment, but also more accurately distinguish normal and abnormal data, greatly improving the accuracy of abnormal detection. In the process of real-time data collection, the real-time network data of each network data asset is estimated and approximated by combining Chebyshev polynomials, which can effectively deal with the problems of data acquisition difficulties and excessive delay caused by data missing and high concurrent data flow, and can quickly obtain network data in high concurrency and data missing environments, significantly reduce delays and computational complexity, ensure the real-time and reliability of data collection, and improve the efficiency and practicality of network data asset security discovery. Finally, the established support vector description model is used at each data processing node to safely and quickly identify the collected data, with low latency and high accuracy, and can cope with large-scale data streams that conform to actual conditions. Throughout the entire process, from data collection to data identification, the delay of asset identification can be effectively reduced, and abnormal data status can be effectively, accurately and timely detected based on historical data rules, with low false alarm rate and higher practicality.

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A network data asset security discovery system based on distributed machine learning, characterized in that: The network data asset security discovery system includes a data center node and a plurality of data processing nodes each connected to the data center node; The network data asset security discovery system also includes: An acquisition module, configured to acquire historical network data through the data center node at a first preset time period, wherein the historical network data includes asset interaction times of various network data assets and asset interaction frequencies at the asset interaction times; Establishing a module for establishing a support vector description model based on the historical network data in combination with the distribution center of the historical network data to determine a minimum hypersphere for describing the normal operation status of the network data asset; A first sending module, used for sending the support vector description model to each data processing node; The first acquisition module is used to approximate the real-time network data of each network data asset by combining Chebyshev polynomials; The second acquisition module is used to collect load data of each data processing node; A second sending module is used to send the real-time network data to each data processing node based on the load data and with the goal of minimizing network data processing delay; The identification module is used to safely identify the received real-time network data through the support vector description model in each data processing node, and output abnormal real-time network data, that is, output the real-time network data outside the minimum hypersphere.

2. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: The network data assets include servers, routers, switches, personal terminal devices, virtual machines, containers, IP addresses, MAC addresses, domain names, network interfaces and protocol ports.

3. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: The establishment module is specifically used for: Based on the asset interaction time and the asset interaction frequency at the asset interaction time, a historical network data state vector of each network data asset is established: Among them, t k represents the kth asset interaction moment of the network data asset, Indicates that the i-th network data asset is at t k The frequency of asset interaction, x i,t represents the interaction state vector of the ith network data asset at time t, T1 and T2 represent the start and end time of the preset duration respectively; Establish the smallest hypersphere that describes the normal operation status of network data assets: ||x i -c|| 2 ≤R 2 +ξ i ,x i ≥0 e≤ε Where R and c represent the radius and center of the hypersphere, respectively, i represents the slack variable for the ith network data asset, i = 1, 2, ..., N, N represents the total number of network data assets, C represents the penalty coefficient for controlling the degree of hypersphere relaxation, x i represents the interaction state vector of the ith network data asset at any time, f represents the distribution center of historical network data, i.e., the data centroid of all historical network data, e represents the distance from the center of the hypersphere c to the data centroid f, i.e., the eccentricity, λ represents the penalty coefficient when the eccentricity e exceeds the dynamic eccentricity ε, L(R,c,ε,λ) represents the Lagrangian function about R,c,ε,λ, and || || represents the calculation of the Euclidean distance; Using the Gaussian kernel function, the historical network data state vectors of each network data asset are mapped in a pairwise high-dimensional manner: Among them, x j represents the interaction state vector of the jth network data asset at any time, K(x i ,x j ) represents the description x i and x j Gaussian kernel function value of similarity in high-dimensional space, exp represents the natural exponential function, σ i and σ j Respectively represent the description x i and x j kernel width of local distribution characteristics; Solve the interaction state vector weight value through the Lagrangian optimization algorithm: Among them, α i represents the interaction state vector weight value of the i-th network data asset, α j Represents the weight value of the interaction state vector of the jth network data asset; Calculate the radius and center of the hypersphere based on the weight value of the interaction state vector obtained: R 2 =||φ(x s )-c|| 2 Among them, x s Represents the interaction state vector of any network data asset, || || 2 Represents calculation of the square of the Euclidean norm.

4. The network data asset security discovery system based on distributed machine learning according to claim 3 is characterized in that: The calculation formula of the core width is specifically: in, and Respectively represent x i and x j The mth nearest neighbor point of , where M represents the preset neighborhood value.

5. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: The first acquisition module is specifically used for: Count the total number of network data assets and record the asset interaction time of each network data asset; The number of asset interactions of each network data asset is counted through Chebyshev polynomials: Among them, N(A i ) represents network data asset A i The total number of interactions, S represents the interaction subset containing network data assets, |S| represents the interaction subset containing network data assets A i The number of interaction subsets, k represents the upper limit of interaction subsets related to m, and m represents the total number of network data assets. Represents other network data assets A in S j The total number of interactions, represents the Chebyshev polynomial coefficients used to adjust the influence weight of the interaction subset S, and ! represents the factorial operation; The asset interaction times are used as the asset interaction frequency, and the real-time network data is obtained by combining the asset interaction times.

6. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: The load data includes CPU usage, memory occupancy, network bandwidth utilization and the length of the queue of pending tasks.

7. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: The second sending module is specifically used for: Determine a data delivery amount constraint function with the goal of minimizing the network data processing delay: Among them, Minimize means taking the minimum value, R q , B q and L q They represent the processing rate, network bandwidth and network delay of data processing node q respectively, q=1,2,…,Q, Q represents the total number of data processing nodes, D q Indicates the amount of network data sent to node q, Represents for any data processing node q; Determine the node priority of each data processing node: Among them, P q Indicates the node priority of the data processing node q; In the case where the data allocation amount constraint function is satisfied, the real-time network data is sent to each data processing node according to the node priority.

8. The network data asset security discovery system based on distributed machine learning according to claim 3 is characterized in that: The identification module is specifically used for: Establish a screening formula for abnormal real-time network data: ||φ(x)-c|| 2 >R 2 Where x represents the input real-time network data, φ(x) represents the high-dimensional mapping value of x, K(x,x) represents the inner product of the high-dimensional mapping value of x, and K(x,x) represents the inner product of the high-dimensional mapping value of x. i ) represents x and x i The kernel function value in high-dimensional space is similarity; The real-time network data that meets the screening formula is output as abnormal real-time network data.

9. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: Also includes: The support vector description model is updated at intervals of a second preset time length.

10. The network data asset security discovery system based on distributed machine learning according to claim 1 is characterized in that: Also includes: In the presence of the abnormal real-time network data, an early warning is issued.

Images