Security and protection data intelligent analysis method based on big data
By automatically marking abnormal data and performing feature recognition using big data analytics methods, the problem of relying on high-quality labeled data in existing security data analysis is solved, thereby reducing costs and improving analysis accuracy.
Patent Information
- Application Number
- CN202510346805.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-11-07
AI Technical Summary
Existing security data analysis methods rely heavily on high-quality labeled data, resulting in high acquisition costs and requiring significant manpower, material resources, and time.
We adopt a big data-based intelligent analysis method for security data. Through data collection, distributed storage, clustering algorithms, and deep learning algorithms, we automatically mark abnormal data and perform feature recognition, reducing our reliance on labeled data.
It reduces the cost of data preparation, improves the comprehensiveness and accuracy of security analysis, reduces reliance on high-quality labeled data, and improves data storage efficiency and the timeliness of security analysis.
Smart Images
Figure CN120910550A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of security management, and particularly relates to a security data intelligent analysis method based on big data. BACKGROUND
[0002] Big data, also known as huge data, refers to the data volume involved being so large that it cannot be captured, managed, processed, and arranged into information to help enterprise decision-making in a reasonable time through mainstream software tools.
[0003] In the field of security today, the existing security data analysis method mainly relies on labeled data for training. However, it is not easy to obtain high-quality labeled data, which usually requires a lot of human, material and time resources. The process of obtaining labeled data often involves professional personnel reviewing and labeling a large amount of security data one by one, which not only requires high professional knowledge and skills, but also requires a lot of effort. It becomes more difficult and expensive to continuously obtain high-quality labeled data, and the dependence on labeled data increases the cost. SUMMARY
[0004] To solve the problem of the dependence on labeled data increasing the cost in the background art, the present application provides a security data intelligent analysis method based on big data.
[0005] The security data intelligent analysis method based on big data provided by the present application comprises the following steps:
[0006] S1, data acquisition: the security equipment collects security data, integrates the collected security data, and preliminarily preprocesses the collected security data;
[0007] S2, data storage: the preprocessed security data is stored by using a distributed storage system, and different types of data are stored in a classified manner;
[0008] S3, security analysis: the security data that has occurred security problems is marked as labeled data, and the rest of the security data is marked as unlabeled data;
[0009] For unlabeled data, abnormal data is identified by a clustering algorithm, and the abnormal data is marked as labeled data;
[0010] For labeled data, a deep learning algorithm is used to extract the features of security data when security problems occur, then the features of unlabeled data are identified according to the features of security data when security problems occur, and whether to issue a warning or an alarm is determined according to the identification result;
[0011] S4, warning notification: when the identification result determines that a warning is needed, a warning signal is sent to notify the relevant personnel.
[0012] Preferably, the security device in S1 includes a monitoring camera, a door lock, an alarm sensor, the data type includes image, video, personnel access record, alarm signal, and the preprocessing includes denoising and format conversion.
[0013] Preferably, the distributed storage system used in S2 is HDFS or a cloud storage system.
[0014] Preferably, in S3, the abnormal data is identified by a clustering algorithm as follows:
[0015] The Euclidean distance between all data points is calculated, and the Euclidean distance between two data points is:
[0016]
[0017] In the formula, X and Y represent two data points, X = (x1, x2,..., xn), Y = (y1, y2,..., yn), xi (i = 1, 2,..., n) and yi (i = 1, 2,..., n) represent the values of the two data points in the i-th dimension, i = 1 represents the start of the accumulation calculation from the 1st dimension, the square of the difference between the corresponding values of the two data points in the first dimension is calculated first, and then the same operation is performed on each subsequent dimension and accumulated, and n represents the total number of dimensions of the data. n n i i
[0018] A threshold T is set, T = μ + kσ, μ is the mean of the distance, σ is the standard deviation, and k is an adjustment parameter, and 2 ≤ k ≤ 3.
[0019] After calculating the distance between all data points, the distribution of the distance is calculated, for each data point, the distance between it and other data points is calculated, and if more than 80% of the distances in the set of distances between a data point and other data points are greater than the threshold T, the data point is marked as abnormal data.
[0020] Preferably, in S3, the input labeled data is Z and the unlabeled data is V. For the labeled data, a deep convolutional neural network algorithm is used to extract the feature mapping H(Z) of the security data when a security problem occurs through the stacking of multiple convolutional layers and pooling layers.
[0021] For unlabeled data V, abnormal data is identified by a clustering algorithm, and the abnormal data is marked as labeled data. Then, the newly marked labeled data and the original labeled data are trained together using a deep convolutional neural network algorithm. Then, the latest V is input into the trained network to obtain its feature representation G(V).
[0022] The similarity between H(Z) and G(V) is calculated by using cosine similarity:
[0023]
[0024] In the formula, G(V)·H(Z) represents vector dot product, and ||G(V)|| and ||H(Z)|| represent the modulus of vectors G(V) and H(Z) respectively;
[0025] When -1≤similarity<0.5, no pre-warning is performed;
[0026] When 0.5≤similarity<0.8, pre-warning is performed;
[0027] When 0.8≤similarity≤1, alarm is performed.
[0028] Preferably, in S2, lossless compression algorithm is used to compress static data, and lossy compression algorithm is used to compress image and video data.
[0029] Preferably, in S2, multi-thread programming technology is used to process different data tasks simultaneously on a single computing node.
[0030] A big data-based intelligent analysis system for security data, comprising:
[0031] A data acquisition module is configured to acquire security data, integrate collected images, videos, personnel access records, and alarm signal data, and perform preliminary preprocessing, including denoising and format conversion.
[0032] A data storage module is configured to receive and store security data, receive security data using multi-thread programming technology, process different data tasks simultaneously on a single computing node, store preprocessed security data using a distributed storage system, compress static data using lossless compression algorithm, and compress image and video data using lossy compression algorithm.
[0033] A security analysis module is configured to mark security data that has occurred security problems as labeled data, mark the rest of the data as unlabeled data, identify abnormal data through clustering algorithm for the unlabeled data and mark it as labeled data, train labeled data using deep convolutional neural network algorithm, extract feature mapping of security data when security problems occur, input unlabeled data into the trained network, obtain the feature of the unlabeled data, compare the feature of the unlabeled data with the feature mapping of the security data when the security problems occur, and determine whether to perform pre-warning, pre-warning or alarm according to the comparison result.
[0034] Early warning notification module: when the identification result is determined to need early warning or alarm, an early warning signal is sent to inform the relevant personnel.
[0035] A terminal comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the big data-based security data intelligent analysis method described above.
[0036] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the big data-based security data intelligent analysis method described above.
[0037] In the present application, the big data-based security data intelligent analysis method has the following beneficial technical effects:
[0038] 1. The traditional security data analysis method highly depends on labeled data, and obtaining high-quality labeled data is costly, requiring a large amount of manpower, material resources and time. However, the method of the present application greatly reduces the dependence on pre-acquisition of a large amount of high-quality labeled data by marking security data with existing security problems as labeled data, and using a clustering algorithm to identify abnormal data and mark it as labeled data, thereby reducing the cost investment in the data preparation stage and thus reducing the cost.
[0039] 2. The distributed storage system can store large-scale security data. Different types of data are stored in a classified manner, facilitating subsequent query and analysis. At the same time, lossless compression algorithms are used for static data, and lossy compression algorithms are used for image and video data, reducing data storage and transmission time, and improving storage efficiency and data access speed.
[0040] 3. For unlabeled data, the Euclidean distance between all data points is calculated, and a reasonable threshold is set, which can effectively identify abnormal data and mark it as labeled data. This clustering algorithm can quickly discover potential security problems and improve the comprehensiveness of security analysis.
[0041] 4. For labeled data, a deep convolutional neural network algorithm is used to extract features of security data when a security problem occurs, which can accurately capture the key features of the security problem. The cosine similarity is used to calculate the feature similarity between labeled data and unlabeled data, and different similarity intervals are used for early warning or alarm judgment, which improves the accuracy and timeliness of security analysis.
[0042] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS
[0043] Figure 1A flowchart of the method of the present application;
[0044] Figure 2 A principle block diagram of the system of the present application. DETAILED DESCRIPTION
[0045] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar symbols represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary, only for explaining the present application, and cannot be understood as limiting the present application.
[0046] As Figure 1 shown in a security data intelligent analysis method based on big data, a security data intelligent analysis method based on big data, comprising the following steps:
[0047] S1, data acquisition: security equipment collects security data, integrates the collected security data, and preliminarily processes the collected security data;
[0048] The security equipment includes monitoring camera, door lock, alarm sensor, the data type includes image, video, personnel access record, alarm signal, and the preprocessing includes denoising and format conversion, which provides high-quality data basis for subsequent analysis.
[0049] S2, data storage: for the received security data, using multi-thread programming technology, using multiple threads on a single computing node to process different data tasks at the same time, storing the preprocessed security data using distributed storage system, and classifying and storing different types of data; Facilitate subsequent query and analysis;
[0050] The distributed storage system used is HDFS or cloud storage system, which can store large-scale security data, HDFS refers to Hadoop Distributed File System, which is a distributed file system, which aims to reliably store large data sets on inexpensive hardware clusters.
[0051] Using lossless compression algorithms ZIP and RAR to compress static data, using lossy compression algorithms JPEG and H.264 to compress image and video data, reducing data storage and transmission time;
[0052] S3, security analysis:
[0053] Mark the security data that has occurred security problems as labeled data, and mark the rest of the security data as unlabeled data;
[0054] For unlabeled data, identify abnormal data through clustering algorithm, and label the abnormal data as labeled data;
[0055] Identify abnormal data through clustering algorithm as follows:
[0056] Calculate the Euclidean distance between all data points, the Euclidean distance between two data points is:
[0057]
[0058] In the formula, X and Y represent two data points, X=(x1, x2,...,xn), Y=(y1, y2,...,yn), X(i=i, 2,...,n), Y(i=i, 2,...,n), i=1 represents the start of the first dimension for accumulation calculation, first calculate the square of the difference between the corresponding values of the two data points in the first dimension, then perform the same operation on each subsequent dimension and accumulate them, n represents the total number of dimensions of the data; n n i i
[0059] Set a threshold T, T=μ+kσ, μ is the mean of the distance, σ is the standard deviation, k is an adjustment parameter, and 2≤k≤3;
[0060] After calculating the distance between all data points, the distribution of the distance is calculated, for each data point, calculate its distance from other data points, in the set of all distances of a data point from other data points, if more than 80% of the distances in the set are greater than the threshold T, then the data point is marked as abnormal data;
[0061] Through the above clustering algorithm, abnormal data in unlabeled data can be effectively identified and labeled as labeled data.
[0062] For unlabeled data, by calculating the Euclidean distance between all data points and setting a reasonable threshold, abnormal data can be effectively identified and labeled as labeled data. This clustering algorithm can quickly discover potential security problems and improve the comprehensiveness of security analysis.
[0063] For labeled data, use deep learning algorithm to extract features of security data when security problems occur, then identify features of unlabeled data according to the features of security data when security problems occur, and determine whether to issue a warning or alarm according to the identification result;
[0064] Let the input labeled data be Z, and the unlabeled data be V. For labeled data, use deep convolutional neural network algorithm to extract feature mapping H(Z) of security data when security problems occur through stacking of multiple convolutional layers and pooling layers;
[0065] For the unlabeled data V, the abnormal data is identified by a clustering algorithm, and the abnormal data is marked as labeled data, then the newly marked labeled data and the original labeled data are trained together by a deep convolutional neural network algorithm, then the latest V is input into the trained network to obtain its feature representation G(V);
[0066] The cosine similarity is used to calculate the similarity between H(Z) and G(V):
[0067]
[0068] In the formula, G(V)·H(Z) represents the vector dot product, and ||G(V)|| and ||H(Z)|| represent the modulus of vectors G(V) and H(Z), respectively;
[0069] When -1≤similarity<0.5, no warning is given;
[0070] When 0.5≤similarity<0.8, a warning is given;
[0071] When 0.8≤similarity≤1, an alarm is given;
[0072] For the labeled data, the features of the security data when the security problem occurs are extracted by a deep convolutional neural network algorithm, which can accurately capture the key features of the security problem. The feature similarity between the labeled data and the unlabeled data is calculated by the cosine similarity, and the warning or alarm judgment is made according to different similarity intervals, which improves the accuracy and timeliness of the security analysis.
[0073] The input of V into the network that has been trained refers to training on labeled data through a deep convolutional neural network algorithm. The training process is to continuously adjust the weights and biases of the network and other parameters, so that the network can learn the feature patterns of security data when a security problem occurs in the labeled data. When the training is completed, the network "remembers" these feature patterns and can accurately extract features and judge security problems for new input labeled data. Therefore, for unlabeled data V, it is input into the network that has been trained, and the knowledge and feature extraction ability learned by the network in the training stage are used to extract and analyze the features of the unlabeled data. Because the network has been trained on labeled data, it has a certain pattern recognition ability and can try to extract features similar or related to labeled data from unlabeled data. Then, according to the comparison result of the extracted features and the features of the labeled data, it is judged whether the unlabeled data may have a security problem, and then it is decided whether to issue a warning. In short, the network trained on labeled data is used to analyze the features and security of unlabeled data to achieve comprehensive intelligent analysis of the entire security data.
[0074] S4, warning notification: when the identification result determines that a warning is needed, a warning signal is sent to notify relevant personnel, and the warning signal is notified to relevant personnel through the ways of short message, email and mobile phone APP, so that timely measures can be taken.
[0075] The traditional security data analysis method highly depends on labeled data, and obtaining high-quality labeled data is costly, requiring a large amount of manpower, material resources and time. The method of the present application greatly reduces the dependence on a large amount of high-quality labeled data obtained in advance, reduces the cost investment in the data preparation stage, and thus reduces the cost.
[0076] Specifically, in the intrusion detection scenario of network security, there are a small amount of malicious intrusion behavior data that have been labeled, such as network traffic features of hacker attacks. There are also a large amount of unlabeled network traffic data. These data are used for training. Using a deep learning algorithm can learn the features of malicious intrusion behavior, and when there is a similarity to the labeled malicious features and the similarity is high, it can be judged as a potential intrusion behavior. For example, some new malicious software may have abnormal packet sending frequency and size when communicating on the network. The method can help to discover these abnormalities, and through the analysis of unlabeled network traffic data, the potential network security risks can be automatically discovered through a clustering algorithm.
[0077] Specifically, in the video monitoring scene, the past theft behavior data is labeled, and the deep learning algorithm can learn the characteristics of the theft behavior, such as the specific personnel action track and the abnormal object movement. A large amount of unlabeled data also has great value. Through the analysis of a large amount of daily video data, the method can also gradually identify new suspicious behaviors different from the normal behavior mode. That is, for the unlabeled data, a deep convolutional neural network algorithm is used to extract the features of the normal behavior mode through the stacking of multiple convolutional layers and pooling layers, so that the new suspicious behaviors different from the normal behavior mode can be analyzed and identified.
[0078] According to the above information, the beneficial effects and working principles of the present application are written.
[0079] As Figure 2 shown, a security data intelligent analysis system based on big data includes:
[0080] The data acquisition module is used for security data acquisition, integrates the collected images, videos, personnel access records, and alarm signal data, and performs preliminary preprocessing, including denoising and format conversion.
[0081] The data storage module is used for receiving and storing security data. When receiving security data, a multi-thread programming technology is used, multiple threads are used to process different data tasks on a single computing node, a distributed storage system is used to store the preprocessed security data, a lossless compression algorithm is used to compress static data, and a lossy compression algorithm is used to compress image and video data.
[0082] The security analysis module is used for marking the security data of the security problem that has occurred as labeled data, marking the remaining data as unlabeled data, identifying abnormal data through a clustering algorithm for the unlabeled data and marking it as labeled data, training the labeled data using a deep convolutional neural network algorithm, extracting the feature mapping of the security data when the security problem occurs, then inputting the unlabeled data into the trained network to obtain the features of the unlabeled data, comparing the features with the feature mapping of the security data when the security problem occurs, and determining whether to issue a warning or an alarm according to the comparison result.
[0083] The warning notification module issues a warning signal to notify relevant personnel when the identification result determines that a warning or an alarm is needed.
[0084] A terminal includes a processor and a storage medium. The storage medium is used to store instructions, and the processor is used to operate according to the instructions to perform the steps of the security data intelligent analysis method based on big data according to the above.
[0085] A computer readable storage medium, having stored thereon a computer program, which when executed by a processor implements the steps of the big data-based security data intelligent analysis method.
[0086] Meanwhile, the contents not described in detail in the specification are all the prior art known to those skilled in the art.
[0087] In the embodiments provided by the present application, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the application described above are merely schematic, for example, the division of modules is only a logical function division, and there can be another division manner in actual implementation.
[0088] The modules illustrated as separate components may or may not be physically separate, and the components illustrated as modules may or may not be physical modules, and can be located in one place, or distributed on multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.
[0089] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically independently, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware, or in the form of hardware plus software functional module.
[0090] For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the essential characteristics of the present application.
[0091] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can make equivalent replacements or changes to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.
Claims
1. A big data-based security data intelligent analysis method, characterized in that, The method comprises the following steps: S1, data collection: the security equipment collects security data, integrates the collected security data, and performs preliminary preprocessing on the collected security data; S2, data storage: the preprocessed security data is stored by using a distributed storage system, and different types of data are stored in a classified manner; S3, security analysis: the security data of the security problem that has occurred is marked as labeled data, and the rest of the security data is marked as unlabeled data; For unlabeled data, abnormal data is identified by a clustering algorithm, and the abnormal data is marked as labeled data; For labeled data, a deep learning algorithm is used to extract the features of the security data when the security problem occurs, then the unlabeled data is identified according to the features of the security data when the security problem occurs, and whether to perform early warning or alarm is determined according to the identification result; S4, early warning notification: when the identification result determines that early warning is needed, an early warning signal is sent to notify relevant personnel. 2.The big data-based security data intelligent analysis method according to claim 1, characterized in that, In S1, the security equipment includes a monitoring camera, a door lock, an access control, an alarm sensor, the data types include images, videos, personnel access records, and alarm signals, and the preprocessing includes denoising and format conversion. 3.The big data-based security data intelligent analysis method according to claim 1, characterized in that, In S2, the distributed storage system used is HDFS or a cloud storage system. 4.The big data-based security data intelligent analysis method according to claim 1, characterized in that, In S3, the abnormal data is identified by a clustering algorithm as follows: Calculate the Euclidean distance between all data points, and the Euclidean distance between two data points is: wherein X and Y represent two data points, X=(x1, x2,..., xn), Y=(y1, y2,..., yn), and n represents the total dimension number of the data. n ), Y=(y1, y2,..., yn), and n represents the total dimension number of the data. n ), Y=(y1, y2,..., yn), and n represents the total dimension number of the data. i ), Y=(y1, y2,..., yn), and n represents the total dimension number of the data. i ), Y=(y1, y2,..., yn), and n represents the total dimension number of the data. i=1 represents that the accumulation calculation is started from the first dimension, the square of the difference between the corresponding values of the two data points in the first dimension is calculated first, and then the same operation is performed on each subsequent dimension and accumulated, and n represents the total dimension number of the data. Set a threshold T, T = μ + kσ, μ is the mean of the distance, σ is the standard deviation, and k is an adjustment parameter, and 2 ≤ k ≤ 3; After calculating the distance between all data points, the distribution of the distance is counted, for each data point, calculate the distance between it and other data points, if more than 80% of the distances in the set of distances between a data point and other data points are greater than the threshold T, the data point is marked as abnormal data. 5.The big data-based security data intelligent analysis method according to claim 1, characterized in that, In S3, the input labeled data is Z, and the unlabeled data is V. For labeled data, a deep convolutional neural network algorithm is used to extract the feature mapping H(Z) of the security data when the security problem occurs through the stacking of multiple convolutional layers and pooling layers. For unlabeled data V, abnormal data is identified by a clustering algorithm, and the abnormal data is marked as labeled data, then the newly marked labeled data and the original labeled data are trained together by using the deep convolutional neural network algorithm, then the latest V is input into the trained network to obtain the feature representation G(V); The similarity between H(Z) and G(V) is calculated by using cosine similarity: In the formula, G(V)·H(Z) represents the vector dot product, and ||G(V)|| and ||H(Z)|| represent the modulus of vectors G(V) and H(Z) respectively; When -1 ≤ similarity < 0.5, no early warning is performed; When 0.5 ≤ similarity < 0.8, early warning is performed; When 0.8 ≤ similarity ≤ 1, alarm is performed. 6.The big data-based security data intelligent analysis method according to claim 1, characterized in that, In S2, lossless compression algorithm is used to compress static data, and lossy compression algorithm is used to compress image and video data. 7.The big data-based security data intelligent analysis method according to claim 1, characterized in that, In S2, for the received security data, multi-thread programming technology is adopted, and multiple threads are used to process different data tasks simultaneously on a single computing node.
8. The big data-based security data intelligent analysis system of claims 1-7, wherein, The method comprises the following steps: a data acquisition module is configured to acquire security data, integrate and perform preliminary preprocessing on the acquired image, video, personnel access record and alarm signal data, including denoising and format conversion; a data storage module is configured to receive and store the security data, adopt multi-thread programming technology to process different data tasks simultaneously on a single computing node, and use a distributed storage system to store the preprocessed security data, wherein a lossless compression algorithm is used to compress static data, and a lossy compression algorithm is used to compress image and video data; a security analysis module is configured to mark security data that has occurred security problems as labeled data, and mark the rest of the data as unlabeled data, identify abnormal data through a clustering algorithm, and mark the abnormal data as labeled data, train the labeled data using a deep convolutional neural network algorithm to extract feature mapping of the security data when the security problems occur, input the unlabeled data into the trained network to obtain the feature of the unlabeled data, compare the feature of the unlabeled data with the feature mapping of the security data when the security problems occur, and determine whether to issue a warning or an alarm according to the comparison result; a warning notification module is configured to issue a warning signal to notify relevant personnel when the identification result determines that a warning or an alarm is needed. 9.A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is configured to store instructions; The processor is configured to operate according to the instructions to perform the steps of the big data-based security data intelligent analysis method according to any one of claims 1-7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the big data-based security data intelligent analysis method according to any one of claims 1-7.