Security and protection monitoring information storage system and method based on big data redundancy analysis

Abstract

The invention relates to the technical field of big data, in particular to a security and protection monitoring information storage system and method based on big data redundancy analysis, and the system comprises a video collection module which is used for collecting and transmitting security and protection monitoring video data in real time; the data transmission module is used for transmitting the collected video data to a storage server through a wired or wireless network; the data redundancy analysis module is used for performing redundancy analysis on the received video data and removing repeated and useless data; the storage server is used for storing the processed security and protection monitoring video data and providing an efficient read-write function; according to the invention, by combining the fluctuation analysis algorithm and the feature extraction algorithm based on machine learning, redundancy detection and data cleaning of video data are realized, data redundancy is effectively reduced, and the storage efficiency is improved; the calculation model of the fluctuation characteristic value and the machine learning characteristic value ensures accurate detection and removal of repeated data.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a security monitoring information storage system and method based on big data redundancy analysis. Background Art

[0002] With the development of modern society, security monitoring systems have been widely used in many fields such as public security, corporate security, and home security. Traditional security monitoring systems mainly rely on continuous video monitoring and data storage, but with the expansion of monitoring scope and the popularization of high-definition monitoring equipment, the amount of video data generated has increased dramatically, bringing huge pressure to the storage system. In addition, traditional storage methods have problems such as serious data redundancy, low retrieval efficiency, and high storage costs, which cannot meet the needs of current intelligent security systems.

[0003] The development of big data technology has provided a new solution for data storage and management of security monitoring systems. By performing redundancy analysis, data cleaning, feature extraction and anomaly detection on video data, storage space can be effectively reduced and data processing and retrieval efficiency can be improved. However, in the existing technology, how to efficiently process and store massive video data in real time still faces many technical difficulties.

[0004] The existing security monitoring information storage system usually adopts a centralized storage architecture, which is prone to single point failure and cannot fully utilize the advantages of distributed storage. In addition, during the data transmission and storage process, the security of the data is not fully guaranteed, which is easy to cause data leakage and tampering. Therefore, in order to solve the above problems, a security monitoring information storage system and method based on big data redundancy analysis are proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a security monitoring information storage system and method based on big data redundancy analysis to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] Security monitoring information storage system based on big data redundancy analysis, including:

[0008] Video acquisition module, used to collect and transmit security monitoring video data in real time;

[0009] A data transmission module, used to transmit the collected video data to a storage server via a wired or wireless network;

[0010] A data redundancy analysis module is used to perform redundancy analysis on the received video data and remove duplicate and useless data;

[0011] Storage server, used to store processed security surveillance video data and provide efficient reading and writing functions;

[0012] A data retrieval module, used to efficiently retrieve video data in the storage server according to user requests;

[0013] The security module is used to ensure the security of transmitted and stored data and protect the integrity and privacy of data through encryption technology.

[0014] Preferably, the data redundancy analysis module includes:

[0015] A data preprocessing unit, used for performing format conversion, frame processing and basic preprocessing on the received video data;

[0016] Duplicate data detection unit, used to detect duplicate content in video data, and achieve efficient detection by calculating hash values ​​and content features;

[0017] The data cleaning unit is used to remove detected duplicate data and useless data to ensure the uniqueness and validity of stored data.

[0018] Preferably, the duplicate data detection unit performs duplicate data detection by combining a hash algorithm and a content feature extraction algorithm, and the specific calculation formula is as follows:

[0019]

[0020] Among them, H(V) represents the video data, the weighted hash value of V, and V i represents the i-th video frame, f represents the hash function, w i Represents the weight coefficient of the i-th frame.

[0021] Preferably, the content feature extraction algorithm adopts a feature extraction method based on deep learning, and the calculation formula is as follows:

[0022] Among them, C(V) represents the comprehensive feature value of video data V, F i represents the feature vector of the i-th frame, g represents the feature extraction function, s i represents the feature weight of the i-th frame.

[0023] Preferably, the storage server adopts a distributed storage architecture, including multiple interconnected storage nodes, and data transmission and synchronization are achieved between the nodes through a high-speed network, specifically including the following functions:

[0024] Redundant storage management: redundant storage of data is achieved through a distributed file system to improve the reliability of data storage;

[0025] Load balancing: optimizes data reading and writing performance through load balancing algorithms to prevent single-point bottlenecks;

[0026] Data sharding and merging: Data sharding technology is used to divide large-scale video data into several small fragments for storage, and merge and reconstruct data fragments when necessary. The specific calculation model is as follows:

[0027]

[0028] Among them, S(V) represents the set of slices of video data V, P(V i ) represents the i-th video segment.

[0029] Preferably, it also includes:

[0030] The anomaly detection module is used to monitor the system operation status and video data quality. It detects abnormal situations and issues alarms by performing real-time analysis on video data. The specific detection algorithm includes an anomaly detection model based on machine learning. The calculation formula is as follows:

[0031]

[0032] Among them, A(V) represents the abnormal value of the video data V, F i represents the feature vector of the i-th frame, h represents the anomaly detection function, and θ represents the model parameters.

[0033] The security monitoring information storage method based on big data redundancy analysis includes the following steps:

[0034] Step 1: Collect security surveillance video data, and use the video acquisition module to collect video data in the surveillance area in real time;

[0035] Step 2: The collected video data is transmitted to the storage server through a data transmission module using a wired or wireless network;

[0036] Step 3: Perform redundancy analysis on the received video data, and perform format conversion, frame processing, duplicate data detection and data cleaning on the video data through the data redundancy analysis module;

[0037] Step 4: Store the processed security surveillance video data and store the cleaned video data in a distributed storage system;

[0038] Step 5: Retrieve video data from the storage server according to user requests, and achieve efficient data retrieval and retrieval through the data retrieval module.

[0039] Preferably, performing redundancy analysis in step 3 includes:

[0040] Perform format conversion and frame preprocessing on video data to ensure the uniformity of subsequent processing;

[0041] By combining the hash algorithm and the deep learning-based feature extraction algorithm to detect duplicate content in video data, the specific calculation formula is as follows:

[0042]

[0043] Where W(V) represents the fluctuation characteristic value of the video data V, Indicates the rate of change of the i-th frame video in the time dimension, α i represents the fluctuation weight coefficient of the i-th frame; M(V) represents the machine learning feature value of the video data V, F i represents the feature vector of the i-th frame, k represents the feature extraction function, φ represents the model parameter, β i represents the feature weight of the i-th frame.

[0044] Preferably, the processed video data is stored in step 4 using a distributed storage system, which includes multiple storage nodes, and the nodes are interconnected through a high-speed network to achieve redundant storage and load balancing of data, so as to improve the reliability of data storage and reading speed. Specific implementation methods include:

[0045] Data fragmentation and reorganization technology stores large-scale video data in fragments to improve storage and reading efficiency, and reorganizes data fragments when necessary;

[0046] Distributed file system architecture, using distributed file system to achieve reliable storage and efficient access to data;

[0047] Data synchronization and consistency algorithm, through the data synchronization and consistency algorithm between multiple nodes, ensure the high availability and security of data. The specific algorithm is as follows:

[0048]

[0049] Among them, S(V) represents the synchronization characteristic value of the video data V, D i represents the i-th data segment, t i represents the transmission time, and γi represents the synchronization weight of the i-th data segment.

[0050] Preferably, it also includes:

[0051] When an abnormal situation is detected, real-time alarm information is sent to the user, and detailed abnormal situation logs and related video clips are provided. The alarm module realizes real-time monitoring and alarm functions of abnormal situations. Anomaly detection uses an algorithm based on time series analysis and pattern recognition. Through real-time analysis of video data, the abnormal value E(V) is calculated. The specific calculation formula is as follows:

[0052]

[0053] Where E(V) represents the outlier value of the video data V, r represents the outlier detection function based on time series analysis, μ and σ represent the mean and standard deviation of the time series respectively, and δ i represents the anomaly detection weight of the i-th frame.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. In the present invention, by combining the fluctuation analysis algorithm and the feature extraction algorithm based on machine learning, redundant detection and data cleaning of video data are realized, which effectively reduces data redundancy and improves storage efficiency. The calculation model of fluctuation eigenvalues ​​and machine learning eigenvalues ​​ensures accurate detection and removal of duplicate data;

[0056] 2. In the present invention, a distributed storage system is adopted, including multiple interconnected storage nodes, and data transmission and synchronization are realized between nodes through a high-speed network. The application of data fragmentation and reorganization technology, distributed file system architecture, and data synchronization and consistency algorithm improves the reliability of data storage and reading speed, prevents single point bottlenecks, and improves the scalability of the system;

[0057] 3. In the present invention, through the abnormal detection module, an algorithm based on time series analysis and pattern recognition is adopted to analyze the video data in real time, calculate the abnormal value, realize the real-time monitoring and alarm function of abnormal situations, and enhance the intelligence and response speed of the security system;

[0058] 4. In the present invention, through the data retrieval module, the video data in the storage server is efficiently retrieved according to the user's request, and the speed and accuracy of data retrieval are improved by combining feature extraction and indexing technology to meet the user's needs for quickly obtaining key information. DETAILED DESCRIPTION

[0059] Embodiment 1: The present invention provides a technical solution:

[0060] Security monitoring information storage system based on big data redundancy analysis, including:

[0061] Video acquisition module, used to collect and transmit security monitoring video data in real time;

[0062] A data transmission module, used to transmit the collected video data to a storage server via a wired or wireless network;

[0063] A data redundancy analysis module is used to perform redundancy analysis on the received video data and remove duplicate and useless data;

[0064] Storage server, used to store processed security surveillance video data and provide efficient reading and writing functions;

[0065] A data retrieval module, used to efficiently retrieve video data in the storage server according to user requests;

[0066] The security module is used to ensure the security of transmitted and stored data and protect the integrity and privacy of data through encryption technology.

[0067] The security monitoring information storage system based on big data redundancy analysis of the present invention can realize the efficient collection, transmission, analysis, storage and retrieval of security monitoring video data through the comprehensive application of video acquisition module, data transmission module, data redundancy analysis module, storage server, data retrieval module and security module, and ensure the efficient management and safe storage of data. Through data redundancy analysis, the amount of stored data is effectively reduced, the storage efficiency and performance of the system are improved, and the security and privacy of the data are guaranteed.

[0068] The data redundancy analysis module comprises:

[0069] A data preprocessing unit, used for performing format conversion, frame processing and basic preprocessing on the received video data;

[0070] Duplicate data detection unit, used to detect duplicate content in video data, and achieve efficient detection by calculating hash values ​​and content features;

[0071] The data cleaning unit is used to remove detected duplicate data and useless data to ensure the uniqueness and validity of stored data.

[0072] The present invention realizes format conversion, frame processing, duplicate content detection and data cleaning of video data by adding a data preprocessing unit, a duplicate data detection unit and a data cleaning unit. Through the combination of these units, the system can more accurately detect and remove duplicate and useless data, further improve the effectiveness of data storage and the processing efficiency of the system, and ensure the uniqueness and high quality of stored data.

[0073] The duplicate data detection unit performs duplicate data detection by combining a hash algorithm and a content feature extraction algorithm. The specific calculation formula is as follows:

[0074]

[0075] Among them, H(V) represents the video data, the weighted hash value of V, and V i represents the i-th video frame, f represents the hash function, w i Represents the weight coefficient of the i-th frame.

[0076] The present invention improves the accuracy and efficiency of duplicate data detection by introducing a duplicate data detection method combining a hash algorithm and a content feature extraction algorithm and utilizing a calculation model of a weighted hash value H(V). The method can quickly identify and remove duplicate content in video data, significantly reduce storage space usage, and improve the overall performance and reliability of the system.

[0077] 4. The security monitoring information storage system based on big data redundancy analysis according to claim 1 is characterized in that: the content feature extraction algorithm adopts a feature extraction method based on deep learning, and the calculation formula is as follows:

[0078] Among them, C(V) represents the comprehensive feature value of video data V, F i represents the feature vector of the i-th frame, g represents the feature extraction function, s i represents the feature weight of the i-th frame.

[0079] The present invention uses a feature extraction method based on deep learning and a calculation model of the comprehensive eigenvalue C(V) to achieve accurate extraction and analysis of video data content features. The method can better identify the features of video data, improve the accuracy and efficiency of data redundancy analysis, and effectively reduce stored data redundancy.

[0080] The storage server adopts a distributed storage architecture, including multiple interconnected storage nodes, and the nodes realize data transmission and synchronization through a high-speed network, specifically including the following functions:

[0081] Redundant storage management: redundant storage of data is achieved through a distributed file system to improve the reliability of data storage;

[0082] Load balancing: optimizes data reading and writing performance through load balancing algorithms to prevent single-point bottlenecks;

[0083] Data sharding and merging: Data sharding technology is used to divide large-scale video data into several small fragments for storage, and merge and reconstruct data fragments when necessary. The specific calculation model is as follows:

[0084]

[0085] Among them, S(V) represents the set of slices of video data V, P(V i ) represents the i-th video segment.

[0086] The present invention realizes efficient data storage and reading by adopting a distributed storage architecture, utilizing redundant storage management, load balancing, and data sharding and merging technologies. Through a distributed file system and high-speed network connection, the reliability and scalability of the system are improved, single point failures are prevented, and high availability and security of data are ensured.

[0087] 6. The security monitoring information storage system and method based on big data redundancy analysis according to claim 1, characterized in that it also includes:

[0088] The anomaly detection module is used to monitor the system operation status and video data quality. It detects abnormal situations and issues alarms by performing real-time analysis on video data. The specific detection algorithm includes an anomaly detection model based on machine learning. The calculation formula is as follows:

[0089]

[0090] Among them, A(V) represents the abnormal value of the video data V, F i represents the feature vector of the i-th frame, h represents the anomaly detection function, and θ represents the model parameters.

[0091] The present invention realizes real-time monitoring and anomaly detection of system operation status and video data quality by introducing an anomaly detection module, adopting an anomaly detection model based on machine learning, and using the calculation model of the anomaly value A(V). This function can timely detect and alarm abnormal situations, improve the security and intelligence level of the system, and enhance the user's response ability.

[0092] 7. A security monitoring information storage method based on big data redundancy analysis, characterized in that it includes the following steps:

[0093] Step 1: Collect security surveillance video data, and use the video acquisition module to collect video data in the surveillance area in real time;

[0094] Step 2: The collected video data is transmitted to the storage server through a data transmission module using a wired or wireless network;

[0095] Step 3: Perform redundancy analysis on the received video data, and perform format conversion, frame processing, duplicate data detection and data cleaning on the video data through the data redundancy analysis module;

[0096] Step 4: Store the processed security surveillance video data and store the cleaned video data in a distributed storage system;

[0097] Step 5: Retrieve video data from the storage server according to user requests, and achieve efficient data retrieval and retrieval through the data retrieval module.

[0098] The security monitoring information storage method based on big data redundancy analysis of the present invention realizes comprehensive management and optimization of security monitoring video data through the steps of collection, transmission, redundancy analysis, storage and retrieval. The method can efficiently process and store massive video data, ensure efficient management and safe storage of data, and improve the overall performance and reliability of the system.

[0099] The redundancy analysis in step 3 includes:

[0100] Perform format conversion and frame preprocessing on video data to ensure the uniformity of subsequent processing;

[0101] By combining the hash algorithm and the deep learning-based feature extraction algorithm to detect duplicate content in video data, the specific calculation formula is as follows:

[0102]

[0103] Where W(V) represents the fluctuation characteristic value of the video data V, Indicates the rate of change of the i-th frame video in the time dimension, α i represents the fluctuation weight coefficient of the i-th frame; M(V) represents the machine learning feature value of the video data V, F i represents the feature vector of the i-th frame, k represents the feature extraction function, φ represents the model parameter, β i represents the feature weight of the i-th frame.

[0104] The present invention combines a fluctuation analysis algorithm with a feature extraction algorithm based on machine learning, and uses a calculation model of fluctuation eigenvalues ​​W(V) and machine learning eigenvalues ​​M(V) to achieve redundancy detection and data cleaning of video data. The method can more accurately identify and remove duplicate content, improving the efficiency and accuracy of data storage.

[0105] The processed video data in step 4 is stored in a distributed storage system, which includes multiple storage nodes. The nodes are interconnected through a high-speed network to achieve redundant storage and load balancing of data, so as to improve the reliability of data storage and reading speed. The specific implementation methods include:

[0106] Data fragmentation and reorganization technology stores large-scale video data in fragments to improve storage and reading efficiency, and reorganizes data fragments when necessary;

[0107] Distributed file system architecture, using distributed file system to achieve reliable storage and efficient access to data;

[0108] Data synchronization and consistency algorithm, through the data synchronization and consistency algorithm between multiple nodes, ensure the high availability and security of data. The specific algorithm is as follows:

[0109]

[0110] Among them, S(V) represents the synchronization characteristic value of the video data V, D i represents the i-th data segment, t i represents the transmission time, and γi represents the synchronization weight of the i-th data segment.

[0111] The present invention realizes efficient data storage and reading by adopting a distributed storage system, combining data fragmentation and reorganization technology, distributed file system architecture and data synchronization and consistency algorithm S(V). The system has high reliability and scalability, can run stably under high load and fault conditions, and ensures high availability and security of data.

[0112] Also includes:

[0113] When an abnormal situation is detected, real-time alarm information is sent to the user, and detailed abnormal situation logs and related video clips are provided. The alarm module realizes real-time monitoring and alarm functions of abnormal situations. Anomaly detection uses an algorithm based on time series analysis and pattern recognition. Through real-time analysis of video data, the abnormal value E(V) is calculated. The specific calculation formula is as follows:

[0114]

[0115] Where E(V) represents the outlier value of the video data V, r represents the outlier detection function based on time series analysis, μ and σ represent the mean and standard deviation of the time series respectively, and δ i represents the anomaly detection weight of the i-th frame.

[0116] The present invention realizes real-time monitoring and alarming of abnormal situations by introducing abnormal detection and alarm functions, adopting algorithms based on time series analysis and pattern recognition, and using the calculation model of abnormal value E(V). This function enhances the intelligence and response speed of the system, improves the safety of the security system and the response ability of users.

[0117] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. The above is only a preferred implementation of the present invention. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make several improvements, modifications or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the protection scope of the present invention.

Claims

1. A security monitoring information storage system based on big data redundancy analysis, characterized by: include: Video acquisition module, used to collect and transmit security monitoring video data in real time; A data transmission module, used to transmit the collected video data to a storage server via a wired or wireless network; A data redundancy analysis module is used to perform redundancy analysis on the received video data and remove duplicate and useless data; Storage server, used to store processed security surveillance video data and provide efficient reading and writing functions; A data retrieval module, used to efficiently retrieve video data in the storage server according to user requests; The security module is used to ensure the security of transmitted and stored data and protect the integrity and privacy of data through encryption technology.

2. The security monitoring information storage system based on big data redundancy analysis according to claim 1 is characterized in that: The data redundancy analysis module comprises: A data preprocessing unit, used for performing format conversion, frame processing and basic preprocessing on the received video data; Duplicate data detection unit, used to detect duplicate content in video data, and achieve efficient detection by calculating hash values ​​and content features; The data cleaning unit is used to remove detected duplicate data and useless data to ensure the uniqueness and validity of stored data.

3. The security monitoring information storage system based on big data redundancy analysis according to claim 2 is characterized in that: The duplicate data detection unit performs duplicate data detection by combining a hash algorithm and a content feature extraction algorithm. The specific calculation formula is as follows: Among them, H(V) represents the video data, the weighted hash value of V, and V i represents the i-th video frame, f represents the hash function, w i Represents the weight coefficient of the i-th frame.

4. The security monitoring information storage system based on big data redundancy analysis according to claim 3 is characterized in that: The content feature extraction algorithm adopts a feature extraction method based on deep learning, and the calculation formula is as follows: Among them, C(V) represents the comprehensive feature value of video data V, F i represents the feature vector of the i-th frame, g represents the feature extraction function, s i represents the feature weight of the i-th frame.

5. The security monitoring information storage system based on big data redundancy analysis according to claim 1 is characterized in that: The storage server adopts a distributed storage architecture, including multiple interconnected storage nodes, and the nodes realize data transmission and synchronization through a high-speed network, specifically including the following functions: Redundant storage management: redundant storage of data is achieved through a distributed file system to improve the reliability of data storage; Load balancing: optimizes data reading and writing performance through load balancing algorithms to prevent single-point bottlenecks; Data sharding and merging: Data sharding technology is used to divide large-scale video data into several small fragments for storage, and merge and reconstruct data fragments when necessary. The specific calculation model is as follows: Among them, S(V) represents the segment set of video data V, and P(Vi) represents the i-th video segment.

6. The security monitoring information storage system based on big data redundancy analysis according to claim 1 is characterized in that: Also includes: The anomaly detection module is used to monitor the system operation status and video data quality. It detects abnormal situations and issues alarms by performing real-time analysis on video data. The specific detection algorithm includes an anomaly detection model based on machine learning. The calculation formula is as follows: Among them, A(V) represents the abnormal value of the video data V, F i represents the feature vector of the i-th frame, h represents the anomaly detection function, and θ represents the model parameters.

7. A security monitoring information storage method based on big data redundancy analysis, characterized in that: The following steps are involved: Step 1: Collect security surveillance video data, and use the video acquisition module to collect video data in the surveillance area in real time; Step 2: The collected video data is transmitted to the storage server through a data transmission module using a wired or wireless network; Step 3: Perform redundancy analysis on the received video data, and perform format conversion, frame processing, duplicate data detection and data cleaning on the video data through the data redundancy analysis module; Step 4: Store the processed security surveillance video data and store the cleaned video data in a distributed storage system; Step 5: Retrieve video data from the storage server according to user requests, and achieve efficient data retrieval and retrieval through the data retrieval module.

8. The security monitoring information storage method based on big data redundancy analysis according to claim 7 is characterized in that: The redundancy analysis in step 3 includes: Perform format conversion and frame preprocessing on video data to ensure the uniformity of subsequent processing; By combining the hash algorithm and the deep learning-based feature extraction algorithm to detect duplicate content in video data, the specific calculation formula is as follows: Where W(V) represents the fluctuation characteristic value of the video data V, Indicates the rate of change of the i-th frame video in the time dimension, α i represents the fluctuation weight coefficient of the i-th frame; M(V) represents the machine learning feature value of the video data V, F i represents the feature vector of the i-th frame, k represents the feature extraction function, φ represents the model parameter, β i represents the feature weight of the i-th frame.

9. The security monitoring information storage method based on big data redundancy analysis according to claim 7 is characterized in that: In step 4, the processed video data is stored in a distributed storage system, which includes multiple storage nodes. The nodes are interconnected through a high-speed network to achieve redundant storage and load balancing of data, so as to improve the reliability of data storage and reading speed. The specific implementation methods include: Data fragmentation and reorganization technology stores large-scale video data in fragments to improve storage and reading efficiency, and reorganizes data fragments when necessary; Distributed file system architecture, using distributed file system to achieve reliable storage and efficient access to data; Data synchronization and consistency algorithm, through the data synchronization and consistency algorithm between multiple nodes, ensure the high availability and security of data. The specific algorithm is as follows: Among them, S(V) represents the synchronization characteristic value of the video data V, D i represents the i-th data segment, t i represents the transmission time, and γi represents the synchronization weight of the i-th data segment.

10. The security monitoring information storage method based on big data redundancy analysis according to claim 7 is characterized in that: Also includes: When an abnormal situation is detected, real-time alarm information is sent to the user, and detailed abnormal situation logs and related video clips are provided. The alarm module realizes real-time monitoring and alarm functions of abnormal situations. Anomaly detection uses an algorithm based on time series analysis and pattern recognition. Through real-time analysis of video data, the abnormal value E(V) is calculated. The specific calculation formula is as follows: Where E(V) represents the outlier value of the video data V, r represents the outlier detection function based on time series analysis, μ and σ represent the mean and standard deviation of the time series respectively, and δ i represents the anomaly detection weight of the i-th frame.

Images