A video surveillance data storage and management system based on big data

Abstract

The present invention discloses a video surveillance data storage and management system based on big data, specifically relating to the field of data analysis, including multi-source heterogeneous data acquisition, cross-layer feature fusion calculation, comprehensive indicator fusion, collaborative decision-making generation, and adaptive control execution. The present invention constructs a three-dimensional tensor structure to eliminate dimensional differences through multi-level data synchronous acquisition and cross-domain feature fusion, realizes global state perception of encoding, transmission, and storage, and solves the local optimization limitations caused by traditional data silos. The dynamic weight adjustment model automatically matches the optimal strategy based on the third-order collaborative management coefficient calculated in real time, realizes the joint elastic regulation of encoding parameters and network protocols and dynamic reconstruction of storage resources, and overcomes the response lag defect of the static threshold mechanism under burst traffic.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and more specifically, to a video surveillance data storage management system based on big data. Background Art

[0002] With the rapid development of ultra-high-definition video, low-latency live streaming, smart security and other businesses, the system needs to meet the stringent requirements of high compression rate, low transmission jitter and high storage reliability. Current industry solutions generally adopt an independent optimization mode, and each module exchanges data through a simple interface, lacking a global perspective resource coordination mechanism.

[0003] Mainstream technical architectures usually configure the encoder in fixed bit rate control mode. The network layer relies on traditional congestion control algorithms for bandwidth adaptation, and the storage layer performs data sharding and hot and cold stratification based on preset rules. This serial processing mode causes the encoding efficiency to be disconnected from the network status. The transmission strategy cannot perceive the topological distribution of storage nodes, and the operation and maintenance system can only trigger single-point policy adjustments through threshold alarms.

[0004] The existing solution has three significant flaws: the fixed encoding parameters cause quality fluctuations under burst traffic; the mismatch between the transport layer retransmission mechanism and the storage node load status causes data scheduling conflicts; the offline decision model cannot respond to dynamic changes in the system in real time; the isolated optimization of each module forms a local optimal trap, making it difficult to achieve optimal allocation of cross-domain resources, which seriously restricts the overall energy efficiency improvement of large-scale video systems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides a video surveillance data storage and management system based on big data, which solves the problems proposed in the above-mentioned background technology, such as data silos in the encoding, transmission, and storage modules leading to response lag, the inability of static strategies to adapt to dynamic environmental changes, and the lack of a closed-loop feedback mechanism leading to suboptimal decision-making, through the following scheme.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: a video surveillance data storage and management system based on big data, comprising:

[0007] Multi-source heterogeneous data acquisition module: This module achieves synchronous capture of raw data by deploying acquisition tools at the video encoding layer, network transmission layer, and distributed storage layer in parallel. The video encoding layer uses video stream analysis tools to extract frame structure and motion parameters. The network transmission layer uses programmable network devices to capture protocol characteristics and channel status. The distributed storage layer obtains physical media status and access logs based on the storage system interface.

[0008] Cross-layer feature fusion calculation module: This module extracts streaming features from the raw data provided by the acquisition layer and calculates three types of indicators: video coding efficiency monitoring group, video transmission feature analysis group, and storage feature analysis group.

[0009] Comprehensive Index Fusion Module: This module performs nonlinear aggregation on the feature matrix output by the feature calculation layer and uses three sets of parameter-calibrated formulas to generate a comprehensive index of video coding efficiency, an aggregated measurement of transmission performance, and an index of storage system entropy change.

[0010] Collaborative decision generation module: Based on the three-dimensional state vector of the indicator analysis layer, the global management coefficient is calculated through the third-order collaborative management decision model, and hierarchical management instructions are generated by combining the preset threshold conditions and state space division rules;

[0011] Adaptive control execution module: This module converts the policy instructions generated by the decision-making layer into three core operations: dynamic optimization of video encoding parameters, real-time switching of network transmission protocols, and reconstruction of storage resource distribution strategies. It drives the execution of codec parameter updates, hot loading of network protocol stack configurations, and storage node data migration tasks through a distributed control system.

[0012] Preferably, the collection layer uses the storage system monitoring interface to collect physical storage status information. All data is stored in the time series database after standardization to provide basic data input for subsequent feature calculations, and its output is directly related to the data processing module of the feature calculation layer.

[0013] Preferably, the multi-source heterogeneous data acquisition module uses the probe function of FFmpeg to parse the video stream metadata, extracts the type and size of each frame through the ffprobe -show_frames command, cooperates with the --analysis-save parameter of the x265 encoder to output macroblock-level entropy values ​​and motion vector residual data, adopts the VMAF toolkit to batch encode and record PSNR values ​​at a preset bit rate ladder, obtains the key frame density by counting the I frame interval time in the GOP structure, and parses the resolution information directly from the video container header metadata. Finally, a Python script is used to aggregate the I frame byte sequence, P frame size sequence, macroblock entropy matrix, motion residual timing data, multi-bitrate PSNR data set and resolution parameters, and store them as a time series database.

[0014] Preferably, the multi-source heterogeneous data acquisition module deploys a P4 programmable switch to mirror traffic, uses TShark to capture packets and sets -d udp.port==1234, and uses an rtp filter to parse the RTP / RTCP protocol. The packet loss rate and retransmission times of the time-space slices are counted through the rtp_analysis module. The network jitter coefficient is calculated by sampling the / proc / net / tcpprobe interface of the Linux tcpprobe module. The protocol header overhead is obtained by disassembling the TCP / IPv6 / RTP packet header length field. The JSON output mode of iperf3 is used to record channel bandwidth fluctuations. The video bit rate change characteristics are extracted from the SEI information output by the encoder. The FEC redundancy is obtained by parsing the ULP FEC parameters of the RTP extension header. The layered coding depth is extracted from the DONC unit of the SVC encoded stream. The RTT data is measured through the ACK reception interval of the QUIC protocol. All data are injected into the InfluxDB time series library in real time through the Kafka pipeline.

[0015] Preferably, the multi-source heterogeneous data acquisition module enables bluestore monitoring in the Ceph cluster, exports the object access heat map and physical location mapping table through the ceph-dencoder tool, obtains the SSD erase count from the Percentage_Used field of the SMART log, uses the sensor-util command of IPMI to collect storage temperature in real time, obtains the compressed block size distribution by the --trace log analysis of the zstd compressor, reads the storage page size through the blockdev --getbsz command, uses Ceph's rados list-inconsistent-pg detection result combined with the MinHash algorithm to calculate the cross-node data similarity, and collects the data block temperature entropy value through the access log sliding window. Finally, all the original indicators are aggregated by Prometheusexporter and stored in the OpenTSDB cluster.

[0016] Preferably, the calculation results of the feature calculation layer form a structured feature matrix which is transmitted to the indicator analysis layer, forming the data basis for subsequent comprehensive evaluation, and its processing logic directly depends on the data completeness of the acquisition layer.

[0017] Preferably, the video coding efficiency monitoring group includes the spatiotemporal compression ratio, entropy coding fluctuation coefficient, motion prediction residual gradient and bit rate-quality inflection point slope; the video transmission feature analysis group includes the spatiotemporal degree correlation, protocol stack efficiency index, channel feature matching degree and fault tolerance diffusion coefficient; the storage feature analysis group includes the hot and cold data vortex degree, coding storage coupling ratio, medium loss equalization factor and topology-aware compression ratio.

[0018] Preferably, the spatiotemporal compression ratio is calculated by using a sliding window to calculate the P frame mean, and the key frame distribution is analyzed in combination with the GoP structure, which is specifically expressed as: , STR represents the time-space compression ratio, Indicates the byte size of a single I frame. Indicates the average byte size of P frames in the last 10 seconds. represents the density of key frames per unit time; the entropy coding fluctuation coefficient is calculated macroblock by macroblock based on the Shannon entropy formula, and the quantile method is used to filter outliers, which is specifically expressed as: , EVF represents the entropy coding fluctuation coefficient, Indicates the maximum macroblock entropy value in a single frame, Indicates the minimum macroblock entropy value in a single frame, Represents the mean entropy value of the macroblock within the frame; the motion prediction residual gradient is calculated by the motion compensation residual matrix, and the SIMD instruction is used to accelerate the summation, which is specifically expressed as: , MRG represents the motion prediction residual gradient, represents the residual variation of the i-th macroblock, A represents the frame area, and N represents the total number of macroblocks. The slope of the bit rate-quality inflection point is constructed by a cubic spline interpolation function and the resolution term is normalized. Specifically, it is expressed as follows: , RQS represents the rate-quality inflection point slope, It represents the derivative of the PSNR-bitrate curve, W and h represent the width and height of the video resolution.

[0019] Preferably, the spatiotemporal metric correlation is based on a Kalman filter to evaluate the network jitter component, specifically expressed as: , STC represents the spatiotemporal correlation, Indicates the spatial fragmentation packet loss rate, Indicates the number of time slice retransmissions, J indicates the network jitter coefficient, Represents a zero-proof constant; the protocol stack efficiency index uses a protocol parser to extract the header field and calculate the MTU matching degree, which is specifically expressed as: , PSE represents the protocol stack efficiency index, Indicates the effective load ratio, Indicates TCP header overhead, Indicates the RTP header overhead, represents the fragment aggregation factor; the channel feature matching degree uses a sliding window to calculate the Shannon entropy, which is specifically expressed as: , CCM represents the channel characteristic matching degree, represents the channel bandwidth distribution entropy, Represents the entropy of video bit rate change; the error-tolerant diffusion coefficient is expressed as follows: , FDC represents the fault diffusion coefficient, represents the forward error correction redundancy, represents the layered coding depth, and RTT represents the network round-trip delay.

[0020] Preferably, the vorticity of the hot and cold data is constructed to access the frequency field calculation gradient, and is positioned using spherical coordinates, which is specifically expressed as: , HDV represents the vorticity of hot and cold data, represents the access frequency gradient, Indicates the discreteness of storage locations, Represents the temperature entropy of the data block; the encoding storage coupling rate uses the memory mapping file to detect the boundary alignment, which is specifically expressed as: , SCR represents the coding storage coupling ratio, Indicates the number of times the block boundary is aligned. Indicates the compressed block size, Indicates the storage page size, Represents the total number of data blocks; the medium wear leveling factor establishes a wear leveling state matrix, combined with the temperature sensor data, specifically expressed as: , WEF represents dielectric loss equalization factor, Indicates the maximum / minimum number of erase and write times. Indicates the average number of erase and write cycles. represents the temperature difference coefficient; the topology-aware compression rate uses locality-sensitive hashing to detect cross-node similarity, which is specifically expressed as: , TCR represents the topology-aware compression ratio, Indicates the standard compression ratio, represents the data similarity between nodes k and j, and n represents the total number of nodes.

[0021] Preferably, the cross-layer feature fusion calculation module implements cross-layer joint feature matrix construction by defining a three-dimensional tensor structure: with the time axis as the first dimension, a fixed-length sliding window is divided; in the feature dimension, STR, EVF, MRG, RQS are mapped to tensor channels 1-4, STC, PSE, CCM, FDC are mapped to channels 5-8, HDV, SCR, WEF, TCR are mapped to channels 9-12; a double buffering mechanism is used to achieve timing alignment, and the splicing operation is triggered when and only when the window timestamp overlap of the three data streams exceeds 95%. The SIMD instruction set is used to optimize the memory layout, and finally a tensor with a shape of [T×12×N] is generated, where T is the number of time windows and N is the batch size, and is normalized in the channel dimension by LayerNorm to ensure dimensional consistency of cross-layer features.

[0022] Preferably, the three comprehensive indicators generated by the indicator analysis layer constitute a three-dimensional representation vector of the system state, and the calculation results are transmitted to the decision-making layer through the data pipeline.

[0023] Preferably, the video coding efficiency comprehensive index is specifically expressed as: , VECI represents the video coding efficiency comprehensive index, α represents the encoder architecture adaptation constant, and β represents the time domain continuity compensation coefficient.

[0024] Preferably, the transmission efficiency aggregation metric is specifically expressed as: , TEM represents the transmission efficiency aggregation metric, γ represents the network topology complexity factor, and δ represents the critical threshold of fault tolerance efficiency.

[0025] Preferably, the storage system entropy change index is specifically expressed as: , SSEI represents the storage system entropy change index, η represents the storage medium aging rate, θ represents the optimal coupling threshold, and σ represents the process fluctuation standard deviation.

[0026] Preferably, the three-order collaborative management decision model is specifically expressed as: , TCDM represents the global management coefficient, κ represents the coding quality weight, μ represents the transmission stability coefficient, ν represents the storage pressure damping factor, ω represents the mutation tolerance threshold, VECI Δ Indicates the rate of change of coding efficiency in the past 3 minutes, TEM Δ Indicates that the transmission efficiency is close to 5 RTT window gradients.

[0027] Preferably, the collaborative decision generation module divides the system status into three response levels by setting two critical thresholds of 80 and 50 based on the hierarchical management trigger mechanism of the TCDM value: when the TCDM value is higher than 80, the system enables the elastic coding mechanism and the BBRv3 congestion control algorithm for joint optimization, automatically switches to the constant quality factor coding mode while maintaining video quality, and optimizes the network queue management to reduce transmission delay; if the TCDM value is between 50 and 80, the time domain layered coding technology is activated to achieve bit rate adaptation, and high-temperature storage nodes are preheated in advance to balance the access load; when the TCDM value falls below the threshold of 50, a forced I frame request is immediately executed to quickly rebuild the integrity of the video stream, and cross-availability zone data migration is initiated to ensure storage availability.

[0028] Preferably, the system state changes generated by the operation execution of the control execution layer are fed back to the acquisition layer in real time through the monitoring loop to form a closed-loop control.

[0029] Preferably, the control execution layer implements three-dimensional control based on the hierarchical management instruction set generated by the decision layer, including the encoding side, the transmission side and the storage side. The encoding side realizes real-time optimization by dynamically reconstructing the picture group GOP structure, and automatically calculates the GOP length according to VECI using an exponential function model. When the VECI value decreases, the GOP is automatically shortened to enhance fault tolerance. Its mathematical expression is GOP_size=⌈2^(2.3-VECI / 28)⌉, where GOP_size represents the picture group size, ensuring a dynamic balance between coding efficiency and error resilience. The transmission side establishes an error correction level FEC hierarchical control mechanism, and determines the error correction level through a composite function of TEM and RTT. When the TEM / √RTT value exceeds 4.7 When the error rate is lower than 2.9, it automatically upgrades to level 5 FEC to enhance anti-packet loss capabilities. When the error rate is lower than 2.9, it downgrades to basic error correction mode to save bandwidth. The storage side implements a time-space-aware garbage collection (GC) strategy, which uses a sinusoidal periodic function combined with the system running time t to dynamically adjust the GC trigger threshold. The expression is GC_threshold = 0.65 + 0.12 sin(2πt / 86400). GC_threshold represents the garbage collection threshold. This increases the activity of storage organization operations during low-access periods and automatically suppresses the GC frequency during peak periods to ensure I / O performance. All control operations take effect synchronously through the distributed control bus, and execution status data is returned to the collection layer in real time through the tracking system, forming a complete closed loop from decision-making to execution.

[0030] The technical effects and advantages of the present invention are as follows:

[0031] This solution builds a multi-level data fusion mechanism to achieve real-time joint perception of coding features, network status, and storage load. It uses a three-dimensional tensor structure to eliminate cross-domain feature dimensional differences and accurately characterizes the global state of the system through a nonlinear aggregation algorithm. This solves the local optimization limitations of traditional methods caused by data silos and significantly improves decision-making integrity in complex scenarios.

[0032] The present invention introduces a dynamic weight adjustment model to automatically match the optimal management strategy based on the third-order collaborative management coefficient calculated in real time. The coding parameters and network protocols are jointly and elastically regulated, and the storage strategy can be dynamically reconfigured based on the transmission load. This overcomes the response lag of the traditional static threshold mechanism under burst traffic, effectively enhancing the system's adaptability to dynamic environments.

[0033] A fully closed-loop control loop from data collection to strategy execution is established. Decision instructions are converted into operational control signals through a distributed execution module. System state changes are fed back to the feature calculation layer in real time to trigger parameter calibration, forming a continuous optimization mechanism. This breaks through the suboptimal strategy solidification problem of traditional offline decision-making models and achieves continuous convergence optimization of the system's operating status. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0035] Figure 2 It is a schematic structural diagram of a complete embodiment of the present invention.

[0036] Figure 3 It is a schematic diagram of the swimming lane diagram structure of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] refer to Figure 1-3 A video surveillance data storage and management system based on big data is shown, comprising:

[0039] Multi-source heterogeneous data acquisition module: By deploying acquisition tools in parallel at the video coding layer, network transmission layer, and distributed storage layer, the original data can be captured synchronously. The video coding layer uses video stream analysis tools to extract frame structure and motion parameters. The network transmission layer captures protocol characteristics and channel status through programmable network devices. The distributed storage layer obtains physical media status and access logs based on the storage system interface.

[0040] The collection layer uses the storage system monitoring interface to collect physical storage status information. All data is standardized and stored in the time series database to provide basic data input for subsequent feature calculations. Its output is directly related to the data processing module of the feature calculation layer.

[0041] The multi-source heterogeneous data acquisition module uses the probe function of FFmpeg to parse the video stream metadata, extracts the type and size of each frame through the ffprobe -show_frames command, cooperates with the --analysis-save parameter of the x265 encoder to output macroblock-level entropy values ​​and motion vector residual data, adopts the VMAF toolkit to batch encode and record PSNR values ​​at a preset bit rate ladder, obtains the key frame density by counting the I frame interval time in the GOP structure, and parses the resolution information directly from the video container header metadata. Finally, a Python script is used to aggregate the I frame byte sequence, P frame size sequence, macroblock entropy value matrix, motion residual timing data, multi-bitrate PSNR dataset and resolution parameters, and store them as a time series database.

[0042] The multi-source heterogeneous data acquisition module deploys a P4 programmable switch to mirror traffic, uses TShark to capture packets and sets -d udp.port==1234, and uses the rtp filter to parse the RTP / RTCP protocol. The packet loss rate and retransmission times of the spatiotemporal slices are counted through the rtp_analysis module. The network jitter coefficient is sampled and calculated using the / proc / net / tcpprobe interface of the Linux tcpprobe module. The protocol header overhead is obtained by disassembling the TCP / IPv6 / RTP packet header length field. The JSON output mode of iperf3 is used to record channel bandwidth fluctuations. The video bit rate change characteristics are extracted from the SEI information output by the encoder. The FEC redundancy is obtained by parsing the ULP FEC parameters of the RTP extension header. The layered coding depth is extracted from the DONC unit of the SVC encoded stream. The RTT data is measured through the ACK reception interval of the QUIC protocol. All data is injected into the InfluxDB time series library in real time through the Kafka pipeline.

[0043] The multi-source heterogeneous data collection module enables bluestore monitoring in the Ceph cluster, exports object access heat maps and physical location mapping tables through the ceph-dencoder tool, obtains the number of SSD erases and writes from the Percentage_Used field of the SMART log, uses the sensor-util command of IPMI to collect storage temperature in real time, obtains the compressed block size distribution through the --trace log analysis of the zstd compressor, reads the storage page size through the blockdev --getbsz command, calculates cross-node data similarity using Ceph's rados list-inconsistent-pg detection results combined with the MinHash algorithm, and calculates the data block temperature entropy through the access log sliding window. Finally, all raw indicators are aggregated by PrometheusExporter and stored in the OpenTSDB cluster.

[0044] Cross-layer feature fusion calculation module: performs streaming feature extraction on the original data provided by the acquisition layer, and calculates three types of indicators: video coding efficiency monitoring group, video transmission feature analysis group, and storage feature analysis group.

[0045] The calculation results of the feature calculation layer form a structured feature matrix and are transmitted to the indicator analysis layer, forming the data basis for subsequent comprehensive evaluation. Its processing logic directly depends on the data completeness of the acquisition layer.

[0046] The video coding efficiency monitoring group includes the spatiotemporal compression ratio, entropy coding fluctuation coefficient, motion prediction residual gradient and bit rate-quality inflection point slope; the video transmission feature analysis group includes the spatiotemporal correlation, protocol stack efficiency index, channel feature matching degree and fault tolerance diffusion coefficient; the storage feature analysis group includes the hot and cold data vortex degree, coding storage coupling ratio, medium loss equalization factor and topology-aware compression ratio.

[0047] The temporal and spatial compression ratio uses a sliding window to calculate the P frame mean, and combines the GoP structure to analyze the key frame distribution, which is specifically expressed as: , STR represents the time-space compression ratio, Indicates the byte size of a single I frame. Indicates the average byte size of P frames in the last 10 seconds. represents the density of key frames per unit time; the entropy coding fluctuation coefficient is calculated macroblock by macroblock based on the Shannon entropy formula, and the quantile method is used to filter outliers, which is specifically expressed as: , EVF represents the entropy coding fluctuation coefficient, Indicates the maximum macroblock entropy value in a single frame, Indicates the minimum macroblock entropy value in a single frame, Represents the mean entropy value of the macroblock within the frame; the motion prediction residual gradient is calculated by the motion compensation residual matrix, and the SIMD instruction is used to accelerate the summation, which is specifically expressed as: , MRG represents the motion prediction residual gradient, represents the residual variation of the i-th macroblock, A represents the frame area, and N represents the total number of macroblocks. The slope of the bit rate-quality inflection point is constructed by a cubic spline interpolation function and the resolution term is normalized. Specifically, it is expressed as follows: , RQS represents the rate-quality inflection point slope, It represents the derivative of the PSNR-bitrate curve, W and h represent the width and height of the video resolution.

[0048] The spatiotemporal correlation is based on the Kalman filter to evaluate the network jitter component, which is specifically expressed as: , STC represents the spatiotemporal correlation, Indicates the spatial fragmentation packet loss rate, Indicates the number of time slice retransmissions, J indicates the network jitter coefficient, Represents a zero-proof constant; the protocol stack efficiency index uses a protocol parser to extract the header field and calculate the MTU matching degree, which is specifically expressed as: , PSE represents the protocol stack efficiency index, Indicates the effective load ratio, Indicates TCP header overhead, Indicates the RTP header overhead, represents the fragment aggregation factor; the channel feature matching degree uses a sliding window to calculate the Shannon entropy, which is specifically expressed as: , CCM represents the channel characteristic matching degree, represents the channel bandwidth distribution entropy, Represents the entropy of video bit rate change; the error-tolerant diffusion coefficient is expressed as follows: , FDC represents the fault diffusion coefficient, represents the forward error correction redundancy, represents the layered coding depth, and RTT represents the network round-trip delay.

[0049] The vorticity of the hot and cold data is constructed to access the frequency field calculation gradient, using spherical coordinate positioning, specifically expressed as: , HDV represents the vorticity of hot and cold data, represents the access frequency gradient, Indicates the discreteness of storage locations, Represents the temperature entropy of the data block; the encoding storage coupling rate uses the memory mapping file to detect the boundary alignment, which is specifically expressed as: , SCR represents the coding storage coupling ratio, Indicates the number of times the block boundary is aligned. Indicates the compressed block size, Indicates the storage page size, Represents the total number of data blocks; the medium wear leveling factor establishes a wear leveling state matrix, combined with the temperature sensor data, specifically expressed as: , WEF represents dielectric loss equalization factor, Indicates the maximum / minimum number of erase and write times. Indicates the average number of erase and write cycles. represents the temperature difference coefficient; the topology-aware compression rate uses locality-sensitive hashing to detect cross-node similarity, which is specifically expressed as: , TCR represents the topology-aware compression ratio, Indicates the standard compression ratio, represents the data similarity between nodes k and j, and n represents the total number of nodes.

[0050] The cross-layer feature fusion calculation module implements cross-layer joint feature matrix construction by defining a three-dimensional tensor structure: with the time axis as the first dimension, a fixed-length sliding window is divided; in the feature dimension, STR, EVF, MRG, and RQS are mapped to tensor channels 1-4, STC, PSE, CCM, and FDC are mapped to channels 5-8, and HDV, SCR, WEF, and TCR are mapped to channels 9-12; a double buffering mechanism is used to achieve timing alignment, and the splicing operation is triggered when and only when the window timestamp overlap of the three data streams exceeds 95%. The SIMD instruction set is used to optimize the memory layout, and finally a tensor with a shape of [T×12×N] is generated, where T is the number of time windows and N is the batch size. It is normalized in the channel dimension using LayerNorm to ensure the dimensional consistency of cross-layer features.

[0051] Comprehensive index fusion module: It performs nonlinear aggregation on the feature matrix output by the feature calculation layer and uses three sets of formulas with calibrated parameters to generate a comprehensive index of video coding efficiency, a transmission performance aggregation metric, and a storage system entropy change index.

[0052] The three comprehensive indicators generated by the indicator analysis layer constitute a three-dimensional representation vector of the system status, and the calculation results are transmitted to the decision-making layer through the data pipeline.

[0053] The video coding efficiency comprehensive index is specifically expressed as: , VECI represents the video coding efficiency comprehensive index, α represents the encoder architecture adaptation constant, and β represents the time domain continuity compensation coefficient.

[0054] The comprehensive video coding efficiency index constructs a basic efficiency factor based on the product of STR and RQS, reflecting the synergistic effect of compression capability and quality sensitivity. It applies 1.5 powers to RQS to amplify the impact of nonlinear quality degradation. The geometric mean of EVF and MRG is introduced into the denominator to suppress the evaluation deviation caused by high-frequency texture and motion mutation. The logarithmic term is used to compensate for the loss of temporal continuity in scenes with low key frame density. The β coefficient is determined by regression analysis of the H.264 / HEVC encoder on the CMCD standard test sequence set. The α value is derived from the principal component analysis of the performance of different encoders on UGC content, and the normalized value of the first principal component load is taken.

[0055] The transmission efficiency aggregation metric is specifically expressed as: , TEM represents the transmission efficiency aggregation metric, γ represents the network topology complexity factor, and δ represents the critical threshold of fault tolerance efficiency.

[0056] The transmission efficiency aggregation metric is based on the orthogonality of protocol efficiency and channel matching. It constructs a coupling term for PSE and CCM. The 0.7 power of CCM weakens the marginal benefit of perfect matching. The STC is used as the denominator to reflect the scheduling conflict loss. The exponential term introduces the impact factor of the fault tolerance mechanism (1-FDC / δ) to achieve adaptive weight adjustment. The δ value is derived from the FDC boundary condition corresponding to 99.999% reliability in the 5G NR URLLC standard. The arctan function maps the ratio of the number of retransmissions to the packet loss rate to the interval [0, π / 2) to suppress metric distortion under extreme network conditions. The γ coefficient is obtained by fitting simulation data of different cell densities based on the 3GPP 38.901 channel model and is corrected using the shape parameter of the Weibull distribution.

[0057] The storage system entropy change index is specifically expressed as: , SSEI represents the storage system entropy change index, η represents the storage medium aging rate, θ represents the optimal coupling threshold, and σ represents the process fluctuation standard deviation.

[0058] The storage system entropy change indicator uses the 2.2th power of HDV to highlight the impact of access mode mutations, and forms a loss balance constraint with WEF. The logarithmic term reflects the diminishing marginal benefits of compression rate improvement. The error function erf() converts the degree to which the SCR deviates from the ideal threshold θ into a penalty coefficient. The value of θ is taken from the optimal write alignment experimental data of 3D NAND flash memory under the 24nm / 16nm process. σ is determined by the distribution of Page Error rates in SSD endurance testing. The coefficient η integrates multiple failure models such as NAND cell erase and write cycles, data retention time, and temperature-accelerated aging. It is calibrated using the Arrhenius equation for an 85°C accelerated aging experiment.

[0059] Collaborative decision generation module: Based on the three-dimensional state vector of the indicator analysis layer, the global management coefficient is calculated through the third-order collaborative management decision model, and hierarchical management instructions are generated in combination with the preset threshold conditions and state space division rules.

[0060] The decision layer converts the system state quantity into a specific operation strategy, and its output instruction set directly drives the operation module of the control execution layer.

[0061] The three-order collaborative management decision model is specifically expressed as follows: , TCDM represents the global management coefficient, κ represents the coding quality weight, μ represents the transmission stability coefficient, ν represents the storage pressure damping factor, ω represents the mutation tolerance threshold, VECI Δ Indicates the rate of change of coding efficiency in the past 3 minutes, TEM Δ Indicates that the transmission efficiency is close to 5 RTT window gradients.

[0062] The construction of the three-order collaborative management decision model begins with the quantitative modeling of the nonlinear interaction relationship between VECI, TEM, and SSEI. First, the basic decision factor is established. To characterize the balance between the system's positive performance and storage constraints, the exponential terms κ=0.6, μ=1.2, and ν=0.9 are used to calculate the marginal contribution of each indicator to QoE on the video conversation dataset using the Shapley value decomposition method to ensure that the encoding quality weight is lower than the transmission stability requirement; =10^−5 prevents the denominator from approaching zero in low storage pressure scenarios, and the mutation suppression term The construction is based on Lyapunov stability theory, where ω=15 is determined by analyzing the 90th percentile of indicator mutations in large-scale video cloud operation and maintenance data. When the indicator change rate exceeds 15 units / time window, the output of the tanh function approaches 1, causing the overall decision value to decay and suppressing policy oscillations caused by sudden loads.

[0063] The collaborative decision generation module divides the system status into three response levels based on the hierarchical management trigger mechanism of the TCDM value by setting two critical thresholds of 80 and 50: when the TCDM value is higher than 80, the system activates the elastic coding mechanism and the BBRv3 congestion control algorithm for joint optimization, automatically switches to the constant quality factor encoding mode while maintaining video quality, and optimizes the network queue management to reduce transmission delay; if the TCDM value is between 50 and 80, the time domain layered coding technology is activated to achieve bit rate adaptation, and high-temperature storage nodes are preheated in advance to balance the access load; when the TCDM value falls below the threshold of 50, a forced I frame request is immediately executed to quickly rebuild the integrity of the video stream, and cross-availability zone data migration is initiated to ensure storage availability.

[0064] Adaptive control execution module: This module converts the policy instructions generated by the decision-making layer into three core operations: dynamic optimization of video encoding parameters, real-time switching of network transmission protocols, and reconstruction of storage resource distribution strategies. It drives the execution of codec parameter updates, hot loading of network protocol stack configurations, and storage node data migration tasks through a distributed control system.

[0065] The system state changes generated by the operation execution of the control execution layer are fed back to the acquisition layer in real time through the monitoring loop to form a closed-loop control.

[0066] The control execution layer implements three-dimensional control based on the hierarchical management instruction set generated by the decision layer, including the encoding side, the transmission side and the storage side. The encoding side realizes real-time optimization by dynamically reconstructing the picture group GOP structure, and automatically calculates the GOP length according to VECI using an exponential function model. When the VECI value drops, the GOP is automatically shortened to enhance fault tolerance. Its mathematical expression is GOP_size=⌈2^(2.3-VECI / 28)⌉, where GOP_size represents the picture group size, ensuring a dynamic balance between coding efficiency and error recovery capability. The transmission side establishes an error correction level FEC hierarchical control mechanism, and determines the error correction level through a composite function of TEM and RTT. When the TEM / √RTT value exceeds 4.7, the error correction level is automatically shortened. It automatically upgrades to level 5 FEC to enhance packet loss resistance. When the error rate falls below 2.9, it downgrades to basic error correction mode to save bandwidth. A time-space-aware garbage collection (GC) strategy is implemented on the storage side. A sinusoidal periodic function is used in combination with the system runtime t to dynamically adjust the GC trigger threshold. The expression is GC_threshold = 0.65 + 0.12 sin(2πt / 86400). GC_threshold represents the garbage collection threshold. This increases the activity of storage organization operations during low-access periods and automatically suppresses the GC frequency during peak periods to ensure I / O performance. All control operations take effect synchronously through the distributed control bus. Execution status data is fed back to the collection layer in real time through the tracking system, forming a complete closed loop from decision-making to execution.

[0067] Specifically, in this embodiment, the constants in all formulas are determined using a two-stage process of "theoretical modeling + experimental calibration." First, a differential equation or probabilistic model is established to describe the system dynamics (such as the rate-distortion equation, network fluid model, and storage wear equation) to derive the theoretical constraint ranges for the parameters. Then, an orthogonal experimental matrix is ​​designed, and over 2,000 sample points are generated using an actual system (FFmpeg encoder, P4 network simulation platform, and Ceph cluster). A Bayesian optimization algorithm is then used to find the parameter values ​​that maximize the Pearson correlation coefficient between the formula output and the actual QoE score.

[0068] The present invention firstly realizes the synchronous capture of multi-source heterogeneous data by deploying acquisition tools of the video coding layer, network transmission layer and distributed storage layer in parallel, uses FFmpeg to parse video stream metadata, adopts P4 programmable switch mirror traffic to obtain network transmission characteristics, and enables storage monitoring function in Ceph cluster to collect physical medium status. All raw data are stored in the time series database after standardization, and then enters the cross-layer feature fusion stage. Through streaming calculation, 12 indicators of video coding efficiency monitoring group, transmission feature analysis group and storage feature analysis group are extracted, and a three-dimensional tensor structure is constructed to realize time window alignment and feature dimension normalization. In the comprehensive indicator fusion link, a nonlinear aggregation algorithm is used to generate a comprehensive video coding efficiency. Index, transmission efficiency aggregation measurement and storage system entropy change index, ensure the reliability of indicators through parameter calibration, generate hierarchical management instructions based on the collaborative decision-making model of the three-dimensional state vector combined with the threshold conditions, enable the joint optimization of elastic coding and BBRv3 congestion control when the global management coefficient is higher than 80, activate the time domain layered coding and storage preheating strategy in the range of 50 to 80, and force the execution of I frame requests and cross-region data migration when it is lower than 50. Finally, the control execution layer converts the decision instructions into dynamic optimization of coding parameters, FEC hierarchical control and time-space aware garbage collection strategy, and realizes codec parameter update, protocol stack hot loading and storage resource reconstruction through the distributed control system, forming a full-link management process from data collection to closed-loop control.

[0069] Secondly: The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention. Other structures may refer to conventional designs. The same embodiment and different embodiments of the present invention may be combined with each other without conflict.

[0070] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video surveillance data storage and management system based on big data, characterized in that: include: Multi-source heterogeneous data acquisition module: This module achieves synchronous capture of raw data by deploying acquisition tools at the video encoding layer, network transmission layer, and distributed storage layer in parallel. The video encoding layer uses video stream analysis tools to extract frame structure and motion parameters. The network transmission layer uses programmable network devices to capture protocol characteristics and channel status. The distributed storage layer obtains physical media status and access logs based on the storage system interface. Cross-layer feature fusion calculation module: This module extracts streaming features from the raw data provided by the acquisition layer and calculates three types of indicators: video coding efficiency monitoring group, video transmission feature analysis group, and storage feature analysis group. Comprehensive index fusion module: This module performs nonlinear aggregation on the feature matrix output by the feature calculation layer and generates a comprehensive index of video coding efficiency, an aggregated measurement of transmission performance, and an entropy change index of the storage system. Collaborative decision generation module: Based on the three-dimensional state vector of the indicator analysis layer, the global management coefficient is calculated through the third-order collaborative management decision model, and hierarchical management instructions are generated by combining the preset threshold conditions and state space division rules; Adaptive control execution module: This module converts the policy instructions generated by the decision-making layer into three core operations: dynamic optimization of video encoding parameters, real-time switching of network transmission protocols, and reconstruction of storage resource distribution strategies. It drives the execution of codec parameter updates, hot loading of network protocol stack configurations, and storage node data migration tasks through a distributed control system.

2. The video surveillance data storage and management system based on big data according to claim 1, characterized in that: The video coding efficiency monitoring group includes the spatiotemporal compression ratio, entropy coding fluctuation coefficient, motion prediction residual gradient and rate-quality inflection point slope, which are marked as STR, EVF, MRG and RQS respectively; the video transmission feature analysis group includes the spatiotemporal correlation, protocol stack efficiency index, channel feature matching and fault tolerance diffusion coefficient, which are marked as STC, PSE, CCM and FDC respectively; The storage feature analysis group includes the vorticity of hot and cold data, the encoding storage coupling ratio, the dielectric loss equalization factor, and the topology-aware compression ratio, which are marked as HDV, SCR, WEF, and TCR respectively.

3. The video surveillance data storage and management system based on big data according to claim 2, characterized in that: The temporal and spatial compression ratio uses a sliding window to calculate the P frame mean, and combines the GoP structure to analyze the key frame distribution, which is specifically expressed as: , Indicates the byte size of a single I frame. Indicates the average byte size of P frames in the last 10 seconds. Indicates the key frame density per unit time; The entropy coding fluctuation coefficient is calculated macroblock by macroblock based on the Shannon entropy formula, and the quantile method is used to filter outliers, which is specifically expressed as: , Indicates the maximum macroblock entropy value in a single frame, Indicates the minimum macroblock entropy value in a single frame, Represents the mean entropy value of the macroblock within the frame; the motion prediction residual gradient is calculated by the motion compensation residual matrix, and the SIMD instruction is used to accelerate the summation, which is specifically expressed as: , represents the residual variation of the i-th macroblock, A represents the frame area, and N represents the total number of macroblocks. The rate-quality inflection point slope is constructed by a cubic spline interpolation function and the resolution term is normalized, which is specifically expressed as follows: , It represents the derivative of the PSNR-bitrate curve, W and h represent the width and height of the video resolution.

4. The video surveillance data storage and management system based on big data according to claim 3, characterized in that: The spatiotemporal correlation is based on the Kalman filter to evaluate the network jitter component, which is specifically expressed as: , Indicates the spatial fragmentation packet loss rate, Indicates the number of time slice retransmissions, J indicates the network jitter coefficient, Represents a zero-proof constant; the protocol stack efficiency index uses a protocol parser to extract the header field and calculate the MTU matching degree, which is specifically expressed as: , Indicates the effective load ratio, Indicates TCP header overhead, Indicates the RTP header overhead, represents the fragment aggregation factor; the channel feature matching degree uses a sliding window to calculate the Shannon entropy, which is specifically expressed as: , CCM represents the channel characteristic matching degree, represents the channel bandwidth distribution entropy, Represents the entropy of video bit rate change; the error-tolerant diffusion coefficient is expressed as follows: , represents the forward error correction redundancy, represents the layered coding depth, and RTT represents the network round-trip delay.

5. The video surveillance data storage and management system based on big data according to claim 2, characterized in that: The vorticity of the hot and cold data is constructed to access the frequency field calculation gradient, using spherical coordinate positioning, specifically expressed as: , represents the access frequency gradient, Indicates the discreteness of storage locations, Represents the temperature entropy of the data block; the encoding storage coupling rate uses the memory mapping file to detect the boundary alignment, which is specifically expressed as: , Indicates the number of times the block boundary is aligned. Indicates the compressed block size, Indicates the storage page size, Represents the total number of data blocks; the medium wear leveling factor establishes a wear leveling state matrix, combined with the temperature sensor data, specifically expressed as: , Indicates the maximum / minimum number of erase and write times. Indicates the average number of erase and write cycles. represents the temperature difference coefficient; the topology-aware compression rate uses locality-sensitive hashing to detect cross-node similarity, which is specifically expressed as: , Indicates the standard compression ratio, represents the data similarity between nodes k and j, and n represents the total number of nodes.

6. The video surveillance data storage and management system based on big data according to claim 2, characterized in that: The cross-layer feature fusion calculation module implements cross-layer joint feature matrix construction by defining a three-dimensional tensor structure: with the time axis as the first dimension, a fixed-length sliding window is divided; in the feature dimension, the spatiotemporal compression ratio, entropy coding fluctuation coefficient, motion prediction residual gradient, and code rate-quality inflection point slope are mapped to tensor channels 1-4, the spatiotemporal correlation, protocol stack efficiency index, channel feature matching, and fault-tolerant diffusion coefficient are mapped to channels 5-8, and the hot and cold data vortex, encoding storage coupling ratio, dielectric loss equalization factor, and topology-aware compression ratio are mapped to channels 9-12; a double buffering mechanism is used to achieve timing alignment, and a splicing operation is triggered when and only when the window timestamp overlap of the three data streams exceeds 95%. The SIMD instruction set is used to optimize the memory layout, and finally a tensor with a shape of [T×12×N] is generated, where T is the number of time windows and N is the batch size. The tensor is normalized in the channel dimension using LayerNorm to ensure dimensional consistency of cross-layer features.

7. The video surveillance data storage and management system based on big data according to claim 4, characterized in that: The video coding efficiency comprehensive index is specifically expressed as: , VECI represents the video coding efficiency comprehensive index, α represents the encoder architecture adaptation constant, and β represents the temporal continuity compensation coefficient; The transmission efficiency aggregation metric is specifically expressed as: ,TEM represents the transmission efficiency aggregation metric, γ represents the network topology complexity factor, and δ represents the critical threshold of fault tolerance efficiency; The storage system entropy change index is specifically expressed as: , SSEI represents the storage system entropy change index, η represents the storage medium aging rate, θ represents the optimal coupling threshold, and σ represents the process fluctuation standard deviation.

8. The big data-based video surveillance data storage and management system according to claim 7, characterized in that: The three-order collaborative management decision model is specifically expressed as follows: , TCDM represents the global management coefficient, κ represents the coding quality weight, μ represents the transmission stability coefficient, ν represents the storage pressure damping factor, ω represents the mutation tolerance threshold, VECI Δ Indicates the rate of change of coding efficiency in the past 3 minutes, TEM Δ Indicates that the transmission efficiency is close to 5 RTT window gradients.

9. The video surveillance data storage and management system based on big data according to claim 2, characterized in that: The collaborative decision generation module divides the system status into three response levels by setting two critical thresholds of 80 and 50 based on the hierarchical management trigger mechanism of the TCDM value: when the TCDM value is higher than 80, the system activates the elastic coding mechanism and the BBRv3 congestion control algorithm for joint optimization, automatically switches to the constant quality factor encoding mode while maintaining video quality, and optimizes the network queue management to reduce transmission delay; if the TCDM value is between 50 and 80, the time domain layered coding technology is activated to achieve bit rate adaptation, and high-temperature storage nodes are preheated in advance to balance the access load; when the TCDM value is less than the threshold of 50, a forced I frame request is immediately executed to quickly rebuild the integrity of the video stream, and cross-availability zone data migration is initiated to ensure storage availability.

Images