Multi-source instantaneous saturated data compression method under strong time-sensitive target scene

Abstract

The invention relates to a multi-source instantaneous saturated data compression method in a strong time-sensitive target scene, and belongs to the technical field of data compression. According to the invention, efficient compression of data is realized through a multi-level layered architecture design. According to the method, data key features are extracted through time sequence analysis and a deep learning model, a compression strategy is dynamically selected based on the importance and change characteristics of sensor data, and the real-time requirement in a strong time-sensitive target scene is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression, and in particular relates to a multi-source instantaneous saturation data compression method in a strong time-sensitive target scenario. Background Art

[0002] Sensor data in time-sensitive target scenarios will experience instantaneous saturation, and the data value will reach the measurement limit in a short time, making it difficult for traditional data processing methods to effectively extract valuable information. Time-sensitive target scenarios have strict requirements on data characteristics and real-time performance in high-speed dynamic environments, and extremely high requirements on the response speed of the system, especially in the fields of military reconnaissance, unmanned driving, and intelligent transportation. The system must quickly acquire, process, and transmit data in order to make real-time decisions. In addition, time-sensitive target scenarios usually require multi-source data fusion, integrating information from multiple sensors such as radar, optics, and infrared to improve the accuracy and robustness of the system. The time scale, spatial resolution, and noise characteristics of various sensors are different, and data needs to be compressed during the data fusion process to reduce redundancy and improve processing efficiency. Traditional data compression algorithms, such as Huffman coding and discrete cosine transform (DCT), are usually used to process static or low-timeliness data. For dynamic data in time-sensitive target scenarios, they are not enough to meet the real-time requirements. Compression methods based on deep learning learn the characteristics of data and optimize the compression process to achieve higher compression efficiency and lower processing delay while ensuring data integrity. In recent years, they have been widely studied. Scenarios with strong time-sensitive targets need to maximize data compression effects while ensuring real-time performance and reduce the burden of data transmission, which is a key issue that needs to be solved urgently. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] The present invention proposes a multi-source instantaneous saturation data compression method in a highly time-sensitive target scenario, which mainly solves two problems: one is to effectively extract useful information from the instantaneous saturation data of the sensor in a high dynamic scenario and perform effective compression to avoid the loss of important data; the other is to improve the data compression efficiency while ensuring high real-time performance, ensuring that a high compression effect and data quality can be achieved under time constraints.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention provides a multi-source instantaneous saturation data compression method in a strong time-sensitive target scenario, comprising the following steps:

[0007] Step 1: Adaptive data preprocessing: Preliminary processing of multi-source instantaneous saturation data, using noise removal and outlier detection algorithms to remove and recover unreliable data due to instantaneous saturation phenomena;

[0008] Step 2: Time series analysis and feature extraction: Use time series analysis methods and deep learning models to extract spatiotemporal features of the data recovered in step 1, identify key changes and optimize data representation;

[0009] Step 3: Multi-source data fusion and dynamic compression: According to the importance and change characteristics of each sensor data, a weighted average algorithm is used to align and fuse the spatiotemporal feature data of different sensors, and the most suitable compression strategy is dynamically selected for each data type to perform data compression.

[0010] The present invention also provides a system for implementing the method.

[0011] (III) Beneficial effects

[0012] The present invention proposes a multi-source instantaneous saturation data compression method in a strong time-sensitive target scenario, and the main beneficial effects are:

[0013] (1) Effectively process instantaneous saturated data. In highly time-sensitive target scenarios, adaptive preprocessing technology is used to identify effective information in saturated data to ensure data reliability and accuracy.

[0014] (2) Extract the key spatiotemporal features of the data and optimize the data expression to improve data compression efficiency.

[0015] (3) Dynamically adjust the compression strategy: Use multi-source data fusion technology to dynamically adjust the compression strategy based on the importance, accuracy, and change characteristics of each sensor data.

[0016] (4) Enhance the real-time and adaptive capabilities of the system. Scenarios with strong time-sensitive targets require that the system must be able to process and transmit data in a very short time. This solution uses an adaptive algorithm to adjust the compression strategy in real time according to the data change pattern and hardware resource status, ensuring efficient processing under different working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0019] This paper proposes a multi-source instantaneous saturation data compression method for time-sensitive target scenarios, which realizes efficient data compression through multi-level hierarchical architecture design. This method extracts key data features through time series analysis and deep learning models, and dynamically selects compression strategies based on the importance and change characteristics of sensor data to meet the real-time requirements in time-sensitive target scenarios.

[0020] Aiming at the problem of multi-source instantaneous saturated data compression in highly time-sensitive target scenarios, the present invention proposes a multi-level hierarchical compression method, which mainly includes four core steps: adaptive data preprocessing, time series analysis and feature extraction, multi-source data fusion and dynamic compression, and real-time response and performance optimization.

[0021] Step 1: Adaptive Data Preprocessing

[0022] In scenarios with strong time-sensitive targets, instantaneous saturation data is often distorted or lost due to sensor range limitations or environmental factors. The purpose of adaptive data preprocessing is to restore instantaneous saturation data, remove noise, detect and remove outliers, and lay the foundation for subsequent data compression.

[0023] (1) Instantaneous saturation data processing

[0024] Instantaneous saturation data processing mainly includes two aspects: noise removal and outlier detection.

[0025] 1) Noise Removal

[0026] Sensors are subject to various noise interferences, such as thermal noise, electromagnetic interference, etc., which affect data accuracy. The present invention uses a Kalman filter to dynamically evaluate the system state and eliminate noise.

[0027] The Kalman filter processes noisy observation data by estimating the current state. The target state transition model is:

[0028]

[0029] in, is the predicted state, A is the state transfer matrix, B is the control input matrix, and u(t) is the control input.

[0030] Updates are measured by the deviation between the observations and the state estimate:

[0031] K(t)=P(t|t-1)H T (HP(t|t-1)H T +R) -1

[0032]

[0033] Where K(t) is the Kalman gain, P(t|t-1) is the error covariance, z(t) is the observation, R is the measurement noise covariance, and H is the measurement matrix.

[0034] The Kalman filter updates the current state estimate based on the estimate at the previous moment and the current observation value, thereby reducing the impact of noise in the system and outputting a more accurate signal.

[0035] 2) Outlier Detection

[0036] When processing instantaneous saturation data, data acquisition errors, sensor failures or external environmental interference can all cause abnormal values ​​to be generated. The present invention implements abnormal value detection through a Gaussian distribution abnormal value detection method.

[0037] Assuming that the data is approximately Gaussian distributed, we can determine outliers by setting the standard deviation range and calculating the mean and standard deviation of the data. When the deviation value of a data point exceeds the set standard deviation range, it is determined to be an outlier. For example, data point x i The probability density function of is:

[0038]

[0039] where μ is the mean of the data set and σ is the standard deviation. i -μ|>k·σ, where k is a set constant, the data point is considered an outlier and is removed or replaced.

[0040] (2) Saturation data recovery

[0041] Instantaneous saturation is usually caused by the sensor input signal exceeding its measurement range, resulting in the output value reaching or exceeding the maximum range. It is necessary to recover the valid information in these data. For different situations, different methods are used for data recovery:

[0042] 1) Linear interpolation

[0043] Assume that the sensor is at the recovery point time t k The measured data x(t k ) is saturated, and x(t k ) before and after the data x(t k-1 ) and x(t k+1 ) is valid, then the time t is restored by linear interpolation method k The value of this point:

[0044]

[0045] The linear interpolation method can restore values ​​that are saturated in a short period of time and is suitable for scenarios where the signal changes relatively smoothly.

[0046] 2) Polynomial fitting method

[0047] For time series data with more complex changes, the polynomial fitting method is used for data recovery. Assume that the recovery point t k The data needs to be fitted through the front and back neighborhood data points. The fitting process can be expressed as:

[0048]

[0049] a0, a1, and a2 are fitting coefficients.

[0050] 3) Frequency domain recovery method

[0051] For signals in scenes with strong time-sensitive targets, Fourier transform or wavelet transform is used to restore saturated signals. In Fourier transform, the frequency domain characteristics of the signal are analyzed, the missing part is frequency compensated, and then the time domain signal is restored through inverse transform. For example, for the time domain signal x(t), its Fourier transform is:

[0052]

[0053] Step 2: Timing Analysis and Feature Extraction

[0054] Key information that can represent the dynamic changes of the target is mined from the recovered time series data to provide effective input for subsequent compression and processing.

[0055] (1) Timing Analysis

[0056] This includes identifying periodicity, trends, and noise components in the data, capturing the target's movement patterns and changing patterns, and improving the efficiency of the compression algorithm.

[0057] 1) Fourier transform analysis

[0058] Through Fourier transform, the time domain signal is converted into the frequency domain signal. In the frequency domain, the low frequency part represents the long-term trend of the signal, while the high frequency part represents the rapid change of the target. For highly time-sensitive targets, the high frequency components of the signal are usually focused.

[0059] 2) Wavelet transform analysis

[0060] Wavelet transform is used to analyze the localized signal for non-stationary signal analysis. Wavelet transform can achieve a balance between time domain and frequency domain by decomposing the signal at different scales, and can better process rapidly changing signals, effectively extract the instantaneous frequency characteristics of the signal, identify the target's motion pattern, and capture the key features of the target's rapid motion process.

[0061] (2) Deep Learning Feature Extraction

[0062] For multi-source data in highly time-sensitive target scenes, deep learning methods such as LSTM (Long Short-Term Memory Network) and CNN (Convolutional Neural Network) are used to automatically extract high-level spatiotemporal features from the data. For video and image data, CNN is used to extract spatial features; for radar or sensor data, recurrent neural network RNN ​​or long short-term memory network LSTM is used to extract temporal features.

[0063] Step 3: Multi-source data fusion and dynamic compression

[0064] In scenarios with strong time-sensitive targets, different spatiotemporal feature data are obtained from multiple sensors or data sources (such as radar, video, infrared sensors, etc.). It is necessary to effectively fuse these data and dynamically adjust the compression strategy according to the changes in the scene.

[0065] (1) Multi-source data fusion

[0066] Multi-source data fusion effectively integrates multi-source data from different sensors to extract more comprehensive and accurate feature information. The present invention adopts a multi-modal data fusion method based on deep learning to perform data fusion, which specifically includes the following steps:

[0067] 1) Data preprocessing: Perform preprocessing operations such as denoising, normalization, and time alignment on spatiotemporal feature data from different data sources.

[0068] 2) Multimodal feature fusion: The spatiotemporal features extracted from different modalities (such as images, radar, sensor data, etc.) are fused using feature-level, decision-level or deep fusion models.

[0069] (2) Dynamic Compression

[0070] The scene data of highly time-sensitive targets is highly time-varying. Rapid target movement, sensor measurement errors, and environmental changes can cause changes in data distribution. The data compression process dynamically adjusts compression parameters and strategies based on the current scene characteristics. The following are the main steps of dynamic compression:

[0071] 1) Real-time scene analysis and adaptive parameter adjustment: Real-time monitoring of changes in data sources and dynamic adjustment of compression algorithm parameters.

[0072] 2) Adaptive dictionary learning based on temporal and spatial characteristics: Using the temporal and spatial features extracted by the deep learning network, based on online or incremental dictionary learning methods, the dictionary is updated in real time to adapt to data changes.

[0073] 3) Adaptive sparse coding: Automatically adjust the sparsity according to the characteristics of the input data and dynamically select the optimal sparsity, thereby improving the efficiency of data compression.

[0074] 4) Use fusion compression strategy for data compression. For data from different sensors, use different compression strategies for data compression.

[0075] In the process from step 1 to step 3, real-time response and performance optimization are also achieved

[0076] This step aims to improve the system's responsiveness to rapidly changing environments, ensure that data compression, transmission, and decoding can be completed smoothly under real-time requirements, and optimize the system's performance to ensure efficient, accurate, and low-latency operation.

[0077] (1) Priority Scheduling and Dynamic Resource Allocation

[0078] In complex scenarios, a priority scheduling mechanism is used to prioritize the processing and transmission of key data based on the importance of the target analyzed in real time. The system's computing and transmission resources are dynamically adjusted according to network conditions, computing resources, and real-time requirements. When the target changes rapidly, more computing resources are allocated to process the target data.

[0079] (2) Adaptive compression strategy

[0080] Automatically adjust the compression ratio and compression accuracy according to the dynamic changes of the current scene. When the target is static or the environment is relatively simple, increase the compression ratio; when the target is moving quickly or the environment is complex, reduce the compression ratio to retain more key information.

[0081] (3) Intelligent traffic control and transmission scheduling

[0082] Based on network status, such as bandwidth, latency, packet loss rate, etc., the system adaptively adjusts the packet size, transmission frequency and transmission order to avoid delays caused by network congestion. When the network is congested or bandwidth is insufficient, the system gives priority to transmitting important packets.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-source instantaneous saturation data compression method in a strong time-sensitive target scenario, characterized in that: The following steps are involved: Step 1: Adaptive data preprocessing: Preliminary processing of multi-source instantaneous saturation data, using noise removal and outlier detection algorithms to remove and recover unreliable data due to instantaneous saturation phenomena; Step 2: Time series analysis and feature extraction: Use time series analysis methods and deep learning models to extract spatiotemporal features of the data recovered in step 1, identify key changes and optimize data representation; Step 3: Multi-source data fusion and dynamic compression: According to the importance and change characteristics of each sensor data, a weighted average algorithm is used to align and fuse the spatiotemporal feature data of different sensors, and the most suitable compression strategy is dynamically selected for each data type to perform data compression.

2. The method according to claim 1, characterized in that During the execution of steps one to three, real-time response and performance optimization are also achieved. The entire compression process uses an adaptive algorithm to adjust the compression ratio and processing strategy according to dynamic changes in data and hardware resources to ensure that data compression and transmission are completed within the time limit.

3. The method according to claim 1, characterized in that Step 1 specifically includes: (1) Instantaneous saturation data processing The processing of instantaneous saturation data includes two aspects: noise removal and outlier detection; 1) Noise removal: Dynamically evaluate the system state through the Kalman filter to eliminate noise; The Kalman filter processes noisy observation data by estimating the current state. The target state transition model is: in, is the predicted state, A is the state transfer matrix, B is the control input matrix, and u(t) is the control input; Updates are measured by the deviation between the observations and the state estimate: K(t)=P(t|t-1)H T (HP(t|t-1)H T +R) -1 Where K(t) is the Kalman gain, P(t|t-1) is the error covariance, z(t) is the observation, R is the measurement noise covariance, and H is the measurement matrix; The Kalman filter updates the current state estimate based on the previous estimate and the current observation, thereby reducing the impact of noise in the system and outputting a more accurate signal. 2) Outlier detection is achieved through Gaussian distribution outlier detection method; Assuming that the data is approximately Gaussian distributed, we can determine the outliers by setting the standard deviation range and calculating the mean and standard deviation of the data. When the deviation value of a data point exceeds the set standard deviation range, it is determined to be an outlier. i The probability density function of is: Where μ is the mean of the data set and σ is the standard deviation. If |x i -μ|>k·σ, k is a set constant, then the data point is considered an outlier and is removed or replaced; (2) Saturation data recovery For different situations, different methods are used to recover data: 1) Linear interpolation Assume that the sensor is at the recovery point time t k The measured data x(t k ) is saturated, and x(t k ) before and after the data x(t k-1 ) and x(t k+1 ) is valid, then the recovery point time t is k Values: 2) Polynomial fitting method For time series data, the polynomial fitting method is used for data recovery; assuming that the recovery point t k The data is fitted through the front and back neighborhood data points, and the fitting process is expressed as: a0, a1, and a2 are fitting coefficients; 3) Frequency domain recovery method For signals in scenes with strong time-sensitive targets, Fourier transform or wavelet transform is used to restore the saturated signal; in Fourier transform, the frequency domain characteristics of the signal are analyzed, the frequency of the missing part is compensated, and then the time domain signal is restored through inverse transform.

4. The method according to claim 1, characterized in that The second step is to mine key information that can represent the dynamic changes of the target from the recovered data.

5. The method according to claim 1, characterized in that Step 2 specifically includes: (1) Time series analysis identifies the periodicity, trend, and noise components in the data and captures the movement pattern and change law of the target; the details are as follows: 1) Fourier transform analysis Through Fourier transform, the time domain signal is converted into the frequency domain signal. In the frequency domain, the low frequency part represents the long-term trend of the signal, while the high frequency part represents the rapid change of the target. For highly time-sensitive targets, focus on the high frequency components of the signal. 2) Wavelet transform analysis Wavelet transform is used to analyze the signal locally and non-stationary signals. By decomposing the signal at different scales, a balance is achieved between the time domain and the frequency domain, the instantaneous frequency characteristics of the signal are extracted, the target's motion pattern is identified, and the key features of the target in rapid motion are captured. (2) Deep Learning Feature Extraction For multi-source data in highly time-sensitive target scenes, deep learning methods are used to extract high-level spatiotemporal features from the data. For video and image data, CNN is used to extract spatial features; for radar or sensor data, recurrent neural networks (RNN) or long short-term memory networks (LSTM) are used to extract temporal features.

6. The method according to claim 1, characterized in that Step three specifically includes: (1) Using a multimodal data fusion method based on deep learning to perform data fusion: The spatiotemporal features extracted from different modalities are fused using feature-level, decision-level or deep fusion models; (2) Dynamic Compression 1) Real-time scene analysis and adaptive parameter adjustment: real-time monitoring of changes in data sources and dynamic adjustment of compression algorithm parameters; 2) Adaptive dictionary learning based on temporal and spatial characteristics: Using the temporal and spatial features extracted by the deep learning network, based on online or incremental dictionary learning methods, the dictionary is updated in real time to adapt to data changes; 3) Adaptive sparse coding: Automatically adjust the sparsity according to the characteristics of the input data and dynamically select the optimal sparsity; 4) For data from different sensors, different compression strategies are used to compress the data.

7. The method according to claim 1, characterized in that During the execution of steps 1 to 3, priority scheduling and dynamic resource allocation are also performed: when using the priority scheduling mechanism, priority is given to processing and transmitting key data based on the importance of the target analyzed in real time; the computing and transmission resources of the system are dynamically adjusted according to the network status, computing resources and real-time requirements; When the target change rate exceeds a preset threshold, more computing resources are allocated to process the target data.

8. The method according to claim 1, characterized in that During the execution of steps 1 to 3, the compression ratio and compression accuracy are automatically adjusted according to the dynamic changes of the current scene. When the target is static, the compression ratio is increased; when the target movement speed exceeds a preset threshold, the compression ratio is reduced.

9. The method according to claim 1, characterized in that During the execution of steps 1 to 3, intelligent flow control and transmission scheduling are also performed: based on the network status, the packet size, transmission frequency and transmission order are adaptively adjusted, and relatively important packets are transmitted first when the network is congested or the bandwidth is lower than the preset value.

10. A system for implementing the method according to any one of claims 1 to 9.

Images