Data stream compression method based on low time delay

Abstract

The invention discloses a data stream compression method based on low time delay, which comprises the following steps: extracting a key frame from an original video frame by using a convolutional neural network, and processing time sequence information of the key frame by using a recurrent neural network to obtain a prediction frame; calculating the difference between the prediction frame and the original frame to obtain a residual signal, and taking the residual signal as a compression target; residual encoding: compressing residual signals by using a neural network in an auto-encoder style; setting an attention mechanism in residual coding; the multi-level structure of the CNN effectively captures the features from low level to high level, enhances the generalization ability of the model, and enables the model to adapt to various video contents and motion scenes. The end-to-end learning simplifies the processing flow, and the strong parallel processing capability greatly improves the calculation efficiency. In addition, CNN is combined with an optimization algorithm and a regularization technology, so that the stability and performance of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to a data stream compression method based on low delay. Background Art

[0002] Wireless image transmission technology is widely used in the fields of drones and aerospace. It can provide high-definition, real-time video signal transmission. In the existing technology, the realization of low latency effect mainly focuses on the design and optimization of its hardware, as well as the selection and optimization of frequency bands. However, the compression delay generated in the encoding and decoding process is still a difficult problem to solve.

[0003] In the prior art, compression is usually performed using macroblock division, DCT, quantization, entropy coding, etc., but there are still many problems, and the hardware requirements and computational complexity are very high. Summary of the invention

[0004] The purpose of the present invention is to provide a data stream compression method based on low delay to solve the above problems.

[0005] The present invention is achieved through the following technical solutions: A data stream compression method based on low delay, comprising: Use convolutional neural networks to extract key frames from original video frames, and use recurrent neural networks to process the time series information of key frames to obtain predicted frames; Calculate the difference between the predicted frame and the original frame to obtain the residual signal, and use it as the compression target; Residual coding, which uses autoencoder-style neural networks to compress the residual signal; Setting the attention mechanism in residual encoding.

[0006] The present invention uses convolutional neural networks to extract video frames and key frames. They can automatically extract and learn key motion features in videos without manually designing feature extractors. The multi-level structure of CNN effectively captures features from low-level to high-level, enhances the generalization ability of the model, and adapts it to a variety of video content and motion scenes. End-to-end learning simplifies the processing flow, while powerful parallel processing capabilities greatly improve computing efficiency. In addition, CNN combines optimization algorithms and regularization techniques to improve the stability and performance of the model. These characteristics make CNN a core technology in video analysis, compression and enhancement tasks.

[0007] Preferably, in residual coding, the autoencoder style neural network is specifically a residual network, and in residual network training, batch normalization and ReLU activation function are used to accelerate training; The residual block in the residual network includes a convolutional layer, a batch normalization layer, and a ReLU activation function layer. The residual block adds the input directly to the output through a skip connection. The "input" here refers to the initial input of the residual block, that is, the original data without any convolution operation; and the "output" refers to the final result after a series of convolution and batch normalization operations inside the residual block.

[0008] Preferably, in the residual network, the output of the residual block is expressed as: in, is the original video frame, which represents a specific frame in the video sequence. It is a matrix containing the image data captured at a specific time point; is the residual signal, calculated as the difference between the original frame and the predicted frame; represents the pixel position, Represents a specific frame index in the time series; is the convolution kernel weight, the weight used in the convolution operation, used to extract features from the residual signal; m and n are the weight elements in the index convolution kernel; It is the convolution layer bias, the bias term in the convolution operation, which is used to translate the output of the convolution layer; is the batch normalization mean; The batch normalization standard deviation is calculated for the batch standard deviation of the convolutional layer output; It is the batch normalization scaling parameter, which is used to adjust the scale of the features after batch normalization; is the batch normalization offset parameter, which is used to adjust the feature offset after batch normalization; For the activation function output, the ReLU activation function is applied to the batch normalized output, all negative values ​​are set to 0, and the nonlinear characteristics of the network are enhanced; is the residual block output, which combines the sum of the activated batch normalization output and the original residual signal, representing the final output of the residual block, and is used to represent the residual information of the encoded video frame; and are the width and height of the convolution kernel respectively; The final encoding output of the residual network is expressed as: in is the predicted frame; is the residual block output; The final encoded output.

[0009] The ReLU activation function layer is improved, and its mathematical expression is: in is the output of the activation function; is the output after batch normalization; is the output of the activation function ReLU; represents the pixel position, Represents a specific frame index in the time series.

[0010] The present invention uses residual networks, which have significant advantages in residual compression, mainly in solving the gradient vanishing problem in deep network training, improving network performance, and maintaining parameter efficiency. They are easy to train, can learn complex mapping relationships, and adapt to network structures of different depths, and are particularly suitable for image compression tasks. Residual networks effectively capture local features, improve compression quality, and reduce artifacts through feature reuse and cross-layer information flow. In addition, the flexibility and scalability of its architecture allow it to adapt to a variety of compression modes.

[0011] Preferably, the residual network also includes a rate-distortion optimization method, which specifically includes: The encoder compares the prediction error produced by different coding modes and the number of coding bits required; Use Lagrange multipliers to balance the quality and file size of the encoded video; Evaluate each mode and select the encoding mode that can achieve the best video quality at a specific bit rate; The quantization parameters and encoding modes are continuously adjusted in an iterative manner until the optimal encoding settings are found.

[0012] Preferably, the attention mechanism is a multi-head attention mechanism. The multi-head attention maps the query Query, key Key and value Value to multiple subspaces through different linear transformations. Each head calculates the attention output independently, and then merges the outputs of all heads to form a comprehensive representation. The specific algorithm is: given a query q, key k and value v, each head The calculation method is: in , and is the learnable parameter matrix, is the attention pooling function, Optionally scale the dot-product attention function.

[0013] Preferably, the implementation of the multi-head attention mechanism requires the following steps: S1: segmentation and linear transformation; The input data passes through different weight matrices , and Perform linear transformation to obtain the query, key and value of each header; S2: attention calculation; Each head independently computes its output using a scaled dot-product attention mechanism; S3: Merge and output; The outputs of all heads are concatenated and then passed through another learnable linear transformation Processing is performed to generate the final output.

[0014] The present invention uses a multi-head attention mechanism, which enables the model to analyze data from different angles at the same time, capture more comprehensive features and information, and thus enhance the understanding of complex data structures. This mechanism not only speeds up training, but also improves the model's robustness to input noise and reduces the risk of overfitting by dispersing attention weights.

[0015] Preferably, the method for processing time series information of a video using a recurrent neural network includes: Extract time series features from video frames, including pixel values, color distribution, and texture information; Capturing the temporal dependencies between video frames through its recurrent structure, including capturing whether there is a clear temporal correlation between consecutive frames of an action; By increasing the network depth, the recurrent neural network is transformed into a deep recurrent neural network, which can capture higher-order temporal features and longer-range dependencies, and can process complex, nonlinear, and long-distance dependent sequence data; When training recurrent neural networks, use gradient clipping optimization methods to adjust learning rate strategies to improve model performance and resolve gradient issues.

[0016] Preferably, it also includes generating an adversarial network to improve the perceived quality of the compressed video, and generating a high-quality reconstructed video through adversarial training, which includes: The loss function of the generative adversarial network is a minimum-maximum problem, involving the adversarial process between the generator and the discriminator. The loss function of the generator is to maximize the probability that the discriminator will judge the generated sample as true, while the loss function of the discriminator is the sum of its classification losses for the original video frame and the generated sample. The loss function of the generator is expressed as: The loss function of the discriminator is expressed as: The loss function in the generative adversarial network defines a generator ( ) and the discriminator ( ). The following are the meanings of the various parameters in the GANs loss function: and : These operators define the optimization objectives in GANs. " " represents the generator Trying to maximize its loss function, while " " represents the discriminator Try to minimize its loss function. This is a min-max problem in GANs, where the generator strives to generate more and more realistic samples, while the discriminator strives to distinguish between the original video frames and the generated samples; : This is the probability distribution over the latent space The noise vector Take the expected representation. Here Usually random noise sampled from a standard normal distribution or other simple distribution, the generator Take this noise vector as input.

[0017] : This is the generator part of the loss function. Hope Discriminator The generated sample The greater the probability of being judged to be true, the better, so Try to maximize this log loss.

[0018] :This is the real data distribution Download the original video frame Take the desired representation. The original video frames are sampled from the training data.

[0019] :This is the discriminator Part of the loss function for the original video frame The discriminator hopes that it can judge the original video frame as having a high probability of being real, so Try to minimize this log loss.

[0020] :This is the discriminator The other part of the loss function is to generate samples The discriminator hopes that it can judge the generated samples as false (i.e., with low probability) with as high accuracy as possible, so Try to minimize this log loss.

[0021] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The present invention uses convolutional neural networks to extract motion information between video frames, which has significant advantages in predicting motion information between video frames. They can automatically extract and learn key motion features in videos without manually designing feature extractors. The multi-level structure of CNN effectively captures features from low-level to high-level, enhances the generalization ability of the model, and adapts it to a variety of video content and motion scenes. End-to-end learning simplifies the processing flow, while powerful parallel processing capabilities greatly improve computing efficiency. In addition, CNN combines optimization algorithms and regularization techniques to improve the stability and performance of the model. These characteristics make CNN a core technology in video analysis, compression, and enhancement tasks; 2. The present invention uses residual networks, which have significant advantages in residual compression, mainly in solving the gradient vanishing problem in deep network training, improving network performance, and maintaining parameter efficiency. They are easy to train, can learn complex mapping relationships, and adapt to network structures of different depths, and are particularly suitable for image compression tasks. Residual networks effectively capture local features, improve compression quality, and reduce artifacts through feature reuse and cross-layer information flow. In addition, the flexibility and scalability of its architecture allow it to adapt to a variety of compression modes; DETAILED DESCRIPTION

[0022] The detailed description of the embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the present invention. It should be noted that relational terms such as the terms "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0023] Moreover, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of more restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0024] Example This embodiment includes: a data stream compression method based on low latency, including: Use convolutional neural networks to extract key frames from original video frames, and use recurrent neural networks to process the time series information of key frames to obtain predicted frames; The process of extracting key frames from original video frames using convolutional neural networks includes the following steps: Preprocess the raw data to make it suitable for network input by scaling and cropping; The convolutional neural network includes multiple convolutional layers, activation functions, and pooling layers, and realizes dynamic extraction of key frames according to the set key frame body strategy; The key frame features are encoded into a vector of fixed length, and the encoded key frames are arranged in chronological order to form time series data.

[0025] Calculate the difference between the predicted frame and the original frame to obtain the residual signal, and use it as the compression target; Residual coding, which uses autoencoder-style neural networks to compress the residual signal; Setting the attention mechanism in residual encoding.

[0026] The present invention uses convolutional neural networks to extract motion information between video frames, which has significant advantages in predicting motion information between video frames. They can automatically extract and learn key motion features in videos without manually designing feature extractors. The multi-level structure of CNN effectively captures features from low-level to high-level, enhances the generalization ability of the model, and adapts it to a variety of video content and motion scenes. End-to-end learning simplifies the processing flow, and powerful parallel processing capabilities greatly improve computing efficiency. In addition, CNN combines optimization algorithms and regularization techniques to improve the stability and performance of the model. These characteristics make CNN a core technology in video analysis, compression and enhancement tasks.

[0027] Preferably, in residual coding, the autoencoder style neural network is specifically a residual network (ResNet), and during training, batch normalization and ReLU activation function are used to accelerate training and improve the generalization ability of the model; The residual block includes a convolutional layer, a batch normalization layer, and a ReLU activation function layer. The residual block adds the input directly to the output through a skip connection, allowing the gradient to flow directly in the network.

[0028] Preferably, in the residual network, the output of the residual block is expressed as: in, is the original video frame, which represents a specific frame in the video sequence. It is a matrix containing the image data captured at a specific time point; is the residual signal, calculated as the difference between the original frame and the predicted frame; represents the pixel position, Represents a specific frame index in the time series; is the convolution kernel weight, the weight used in the convolution operation, used to extract features from the residual signal; m and n are the weight elements in the index convolution kernel; It is the convolution layer bias, the bias term in the convolution operation, which is used to translate the output of the convolution layer; is the batch normalization mean; The batch normalization standard deviation is calculated for the batch standard deviation of the convolutional layer output; It is the batch normalization scaling parameter, which is used to adjust the scale of the features after batch normalization; is the batch normalization offset parameter, which is used to adjust the feature offset after batch normalization; For the activation function output, the ReLU activation function is applied to the batch normalized output, all negative values ​​are set to 0, and the nonlinear characteristics of the network are enhanced; is the residual block output, which combines the sum of the activated batch normalization output and the original residual signal, representing the final output of the residual block, and is used to represent the residual information of the encoded video frame; and are the width and height of the convolution kernel respectively; The final encoding output of the residual network is expressed as: in is the predicted frame; is the residual block output; The final encoded output.

[0029] The ReLU activation function layer is improved, and its mathematical expression is: in is the output of the activation function; is the output after batch normalization; is the output of the activation function ReLU; represents the pixel position, Represents a specific frame index in the time series.

[0030] The present invention uses residual networks, which have significant advantages in residual compression, mainly in solving the gradient vanishing problem in deep network training, improving network performance, and maintaining parameter efficiency. They are easy to train, can learn complex mapping relationships, and adapt to network structures of different depths, and are particularly suitable for image compression tasks. Residual networks effectively capture local features, improve compression quality, and reduce artifacts through feature reuse and cross-layer information flow. In addition, the flexibility and scalability of its architecture allow it to adapt to a variety of compression modes.

[0031] Preferably, the residual network also includes a rate-distortion optimization method, which specifically includes: The encoder compares the prediction error produced by different coding modes and the number of coding bits required; Use Lagrange multipliers to balance the quality and file size of the encoded video; Evaluate each mode and select the encoding mode that can achieve the best video quality at a specific bit rate; The quantization parameters and encoding modes are continuously adjusted in an iterative manner until the optimal encoding settings are found.

[0032] Preferably, the attention mechanism is a multi-head attention mechanism. The multi-head attention maps the query Query, key Key and value Value to multiple subspaces through different linear transformations. Each head calculates the attention output independently, and then merges the outputs of all heads to form a comprehensive representation. The specific algorithm is: given a query q, key k and value v, each head The calculation method is: in , and is the learnable parameter matrix, is the attention pooling function, Optionally scale the dot-product attention function.

[0033] Preferably, the implementation of the multi-head attention mechanism requires the following steps: S1: segmentation and linear transformation; The input data passes through different weight matrices , and Perform linear transformation to obtain the query, key and value of each header; S2: attention calculation; Each head independently computes its output using a scaled dot-product attention mechanism; S3: Merge and output; The outputs of all heads are concatenated and then passed through another learnable linear transformation Processing is performed to generate the final output.

[0034] The present invention uses a multi-head attention mechanism, which enables the model to analyze data from different angles at the same time, capture more comprehensive features and information, and thus enhance the understanding of complex data structures. This mechanism not only speeds up training, but also improves the model's robustness to input noise and reduces the risk of overfitting by dispersing attention weights.

[0035] Preferably, the method for processing time series information of a video using a recurrent neural network includes: Extract time series features from video frames, including pixel values, color distribution, and texture information; Capturing the temporal dependencies between video frames through its recurrent structure, including capturing whether there is a clear temporal correlation between consecutive frames of an action; By increasing the network depth, the recurrent neural network is transformed into a deep recurrent neural network, which can capture higher-order temporal features and longer-range dependencies, and can process complex, nonlinear, and long-distance dependent sequence data; When training recurrent neural networks, use gradient clipping optimization methods to adjust learning rate strategies to improve model performance and resolve gradient issues.

[0036] Preferably, it also includes generating an adversarial network to improve the perceived quality of the compressed video, and generating a high-quality reconstructed video through adversarial training, which includes: The loss function of the generative adversarial network is a minimum-maximum problem, involving the adversarial process between the generator and the discriminator. The loss function of the generator is to maximize the probability that the discriminator will judge the generated sample as true, while the loss function of the discriminator is the sum of its classification losses for the original video frame and the generated sample. The loss function of the generator is expressed as: The loss function of the discriminator is expressed as: The loss function in the generative adversarial network defines a generator ( ) and the discriminator ( ). The following are the meanings of the various parameters in the GANs loss function: and : These operators define the optimization objectives in GANs. " " represents the generator Trying to maximize its loss function, while " " represents the discriminator Try to minimize its loss function. This is a min-max problem in GANs, where the generator strives to generate more and more realistic samples, while the discriminator strives to distinguish between the original video frames and the generated samples; : This is the probability distribution over the latent space The noise vector Take the expected representation. Here Usually random noise sampled from a standard normal distribution or other simple distribution, the generator Take this noise vector as input.

[0037] : This is the generator part of the loss function. Hope Discriminator The generated sample The greater the probability of being judged to be true, the better, so Try to maximize this log loss.

[0038] :This is the real data distribution Download the original video frame Take the desired representation. The original video frames are sampled from the training data.

[0039] :This is the discriminator Part of the loss function for the original video frame .

[0040] :This is the discriminator The other part of the loss function is to generate samples .

[0041] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data stream compression method based on low latency, characterized in that: include: Use convolutional neural networks to extract key frames from original video frames, and use recurrent neural networks to process the time series information of key frames to obtain predicted frames; Calculate the difference between the predicted frame and the original frame to obtain the residual signal, and use it as the compression target; Residual coding, which uses autoencoder-style neural networks to compress the residual signal; Setting the attention mechanism in residual encoding.

2. A data stream compression method based on low latency according to claim 1, characterized in that: In residual coding, the autoencoder-style neural network is specifically a residual network. In residual network training, batch normalization and ReLU activation function are used to accelerate training; The residual block in the residual network includes a convolution layer, a batch normalization layer and a ReLU activation function layer. The residual block directly adds the input to the output through a jump connection. The input is the initial input of the residual block; the output is the final output after the convolution and batch normalization operations inside the residual block.

3. A low-latency data stream compression method according to claim 2, characterized in that: In the residual network, the output of the residual block is expressed as: in, is the original video frame, which represents a specific frame in the video sequence. It is a matrix containing the image data captured at a specific time point; is the residual signal, calculated as the difference between the original frame and the predicted frame; represents the pixel position, Represents a specific frame index in the time series; is the convolution kernel weight, the weight used in the convolution operation, used to extract features from the residual signal; m and n are the weight elements in the index convolution kernel; It is the convolution layer bias, the bias term in the convolution operation, which is used to translate the output of the convolution layer; is the batch normalization mean; The batch normalization standard deviation is calculated for the batch standard deviation of the convolutional layer output; It is the batch normalization scaling parameter, which is used to adjust the scale of the features after batch normalization; is the batch normalization offset parameter, which is used to adjust the feature offset after batch normalization; For the activation function output, the ReLU activation function is applied to the batch normalized output, all negative values ​​are set to 0, and the nonlinear characteristics of the network are enhanced; is the residual block output, which combines the sum of the activated batch normalization output and the original residual signal, represents the final output of the residual block, and is used to represent the residual information of the encoded video frame; and are the width and height of the convolution kernel respectively.

4. A low-latency data stream compression method according to claim 3, characterized in that: The final encoding output of the residual network is expressed as: in is the predicted frame; is the residual block output; The final encoded output.

5. A low-latency data stream compression method according to claim 4, characterized in that: The ReLU activation function layer is improved, and its mathematical expression is: in is the output of the activation function; is the output after batch normalization; is the output of the activation function ReLU; represents the pixel position, Represents a specific frame index in the time series.

6. A low-latency data stream compression method according to claim 5, characterized in that: The residual network also includes rate-distortion optimization, which specifically includes: The encoder compares the prediction error produced by different coding modes and the number of coding bits required; Use Lagrange multipliers to balance the quality and file size of the encoded video; Evaluate each mode and select the encoding mode that can achieve the best video quality at a specific bit rate; The quantization parameters and encoding modes are continuously adjusted in an iterative manner until the optimal encoding settings are found.

7. A low-latency data stream compression method according to claim 6, characterized in that: The attention mechanism is specifically a multi-head attention mechanism. The multi-head attention maps the query Query, key Key and value Value to multiple subspaces through different linear transformations. Each head calculates the attention output independently, and then merges the outputs of all heads to form a comprehensive representation. The specific algorithm is: given a query q, key k and value v, each head The calculation method is: in , and is the learnable parameter matrix, is the attention pooling function.

8. The low-latency data stream compression method according to claim 7, characterized in that: The implementation of the multi-head attention mechanism requires the following steps: S1: segmentation and linear transformation; The input data passes through different weight matrices , and Perform linear transformation to obtain the query, key and value of each header; S2: attention calculation; Each head computes its output independently using a scaled dot-product attention mechanism; S3: Merge and output; The outputs of all heads are concatenated and then passed through another learnable linear transformation Processing is performed to generate the final output.

9. The low-latency data stream compression method according to claim 8, characterized in that: The methods of processing time series information of videos by recurrent neural networks include: Extract time series features from video frames, including pixel values, color distribution, and texture information; Capturing the temporal dependencies between video frames through its recurrent structure, including capturing whether there is a clear temporal correlation between consecutive frames of an action; By increasing the network depth, the recurrent neural network is transformed into a deep recurrent neural network; When training recurrent neural networks, use the gradient clipping optimization method to adjust the learning rate strategy.

10. A low-latency data stream compression method according to claim 9, characterized in that: It also includes a generative adversarial network that generates high-quality reconstructed videos through adversarial training, including: The loss function of the generator is expressed as: The loss function of the discriminator is expressed as: The following are the meanings of the various parameters in the GANs loss function: and : These operators define the optimization objectives in GANs; Representation Generator Trying to maximize its loss function, while Representation Discriminator Try to minimize its loss function; : This is the probability distribution over the latent space The noise vector Take the expected representation; here is random noise sampled from a standard normal distribution or other simple distribution, the generator Take this noise vector as input; : This is the generator Part of the loss function; the discriminator The generated sample The probability of being judged to be true; :This is the real data distribution Download the original video frame Take the desired representation; the original video frames are sampled from the training data; :This is the discriminator Part of the loss function for the original video frame ; :This is the discriminator The other part of the loss function is to generate samples , Minimize this log loss.

Images