Video feature coding method for machine vision

Abstract

The invention discloses a machine vision-oriented video feature coding method, which comprises the following steps of: coding a first frame x0 in a video sequence, performing feature extraction on the first frame x0 by using a backbone network to obtain a multi-scale visual feature, converting # imgabs0 # into compact representation by using an analysis conversion module, and coding and decoding the compact representation by using an entropy coder-decoder to obtain a multi-scale visual feature; obtaining a target detection result; for P frames {x1,..., xN} encoding, feature extraction is performed on the Nth frame xN by using a backbone network to obtain visual features, and the visual features are encoded and decoded based on the machine vision-oriented efficient video feature encoding network to obtain reconstructed visual features; and performing end-to-end joint optimization on the efficient video feature coding network by using the distortion loss function, and performing video feature coding based on the optimized network. According to the method, various time domain reference information in the reference information cache is utilized to realize efficient coding of the visual features of the current frame.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and video coding, and in particular to a video feature coding method for machine vision. Background Art

[0002] With the continuous development of artificial intelligence technology in the field of video analysis and processing, the application scope of video data has exceeded the scope of human visual viewing, and video coding technology for machine vision has broad application prospects. Traditional video coding methods are mainly guided by human vision and aim to reconstruct high-fidelity videos at the lowest possible bit rate. At the same time, the development of deep learning technology has spawned some end-to-end video coding methods. Although these methods have performed well in improving video coding efficiency, their encoding process is still mainly oriented towards human vision rather than machine vision, which to some extent limits their application in intelligent platforms. Therefore, studying video coding technology for machine vision has important research significance and application value.

[0003] Considering the video coding needs for machine vision, MPEG, an international image / video coding standard setting organization, established the VCM (machine vision coding) standard working group, aiming to develop an efficient bitstream format, which can reduce the bit rate consumption and obtain better machine vision task performance by encoding and decoding the original video or video visual features. The VCM standard working group proposed two modes: analysis first, compression later, and compression first, analysis later. Among them, the analysis first, compression later mode only encodes the visual features related to the machine vision task, thus having a smaller decoding complexity and theoretically better coding efficiency. Considering that visual features usually cover the semantic-level description of the same scene at different scales and channels, and their numerical distribution has high sparsity and discontinuity, traditional video coding methods are not completely suitable for visual feature coding. In order to solve the above problems, the VCM standard working group proposed a variety of efficient visual feature coding methods, which effectively improved the coding efficiency of VCM by modeling the relationship between visual feature channels and scales. Specifically, Liu et al. proposed an efficient image feature coding network, which uses the correlation between image features at different scales and uses the encoded small-scale features to predict large-scale features, effectively improving the efficiency of visual feature coding. Kim et al. proposed a multi-scale image feature compression method, which jointly optimized the visual feature extraction and visual feature compression network, significantly improving the efficiency of visual feature coding. However, the above research work mainly focuses on image visual feature coding methods, and there are few studies exploring coding methods for video visual features.

[0004] Compared with image visual feature encoding, methods designed for video visual feature encoding need to more deeply explore the intrinsic relationship between the visual features of consecutive frames to further reduce the bit rate required for encoding the video and improve the performance of machine vision tasks. Summary of the invention

[0005] The present invention provides a video feature coding method for machine vision. The present invention designs an efficient video feature coding network based on inter-frame semantic consistency, explores the temporal correlation between visual features of consecutive frames, and effectively improves the coding efficiency of VCM. In addition, the present invention proposes a feature prediction module based on inter-frame semantic consistency, which dynamically maintains the inter-frame semantic consistency of continuous visual features through a long short-term memory mechanism to improve the accuracy of inter-frame feature prediction, as described below:

[0006] A video feature encoding method for machine vision, the method comprising:

[0007] For the first frame x0 in the video sequence, the backbone network is used to extract the features of the first frame x0 to obtain multi-scale visual features. Transform into a compact representation, and use the entropy codec to encode and decode the compact representation to obtain the target detection result;

[0008] For a P frame {x1,...,x N} encoding, using the backbone network to encode the Nth frame x N Perform feature extraction to obtain visual features, encode and decode the visual features based on an efficient video feature coding network for machine vision to obtain reconstructed visual features;

[0009] The efficient video feature coding network is end-to-end jointly optimized using the rate-distortion loss function, and video feature encoding is performed based on the optimized network.

[0010] The efficient video feature coding network includes four parts: a feature prediction module based on inter-frame semantic consistency, an analysis transform, a synthesis transform and an entropy codec.

[0011] The feature prediction module based on inter-frame semantic consistency is: using the video long short-term memory in the reference information cache and the encoded visual representation of the previous frame Generate prediction features with semantic consistency across frames The reference information cache Dynamically updated with reference information at the current moment

[0012] The analysis transformation is: transforming Progressively transforming into a joint compact visual representation The formula is as follows:

[0013]

[0014] Among them, conv ↓2 (·) represents a downsampling convolutional layer with a size of 5×5 and a stride of 2, h(·) represents a downsampling convolutional layer with a size of 5×5 and a stride of 2 and a GDN layer, Represents the channel-dimensional concatenation of features.

[0015] The entropy model aggregates The time domain prior and r N The spatial domain prior is used in the arithmetic coding and decoding process; the synthetic transform adopts a structure that is inverse to the analytical transform by and Add together to obtain the reconstructed joint visual representation and from The reconstructed multi-scale visual features are gradually decoded

[0016] The entropy model includes: a super priori entropy codec and a probability estimation network, and the residual r N Perform super-prior encoding and decoding, and convert the decoded residual information and time domain prior Combined, sent to the probability estimation network for processing, the residual r N The probability distribution parameters μ and σ.

[0017] The feature prediction module based on inter-frame semantic consistency uses a low-frequency forgetting mechanism to predict the long-term memory c of the video. N-1 The low-frequency semantic information in c is filtered out. N-1 Focus more on preserving high-frequency semantic components that are beneficial to visual tasks;

[0018] The feature prediction module based on inter-frame semantic consistency adopts a high-frequency memory mechanism to The high-frequency semantic information of the current encoding moment is supplemented in the , generating an updated video long-term memory c N ;

[0019] By adding reference features Medium adaptive supplementary long-term memory c N The inter-frame semantic information in the frame is used to generate inter-frame prediction features with inter-frame semantic consistency.

[0020] The beneficial effects of the technical solution provided by the present invention are:

[0021] 1. The present invention efficiently encodes video visual features in an end-to-end manner, effectively improving the encoding efficiency of VCM;

[0022] 2. The present invention designs an efficient video feature coding network based on inter-frame semantic consistency, and utilizes multiple temporal reference information in the reference information cache to achieve efficient coding of the visual features of the current frame;

[0023] 3. The present invention proposes a feature prediction module based on inter-frame semantic consistency, which continuously memorizes and updates the semantic information of video frames by constructing a long short-term memory mechanism, and adaptively supplements the relevant inter-frame semantics for the reference features to generate prediction features with inter-frame semantic consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The flowchart of a video feature encoding method for machine vision. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0026] 1. Constructing I-frame visual feature encoding network

[0027] For the encoding of the first frame x0 in the video sequence, we first use the backbone network DarkNet-53 to extract features from the first frame x0 to obtain multi-scale visual features. Then, the analysis transformation module Transform into a compact representation, and use the entropy codec to encode and decode the compact representation to obtain the reconstructed compact representation Then, the reconstructed compact representation is transformed into Inverse transformation to reconstruct multi-scale visual features During decoding, the compact representation of As time domain reference information, it is put into the reference information cache as reference information for subsequent P frame encoding. The target detection result D0 is input into the visual analysis model, and the target detection result D0 includes: the detected target and the target category of the current encoding frame.

[0028] 2. Constructing P-frame visual feature encoding network

[0029] For a P frame {x1,...,x N} encoding, with the Nth frame x N For example, we first use the backbone network DarkNet-53 to train the Nth frame x N Perform feature extraction to obtain its visual features Then, the efficient video feature coding network EVFC-Net for machine vision is used to Perform encoding and decoding to obtain reconstructed visual features Among them, the proposed feature prediction module based on inter-frame semantic consistency is used to generate prediction features To improve the efficiency of visual feature inter-frame coding. Finally, the visual analysis model is used to obtain the target detection result D of the current coding frame. N The visual analysis model in the embodiment of the present invention refers to the Feature Pyramid Network (FPN).

[0030] 3. Building an efficient video feature coding network based on inter-frame semantic consistency

[0031] The video multi-target tracking task requires online matching of the same objects in the previous and next frames. In order to ensure the temporal continuity of the video tracking results, the visual features of the previous and next frames need to have inter-frame semantic consistency. To this end, an embodiment of the present invention designs an efficient video feature coding network EVFC-Net for machine vision, which mainly includes:

[0032] The feature prediction module ISCFP based on inter-frame semantic consistency, analysis transform, synthesis transform and entropy codec are composed of four parts.

[0033] First, the feature prediction module ISCFP uses the video long short-term memory {c N-1 ,h N-1} and the encoded visual representation of the previous frame Generate prediction features with semantic consistency across frames During this process, the ISCFP module references the information cache. Dynamically updated with reference information at the current moment Subsequently, in order to fully remove the redundant information between spatial domains and scales in visual features, the analytic transformation is used to transform the images with different resolutions. Progressively transforming into a joint compact visual representation The formula is as follows:

[0034]

[0035] Among them, conv ↓2 (·) represents a downsampling convolutional layer with a size of 5×5 and a stride of 2, h(·) represents a downsampling convolutional layer with a size of 5×5 and a stride of 2 and a GDN layer, Represents the channel-dimensional concatenation of features. The visual features of the previous P frame are encoded through network coding, and the encoded visual representation is stored in the reference information cache as the reference information for encoding the visual features of the current P frame. Prediction features obtained with the ISCFP module Subtract and obtain the final entropy coded residual r N .

[0036] Then, in the entropy codec, r N Perform quantization, arithmetic coding and arithmetic decoding to generate the decoded residual Among them, considering that the visual features of the current frame have redundant information in the temporal and spatial domains, the entropy model aggregates The time domain prior and r N The spatial domain prior provides a more accurate probability estimation for the arithmetic coding and decoding process, thereby improving the efficiency of arithmetic coding and decoding.

[0037] Finally, the synthetic transformation adopts a structure that is the inverse of the analytical transformation, by and Add together to obtain the reconstructed joint visual representation and from The reconstructed multi-scale visual features are gradually decoded

[0038] Among them, the entropy model mainly includes: super prior entropy codec and probability estimation network. First, the residual r N Perform super-prior encoding and decoding, and convert the decoded residual information and time domain prior Combined, and finally sent to the probability estimation network for processing, and then the residual r is estimated N The probability distribution parameters μ and σ of , where both the super-prior encoder-decoder and the probability estimation network are composed of several convolutional layers.

[0039] 4. Constructing a feature prediction module based on inter-frame semantic consistency

[0040] In order to improve the inter-frame coding efficiency of video features, the embodiment of the present invention constructs a feature prediction module ISCFP based on inter-frame semantic consistency, which forgets low-frequency semantic information such as image background by constructing a low-frequency forgetting mechanism, and memorizes high-frequency semantic information such as target texture by using a high-frequency memory mechanism, and finally generates prediction features with inter-frame semantic consistency. The input of the ISCFP module is the reference visual feature Video long-term memoryc N-1 and video short-term memory h N-1 , the output is the inter-frame prediction feature and the updated video LSTM {c N ,h N}. First, use DWT to transform and h N-1 Perform frequency band decomposition, the formula is as follows:

[0041]

[0042] Where DWT(·) represents discrete wavelet transform, LL J and LL hRespectively and h N-1 low frequency components. for The high frequency component of h N-1 The high frequency component, HL J represents the vertical edge feature of inter-frame prediction, HL h The vertical edge feature representing video short-term memory, LH J Indicates the horizontal edge features of inter-frame prediction, LH h Represents the horizontal edge features of video short-term memory, HH J represents the diagonal features of inter-frame prediction representation, HH h Diagonal features representing short-term memory of videos.

[0043] Then, considering that low-frequency components usually contain smooth content such as image background, the ISCFP module uses a low-frequency forgetting mechanism to store long-term memory c in the video. N-1 The low-frequency semantic information in c is filtered out. N-1 Focus more on retaining high-frequency semantic components that are beneficial to visual tasks. This process can be formulated as:

[0044]

[0045] Among them, conv ↑2 (·) represents the upsampling convolutional layer with size 5×5 and stride 2, σ(·) represents the Sigmoid function, Represents the long-term memory of the video after forgetting the low-frequency components.

[0046] Then, in order to ensure that the video long-term memory can effectively adapt to the changes in the video target texture, the ISCFP module adopts a high-frequency memory mechanism to The high-frequency semantic information of the current encoding moment is supplemented in the , generating an updated video long-term memory c N , the specific formula is as follows:

[0047]

[0048] Where t(·) represents the Tanh function.

[0049] Finally, by adding Medium adaptive supplementary long-term memory c N The inter-frame semantic information in the frame is used to generate inter-frame prediction features with inter-frame semantic consistency. The specific formula is as follows:

[0050]

[0051] Inter-frame prediction features The main functions are divided into two aspects: first, by using the inter-frame prediction features and joint visual representation Subtracting, we can get the residual information r N , entropy coding of residual information can effectively reduce the bit rate; secondly, inter-frame prediction features It can be used as time domain reference information in the entropy model to provide more accurate probability estimation for the arithmetic coding and decoding process, thereby improving the efficiency of arithmetic coding and decoding.

[0052] 5. Training a Video Feature Coding Network for Machine Vision

[0053] The embodiment of the present invention uses the rate-distortion loss function L EVFC The proposed EVFC-Net is end-to-end jointly optimized, and the specific formula is as follows:

[0054] L EVFC =λD v +R

[0055] Where R represents the encoding bit rate, λ represents the hyperparameter controlling the bit rate, and D v represents the video multi-target tracking loss. Specifically, D v Including foreground / background classification loss L a , detection box regression loss L β and the embedding loss L r The three parts are expressed as follows:

[0056]

[0057] Where M represents the number of prediction heads in visual analysis, u=1,...,M,j=α,β,γ represents the uncertainty associated with the task.

[0058] In order to meet the requirements of different encoding bit rates, the embodiment of the present invention trains 7 network models when λ={4, 2, 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16}. In terms of network structure, according to the experimental settings specified by MPEG, the backbone network and visual analysis module of the overall encoding framework of the proposed method are consistent with the JDE-1088x608 model.

[0059] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video feature encoding method for machine vision, It is characterized in that The method comprises: For the first frame x in the video sequence 0 Encoding, using the backbone network to encode the first frame x 0 Perform feature extraction to obtain multi-scale visual features, and transform them into Transform into a compact representation, and use the entropy codec to encode and decode the compact representation to obtain the target detection result; For P frame {x 1 ,...,x N } encoding, using the backbone network to encode the Nth frame x N Perform feature extraction to obtain visual features, encode and decode the visual features based on an efficient video feature coding network for machine vision to obtain reconstructed visual features; The efficient video feature coding network is end-to-end jointly optimized using the rate-distortion loss function, and video feature encoding is performed based on the optimized network.

2. The video feature encoding method for machine vision according to claim 1, It is characterized in that The efficient video feature coding network includes four parts: a feature prediction module based on inter-frame semantic consistency, an analysis transform, a synthesis transform and an entropy codec.

3. The video feature encoding method for machine vision according to claim 2, It is characterized in that The feature prediction module based on inter-frame semantic consistency is: using the video long short-term memory in the reference information cache and the encoded previous frame visual representation Generate prediction features with semantic consistency across frames The reference information cache Dynamically updated with reference information at the current moment 4. The method for encoding video features for machine vision according to claim 2, It is characterized in that The analysis transformation is: Progressively transforming into a joint compact visual representation The formula is as follows: Among them, conv ↓2 (·) represents a downsampling convolutional layer with a size of 5×5 and a stride of 2, h(·) represents a downsampling convolutional layer with a size of 5×5 and a stride of 2 and a GDN layer, Represents the channel-dimensional concatenation of features.

5. The video feature encoding method for machine vision according to claim 2, It is characterized in that The entropy model aggregates The time domain prior and r N The spatial domain prior is used in the arithmetic encoding and decoding process; The synthetic transformation adopts a structure that is inverse to the analytical transformation by and Add together to obtain the reconstructed joint visual representation and from The reconstructed multi-scale visual features are gradually decoded 6. The video feature encoding method for machine vision according to claim 2, It is characterized in that The entropy model includes: a super priori entropy codec and a probability estimation network, and the residual r N Perform super-prior encoding and decoding, and convert the decoded residual information and time domain prior Combined, sent to the probability estimation network for processing, the residual r N The probability distribution parameters μ and σ.

7. The video feature encoding method for machine vision according to claim 2, It is characterized in that The feature prediction module based on inter-frame semantic consistency uses a low-frequency forgetting mechanism to store video long-term memory c N-1 The low-frequency semantic information in c is filtered out. N-1 Focus more on preserving high-frequency semantic components that are beneficial to visual tasks; The feature prediction module based on inter-frame semantic consistency adopts a high-frequency memory mechanism to The high-frequency semantic information of the current encoding moment is supplemented in the , generating an updated video long-term memory c N ; By adding reference features Medium adaptive supplementary long-term memory c N The inter-frame semantic information in the frame is used to generate inter-frame prediction features with inter-frame semantic consistency.

Images