A machine vision-oriented video feature encoding method

Abstract

The application discloses a kind of machine vision-oriented video feature encoding methods, comprising: for the first frame x0 in video sequence Coding, using backbone network to the first frame x0 Feature extraction is carried out, obtains multi-scale visual feature, is transformed into compact representation by analysis transformation module, and using entropy encoder and decoder is encoded and decoded to compact representation, obtains target detection result;For P frame {x1,...,x N}Coding, using backbone network to the Nth frame x N Feature extraction is carried out, obtains visual feature, based on machine vision-oriented efficient video feature encoding network to visual feature Encoding, obtains reconstructed visual feature;Using rate-distortion loss function End-to-end joint optimization is carried out to efficient video feature encoding network, and video feature encoding is carried out based on the network after optimization.The application utilizes the multiple time domain reference information in reference information cache, realizes the efficient coding of current frame visual feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and video coding, and in particular to a video feature coding method for machine vision. BACKGROUND

[0002] With the continuous development of artificial intelligence technology in the field of video analysis and processing, the application range of video data has gone beyond the scope of human visual observation, and video coding technology for machine vision has broad application prospects. Traditional video coding methods are mainly oriented towards human visual observation, aiming to reconstruct high-fidelity video with as low a code rate as possible. At the same time, the development of deep learning technology has given rise to some end-to-end video coding methods. Although these methods perform well in improving video coding efficiency, their coding process is still mainly oriented towards human visual observation rather than machine visual observation, which to some extent limits their application in intelligent platforms. Therefore, it is of great research significance and application value to study video coding technology for machine vision.

[0003] Considering the video coding needs for machine vision, the international image / video coding standard organization MPEG has established a VCM (machine vision coding) standard working group, aiming to develop an efficient code stream format to code and decode original video or video visual features, so as to reduce code rate consumption while obtaining better machine vision task performance. The VCM standard working group proposes two modes of analysis first and compression second and compression first and analysis second. Among them, the analysis first and compression second mode only codes visual features related to machine vision tasks, thereby having smaller decoding complexity and theoretically better coding efficiency. Considering that visual features usually cover semantic-level descriptions of the same scene at different scales and channels, and their numerical distribution has high sparsity and non-continuity, traditional video coding methods are not completely suitable for visual feature coding. In order to solve the above problems, the VCM standard working group proposes a variety of efficient visual feature coding methods, which effectively improve the coding efficiency of VCM by modeling the relationship between visual feature channels and scales. Specifically, Liu et al. propose an efficient image feature coding network, which uses the correlation between different scale features of an image to predict large scale features using already coded small scale features, effectively improving the coding efficiency of visual features. Kim et al. propose a multi-scale image feature compression method, which jointly optimizes the visual feature extraction and visual feature compression network, significantly improving the coding efficiency of visual features. However, the above research works mainly focus on image visual feature coding methods, and currently there are few researches exploring coding methods for video visual features.

[0004] Compared with image visual feature coding, methods designed for video visual feature coding need to further explore the internal relationship between consecutive frame visual features to further reduce the code rate required for coding video and improve the performance of machine vision tasks. SUMMARY

[0005] The application provides a machine vision-oriented video feature coding method. The application designs an efficient video feature coding network based on inter-frame semantic consistency, explores the time domain correlation between continuous frame visual features, and effectively improves the coding efficiency of VCM. In addition, the application proposes a feature prediction module based on inter-frame semantic consistency. The long short-term memory mechanism is used to dynamically maintain the inter-frame semantic consistency of continuous visual features, so as to improve the inter-frame feature prediction accuracy. Details are described below:

[0006] A machine vision-oriented video feature coding method, the method comprising:

[0007] For coding of a first frame x0 in a video sequence, a backbone network is used to extract features of the first frame x0, to obtain multi-scale visual features, and an analysis transform module is used to transform the multi-scale visual features into a compact representation, and an entropy encoder is used to encode the compact representation, to obtain a target detection result;

[0008] For coding of P frames {x1,...,x N}, a backbone network is used to extract features of an Nth frame x N , to obtain visual features, and an efficient machine vision-oriented video feature coding network is used to encode the visual features, to obtain reconstructed visual features;

[0009] An end-to-end joint optimization is performed on the efficient video feature coding network using a rate-distortion loss function, and video feature coding is performed based on the optimized network.

[0010] The efficient video feature coding network comprises four parts: a feature prediction module based on inter-frame semantic consistency, an analysis transform, a synthesis transform, and an entropy encoder.

[0011] The feature prediction module based on inter-frame semantic consistency uses video long short-term memory in a reference information buffer and previously encoded visual representation to generate predicted features with inter-frame semantic consistency The video long short-term memory in the reference information buffer is dynamically updated to referenceable information at the current time

[0012] The analysis transform transforms visual features with different resolutions into a joint compact visual representation The formula is as follows:

[0013]

[0014] ​​wherein conv ↓2 (·) represents a down-sampling convolutional layer with size 5x5 and stride 2, h(·) represents a down-sampling convolutional layer with size 5x5 and stride 2 and a GDN layer, represents the channel dimension concatenation of features.

[0015] wherein the entropy model aggregates temporal priors of r N spatial priors of r for the arithmetic coding process; the synthesis transform adopts a structure reciprocal to the analysis transform, and the reconstructed joint visual representation is obtained by adding and the reconstructed multi-scale visual features are decoded from step by step.

[0016] wherein the entropy model comprises: a hyper-prior entropy coder and a probability estimation network, and the residual r N is hyper-prior coded, and the decoded residual information and temporal priors are combined and sent into the probability estimation network for processing to obtain the probability distribution parameters μ and σ of the residual r N .

[0017] wherein the feature prediction module based on inter-frame semantic consistency filters the low-frequency semantic information in the video long-term memory c N-1 through a low-frequency forgetting mechanism, so that c N-1 is more focused on retaining the high-frequency semantic components beneficial to the visual task;

[0018] The feature prediction module based on inter-frame semantic consistency supplements the high-frequency semantic information of the current encoding time into through a high-frequency memory mechanism, to generate an updated video long-term memory c N .

[0019] By adaptively supplementing the inter-frame semantic information in the long-term memory c N into the reference feature , inter-frame prediction features with inter-frame semantic consistency are generated.

[0020] The beneficial effects of the technical solutions provided by the present application are:

[0021] 1. The present application efficiently encodes video visual features in an end-to-end manner, effectively improving the coding efficiency of VCM;

[0022] 2. The present application designs an efficient video feature coding network based on inter-frame semantic consistency, which utilizes various temporal reference information in the reference information cache to realize efficient coding of the current frame visual features.

[0023] 3. The present invention proposes a feature prediction module based on inter-frame semantic consistency, which continuously memorizes and updates the semantic information of video frames by constructing a long-short-term memory mechanism, and adaptively supplements the relevant inter-frame semantics for the reference features to generate prediction features with inter-frame semantic consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flowchart of a video feature encoding method for machine vision. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0026] 1. Constructing I-frame visual feature encoding network

[0027] For the encoding of the first frame x0 in the video sequence, we first use the backbone network DarkNet-53 to extract features of the first frame x0 and obtain multi-scale visual features. Then, the analysis transformation module Transform into a compact representation and use the entropy codec to encode and decode the compact representation to obtain the reconstructed compact representation Then, the reconstructed compact representation is transformed into Inverse transformation to reconstructed multi-scale visual features During decoding, the reconstructed compact representation As time domain reference information, it is put into the reference information cache and used as reference information for subsequent P frame encoding. The target detection result D0 is input into the visual analysis model, where the target detection result D0 includes: the detected target and target category of the current coding frame.

[0028] 2. Building a P-frame visual feature encoding network

[0029] For a P frame {x1,...,x N} encoding, with the Nth frame x N For example, we first use the backbone network DarkNet-53 to analyze the Nth frame x N Perform feature extraction to obtain its visual features Then, the efficient video feature coding network EVFC-Net for machine vision is used to Perform encoding and decoding to obtain reconstructed visual features Among them, the proposed feature prediction module based on inter-frame semantic consistency is used to generate prediction features To improve the inter-frame coding efficiency of visual features. Finally, the target detection result D of the current coding frame is obtained using a visual analysis model N The visual analysis model in the embodiment of the application refers to a feature pyramid network FPN.

[0030] III. Constructing an efficient video feature coding network based on inter-frame semantic consistency

[0031] The video multi-target tracking task needs to perform online matching on the same objects in the front and back frames. In order to ensure the time sequence continuity of the video tracking result, the visual features of the front and back frames need to have inter-frame semantic consistency. Therefore, an efficient video feature coding network EVFC-Net for machine vision is designed in the embodiment of the application. The network mainly includes:

[0032] The feature prediction module ISCFP, the analysis transform, the synthesis transform, and the entropy encoder and decoder.

[0033] First, the feature prediction module ISCFP uses the video long short-term memory {c N-1 ,h N-1} in the reference information cache and the coded visual representation of the previous frame to generate a predicted feature with inter-frame semantic consistency. In this process, the ISCFP module dynamically updates the video long short-term memory {c in the reference information cache to the reference information at the current time. Subsequently, in order to sufficiently remove the spatial and scale redundancy information in the visual features, the analysis transform is used to gradually transform the visual features with different resolutions into a joint compact visual representation. The formula is as follows:

[0034]

[0035] Wherein, conv ↓2 (·) represents a down-sampling convolution layer with a size of 5*5 and a step of 2, h(·) represents a down-sampling convolution layer with a size of 5*5 and a step of 2 and a GDN layer, represents the channel dimension concatenation of the features. The visual representation of the previous time P frame after network coding is obtained, and the visual representation is stored in the reference information cache as the reference information for coding the visual features of the current time P frame. In order to reduce the coding rate, the residual r is obtained by subtracting the predicted feature obtained by the ISCFP module, and the residual r N is finally entropy coded.

[0036] ​​Then, in the entropy codec, the r N is quantized, arithmetically encoded and arithmetically decoded in turn to generate the decoded residual , wherein, considering that the visual features of the current frame have temporal and spatial redundancy information, the entropy model provides more accurate probability estimation for the arithmetically encoding and decoding process by aggregating the temporal prior of the r and the spatial prior of the r N , thereby improving the arithmetically encoding and decoding efficiency.

[0037] Finally, the synthesis transform adopts a structure reciprocal to the analysis transform, and the reconstructed joint visual representation is obtained by adding the and the , and the reconstructed multi-scale visual features are decoded from the .

[0038] , wherein the entropy model mainly includes a hyper-prior entropy codec and a probability estimation network, the residual r N is first hyper-prior encoded, the decoded residual information and the temporal prior are combined, and finally, they are jointly sent to the probability estimation network for processing, thereby estimating the probability distribution parameters mu and sigma of the residual r N , wherein the hyper-prior encoder and the probability estimation network are both composed of a plurality of convolutional layers.

[0039] Four, constructing a feature prediction module based on inter-frame semantic consistency

[0040] In order to improve the inter-frame encoding efficiency of video features, an embodiment of the present application constructs a feature prediction module ISCFP based on inter-frame semantic consistency, uses a low-frequency forgetting mechanism to forget low-frequency semantic information such as image background, uses a high-frequency memory mechanism to remember high-frequency semantic information such as target texture, and finally generates predicted features with inter-frame semantic consistency. The input of the ISCFP module is the reference visual feature , the video long-term memory c N-1 and the video short-term memory h N-1 , and the output is the inter-frame predicted feature and the updated video long-term and short-term memory {c N , h N}. First, use DWT to perform frequency band decomposition on and h N-1 , which can be expressed as follows:

[0041]

[0042] , wherein DWT(·) represents a discrete wavelet transform, LL J and LL hrespectively represent and h N-1 low-frequency components. is high-frequency components, h N-1 high-frequency components, HL J represents the vertical edge feature of the inter prediction representation, HL h represents the vertical edge feature of the video short-term memory, LH J represents the horizontal edge feature of the inter prediction representation, LH h represents the horizontal edge feature of the video short-term memory, HH J represents the diagonal feature of the inter prediction representation, HH h represents the diagonal feature of the video short-term memory.

[0043] Subsequently, considering that the low-frequency components usually contain smooth content such as image background, the ISCFP module filters out the low-frequency semantic information in the video long-term memory c N-1 , so that c N-1 is more focused on retaining high-frequency semantic components that are beneficial to the visual task. This process can be formulated as:

[0044]

[0045] where conv ↑2 (·) represents an up-sampling convolution layer with a size of 5x5 and a step of 2, and σ(·) represents a Sigmoid function, represents the video long-term memory after forgetting the low-frequency components.

[0046] Then, in order to ensure that the video long-term memory can effectively adapt to the texture changes of the video target, the ISCFP module uses a high-frequency memory mechanism to supplement the high-frequency semantic information of the current encoding moment into c , to generate an updated video long-term memory c N . The specific formula is as follows:

[0047]

[0048] where t(·) represents a Tanh function.

[0049] Finally, by adaptively supplementing the inter-frame semantic information in the long-term memory c N into the reference feature , the inter-frame prediction feature with inter-frame semantic consistency is generated. The specific formula is as follows:

[0050]

[0051] Inter-frame prediction feature The main role is two aspects: one, by subtracting the inter-frame prediction feature And joint visual representation Residual information r N , entropy coding of residual information can effectively reduce the code rate; second, the inter-frame prediction feature As the entropy model in the time domain reference information, provide more accurate probability estimation for the arithmetic encoding and decoding process, and then improve the arithmetic coding efficiency.

[0052] Five, training machine vision-oriented video feature coding network

[0053] The rate-distortion loss function L EVFC The proposed EVFC-Net is optimized end-to-end, and the specific formula is as follows:

[0054] L EVFC =λD v +R

[0055] Wherein, R represents the encoding bit rate, λ represents the hyperparameter of controlling the code rate, D v Indicates the video multi-target tracking loss. Specifically, D v Include foreground / background classification loss L a , detection frame regression loss L β And embedding loss L r Three parts, the formula is as follows:

[0056]

[0057] Wherein, M represents the number of prediction heads in visual analysis, U=1,...,M,j=alpha,beta,gamma represents the uncertainty related to the task.

[0058] In order to meet the demand of different encoding code rate, the embodiment of the application trains 7 network models when λ={4,2,1,1 / 2,1 / 4,1 / 8,1 / 16}. In terms of network structure, according to the experimental setting specified by MPEG, the backbone network and visual analysis module of the overall coding framework of the method are consistent with JDE-1088x608 model.

[0059] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of a preferred embodiment, and the above embodiment serial numbers are only for description, not representing the advantages and disadvantages of the embodiment.

[0060] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for encoding video features for machine vision, characterized in that, The method comprises: For the first frame x0 in the video sequence, the backbone network is used to extract features from the first frame x0 to obtain multi-scale visual features. Through the analysis transformation module Transform into a compact representation, and use the entropy codec to encode and decode the compact representation to obtain the target detection result; For P frame {x1,...,x N}, the backbone network is used for feature extraction on the Nth frame x N , visual features are obtained, the machine vision-oriented efficient video feature coding network is used for encoding and decoding on the visual features, and the reconstructed visual features are obtained; using a rate-distortion loss function to perform end-to-end joint optimization on a high-efficiency video feature coding network, and performing video feature coding based on the optimized network; wherein the high-efficiency video feature coding network comprises four parts of a feature prediction module based on inter-frame semantic consistency, an analysis transform, a synthesis transform, and an entropy model. The feature prediction module based on inter-frame semantic consistency is: using video long short-term memory in the reference information cache and the coded previous frame visual representation Generating predicted features with inter-frame semantic consistency The reference information cache is dynamically updated to the referenceable information at the current time ​ The analysis transform is to have different resolutions Progressive transform into a joint compact visual representation The formula is expressed as follows: where conv ↓2 (·) denotes a down-sampling convolutional layer with size 5x5 and stride 2, h(·) denotes a down-sampling convolutional layer with size 5x5 and stride 2 and a GDN layer, denotes the concatenation of the channel dimensions of the features; The entropy model aggregates temporal priors and entropy coded residuals r N spatial priors for an arithmetic coding process; The synthetic transform adopts a structure that is inverse to the analysis transform, by and Add together to obtain the reconstructed joint visual representation and from The reconstructed multi-scale visual features are gradually decoded 2. The method of claim 1, wherein, The entropy model comprises: a hyper-prior entropy codec and a probability estimation network, wherein the residual r N is hyper-prior coded, and the decoded residual information and the inter-frame prediction feature are combined and input into the probability estimation network for processing to obtain the probability distribution parameters μ and σ of the residual r N .

3. The method of claim 1, wherein, The feature prediction module based on inter-frame semantic consistency filters low-frequency semantic information in the long-term memory c N-1 of the video through a low-frequency forgetting mechanism, so that c N-1 is more focused on retaining high-frequency semantic components that are beneficial to visual tasks; The feature prediction module based on inter-frame semantic consistency adopts a high-frequency memory mechanism to the long-term memory of the video after forgetting the low-frequency component Supplement the high-frequency semantic information of the current encoding moment to generate an updated video long-term memory c N ; by adapting the reference feature long-term memory c N inter-frame semantic information in the reference feature

Images