Three-dimensional image coding method for machine vision

Abstract

The invention discloses a machine vision-oriented three-dimensional image coding method, which comprises the following steps of: performing feature extraction on input left and right views by using a three-dimensional image feature extraction module to obtain three-dimensional image multi-scale features containing 3D scene information; a stereo image multi-scale feature coding module is used for coding the stereo image multi-scale features to obtain compact left and right viewpoint potential representations, and stereo entropy coding is carried out on the left and right viewpoint potential representations to obtain code streams for transmission; performing stereo entropy decoding on the transmitted code stream to obtain decoded left and right viewpoint potential representations, and decoding the left and right viewpoint potential representations at a decoding end by using a stereo image multi-scale feature decoding module to obtain reconstructed stereo image multi-scale features; and performing visual analysis on the reconstructed stereo image multi-scale features by using a 3D visual analysis module to obtain a 3D target detection result. According to the invention, the 3D visual task performance and the stereo image compression efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of stereoscopic image coding, and in particular to a stereoscopic image coding method oriented to machine vision. Background Art

[0002] Stereoscopic images provide viewers with images from different viewpoints, allowing users to have an immersive 3D visual experience. With the development of stereoscopic image technology and deep learning technology, a large number of intelligent applications such as self-driving cars, smart cities, and smart homes have been widely promoted to gradually replace traditional human-driven solutions. In order to achieve a variety of 3D visual tasks, these intelligent applications usually require efficient processing and analysis of massive stereoscopic image data, which brings great challenges to the transmission and storage of stereoscopic image data. Therefore, there is an urgent need to develop efficient stereoscopic image coding methods for 3D visual tasks.

[0003] In recent years, video coding methods for machine vision have been preliminarily studied. Existing methods usually adopt a compression-first-analysis mode or an analysis-first-compression mode to improve the coding efficiency for machine vision by efficiently encoding visual semantic information. However, existing methods have only made useful explorations in video coding technology for 2D visual tasks, and there are currently few studies exploring stereo image coding for 3D visual tasks. Unlike planar images, stereo images contain content from multiple viewpoints of dynamic scenes, and have the characteristics of high sparsity, high dimensionality, and large data volume. Therefore, in stereo image coding for 3D visual tasks, it is necessary to consider not only the semantic self-similarity of single viewpoint images, but also the semantic similarity between different viewpoints to further improve compression efficiency and 3D visual task performance. Since existing video coding methods for 2D visual tasks do not explicitly explore the correlation between viewpoints, it is difficult to achieve satisfactory performance by directly using these methods to solve the stereo image coding problem for 3D visual tasks.

[0004] In order to solve the above problems, a direct approach is to migrate the existing stereo image coding method from human vision optimization to machine vision optimization. However, this human vision optimization stereo image coding method mainly focuses on the pixel-level fidelity of stereo images rather than the semantic-level fidelity, so it is difficult to achieve good performance on 3D vision tasks. Summary of the invention

[0005] Existing machine vision coding methods are mainly designed for 2D vision tasks, and there is currently little work exploring stereo image coding methods for 3D vision tasks. In order to improve the performance of 3D vision tasks and the compression efficiency of stereo images, the present invention proposes a stereo image visual feature coding network, which effectively reduces the stereo image coding bit rate while maintaining the performance of 3D vision tasks. In addition, in the stereo image visual feature coding network, a stereo image multi-scale feature coding module is proposed, which gradually transforms the sparse stereo image multi-scale features into a compact joint visual representation by simultaneously removing the information redundancy within the visual feature viewpoint, between viewpoints, and across scales. See the following description for details:

[0006] A stereoscopic image encoding method for machine vision, the method comprising:

[0007] The stereo image feature extraction module is used to extract features from the input left and right views to obtain multi-scale features of the stereo image containing 3D scene information;

[0008] Using a stereo image multi-scale feature encoding module to encode the multi-scale features of the stereo image, to obtain compact left and right viewpoint potential representations, and performing stereo entropy encoding on the left and right viewpoint potential representations to obtain a code stream for transmission;

[0009] The transmitted bitstream is stereo entropy decoded to obtain the decoded potential representation of the left and right viewpoints. The decoding end uses the stereo image multi-scale feature decoding module to decode the potential representation of the left and right viewpoints to obtain the reconstructed multi-scale features of the stereo image.

[0010] Use the 3D visual analysis module to perform visual analysis on the multi-scale features of the reconstructed stereo image to obtain 3D target detection results.

[0011] Wherein, the stereoscopic image multi-scale feature encoding module is:

[0012] The low-level stereo image features Input to the stereo encoding unit to obtain a compact representation

[0013] On the encoding side and Perform joint compression to generate a joint compact representation of low-level stereo image features and mid-level stereo image features

[0014]

[0015] in, The channel dimension cascade of the representation feature; the high-level stereo image features and Merge to obtain a joint compact representation of multi-scale features of stereo images

[0016] Wherein, the stereoscopic image multi-scale feature decoding module is:

[0017] Code stream Perform arithmetic decoding to obtain the reconstructed joint compact representation and reconstruct the joint compact representation from Decode the reconstructed high-level stereo image features and the joint compact representation

[0018] Using stereo decoding unit from joint compact representation Decode the reconstructed mid-level stereo image features and compact representation

[0019] Using stereo decoding unit from compact representation Decode the reconstructed low-level stereo image features

[0020] The method further includes: using a rate-distortion loss function to perform end-to-end joint optimization on the proposed stereoscopic image coding network for machine vision.

[0021] Wherein, the stereo image coding network for machine vision is:

[0022] Using rate-distortion loss function L MVSFC The proposed stereo image coding network for machine vision is end-to-end jointly optimized. MVSFC It includes a distortion term and a bit rate consumption term, and takes the 3D visual task loss as the distortion term. The visual task loss D v Includes: classification loss L based on Focal loss cls , regression loss L based on smooth L1 norm reg And the disparity loss L based on stereo focal loss dis Three parts:

[0023] D v =L cls +L reg +L dis .

[0024] Finally, the rate-distortion loss function L MVSFC The calculation formula is:

[0025] L MVSFC =λD v +R R +R L,

[0026] Among them, R R and R L denote the bit rate consumption of the right view and the left view respectively, and λ is a hyperparameter that controls the balance between bit rate and 3D vision task performance.

[0027] The beneficial effects of the technical solution provided by the present invention are:

[0028] 1. The present invention proposes a stereoscopic image coding method for machine vision, which efficiently encodes the visual features of stereoscopic images in an end-to-end manner, effectively improving the performance of 3D vision tasks and the efficiency of stereoscopic image compression;

[0029] 2. The present invention proposes a stereoscopic image visual feature coding network, which effectively improves the stereoscopic image coding efficiency for 3D vision tasks by extracting, compressing and transmitting compact visual features in a 3D vision task-driven manner;

[0030] 3. The present invention proposes a stereoscopic image multi-scale feature coding module, which effectively improves the 3D vision task performance and feature coding efficiency by removing information redundancy within the visual feature viewpoint, between viewpoints and across scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flowchart of a stereo image encoding method for machine vision. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0033] To overcome the shortcomings of the prior art, an embodiment of the present invention provides a stereoscopic image encoding method for machine vision, and the specific steps are as follows:

[0034] 1. Constructing a Stereo Image Visual Feature Encoding Network

[0035] For the stereo image visual feature encoding network, the stereo image feature extraction module is first used to extract features from the input left and right views to obtain the stereo image multi-scale features containing 3D scene information. The backbone network of the stereo image feature extraction module is ResNet-34, and the stereo image multi-scale features are derived from the first, second, and third stages of ResNet-34. Generally speaking, the high-resolution features extracted from the shallow convolutional layers of the network model contain more original image detail information including object textures and contours, while the low-resolution features obtained from the deep convolutional layers contain more semantic and contextual information.

[0036] Therefore, in order to ensure the performance of 3D vision tasks, the embodiment of the present invention compresses and transmits the stereo image multi-scale features from different stages of ResNet-34. After the stereo image feature extraction module, the stereo image multi-scale features are efficiently encoded using the stereo image multi-scale feature encoding module by modeling the complex correlation between the stereo image multi-scale features to obtain a compact left and right viewpoint potential representation. Finally, stereo entropy coding is performed on the potential representations of the left and right views to obtain a bitstream for transmission.

[0037] 2. Constructing a Stereo Image Visual Feature Decoding Network

[0038] For the stereo image visual feature decoding network, the transmitted bitstream is firstly stereo entropy decoded to obtain the decoded left and right viewpoint potential representations Then, at the decoding end, the stereo image multi-scale feature decoding module is used to decode the potential representation of the left and right viewpoints. Perform efficient decoding to obtain multi-scale features of reconstructed stereo images Finally, the 3D visual analysis module is used to perform visual analysis on the multi-scale features of the reconstructed stereo image to obtain the 3D target detection result P Task The detection results can be used for high-level path planning and decision-making in autonomous driving scenarios. Specifically, the detection results can be used to identify the location and movement trend of obstacles and adjust the driving path. At the same time, priority strategies (for example, avoiding pedestrians) can be formulated based on the type and location of the target.

[0039] 3. Constructing a Stereo Image Multi-Scale Feature Coding Module

[0040] Stereoscopic image multi-scale features contain less information entropy while having higher-dimensional data representation, so effectively removing complex redundant information between sparse features is of great significance for realizing stereoscopic machine vision coding. In view of this, an embodiment of the present invention proposes a stereoscopic image multi-scale feature coding module, which improves the efficiency of stereoscopic image multi-scale feature coding by simultaneously removing information redundancy within visual feature viewpoints, between viewpoints, and across scales.

[0041] In the stereo image multi-scale feature encoding module, the low-level stereo image features are first Input to the stereo encoding unit to eliminate the intra-viewpoint redundancy and inter-viewpoint redundancy of stereo image features and obtain a compact representation

[0042]

[0043] Among them, SEU(·) represents the stereo encoding unit, which consists of two convolutional layers and a bidirectional context transformation module. Considering that there is cross-scale redundancy between the features of stereo images of different scales, and Perform joint compression to generate a joint compact representation of low-level stereo image features and mid-level stereo image features

[0044]

[0045] in, The channel dimension of the feature is cascaded. Then, the high-level stereo image features are and Merge to obtain a joint compact representation of multi-scale features of stereo images Compared with compressing the stereo image features of each scale separately, the joint compact representation generated by joint coding has less cross-scale redundancy and thus has higher coding efficiency. Finally, the arithmetic encoder in the entropy codec is used to convert Encode into a code stream.

[0046] Compared with encoding multi-scale features of stereo images separately In contrast, the stereo image multi-scale feature encoding module gradually transforms the sparse stereo image multi-scale features into a compact joint representation by constructing bidirectional dependencies between viewpoints and hierarchical progressive relationships between scales. The intra-viewpoint, inter-viewpoint and cross-scale redundancy of the multi-scale features of stereo images is effectively reduced.

[0047] 4. Constructing a stereo image multi-scale feature decoding module

[0048] In the stereo image multi-scale feature decoding module, the bitstream is first Perform arithmetic decoding to obtain the reconstructed joint compact representation and reconstruct the joint compact representation from Decode the reconstructed high-level stereo image features and the joint compact representation The specific formula is as follows:

[0049]

[0050] Among them, Split(·) represents the split in the channel dimension, and SDU(·) represents the stereo decoding unit, which consists of two deconvolution layers and a bidirectional context transform module.

[0051] Then, the stereo decoding unit is used to extract the joint compact representation Decode the reconstructed mid-level stereo image features and compact representation The specific formula is as follows:

[0052]

[0053] Finally, the stereo decoding unit is used to extract Decode the reconstructed low-level stereo image features The specific formula is as follows:

[0054]

[0055] 5. Training a Stereo Image Coding Network for Machine Vision

[0056] The embodiment of the present invention uses the rate-distortion loss function L MVSFC The proposed stereo image coding network for machine vision is end-to-end jointly optimized. MVSFC It includes a distortion term and a bit rate consumption term. In order to achieve the best stereo image machine vision coding efficiency, the 3D vision task loss is used as a distortion term to ensure that the reconstructed multi-scale stereo image features are more suitable for subsequent visual tasks.

[0057] Specifically, the visual task loss D v Includes: classification loss L based on Focal loss cls , regression loss L based on smooth L1 norm reg And the disparity loss L based on stereo focal loss dis Three parts:

[0058] D v =L cls +L reg +L sis .

[0059] Finally, the rate-distortion loss function L MVSFC The calculation formula is:

[0060] L MVSFC =λD v +R R +R L ,

[0061] Among them, R R and R L denote the bit rate consumption of the right view and the left view respectively, and λ is a hyperparameter that controls the balance between bit rate and 3D vision task performance.

[0062] 6. Encode the stereo image based on the trained stereo image coding network for machine vision.

[0063] The embodiments of the present invention realize stereoscopic image coding for machine vision, and effectively remove information redundancy within viewpoints, between viewpoints, and across scales in the visual features of stereoscopic images while ensuring the performance of 3D vision tasks.

[0064] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A stereo image coding method for machine vision, characterized in that: The method comprises: The stereo image feature extraction module is used to extract features from the input left and right views to obtain multi-scale features of the stereo image containing 3D scene information; Using a stereo image multi-scale feature encoding module to encode the multi-scale features of the stereo image, to obtain compact left and right viewpoint potential representations, and performing stereo entropy encoding on the left and right viewpoint potential representations to obtain a code stream for transmission; The transmitted bitstream is stereo entropy decoded to obtain the decoded potential representation of the left and right viewpoints. The decoding end uses the stereo image multi-scale feature decoding module to decode the potential representation of the left and right viewpoints to obtain the reconstructed multi-scale features of the stereo image. Use the 3D visual analysis module to perform visual analysis on the multi-scale features of the reconstructed stereo image to obtain 3D target detection results.

2. The method for machine vision-oriented stereoscopic image encoding according to claim 1, characterized in that: The stereo image multi-scale feature encoding module is: The low-level stereo image features Input to the stereo encoding unit to obtain a compact representation On the encoding side and Perform joint compression to generate a joint compact representation of low-level stereo image features and mid-level stereo image features in, The channel dimension cascade of the representation feature; the high-level stereo image features and Merge to obtain a joint compact representation of multi-scale features of stereo images 3. The method for machine vision-oriented stereoscopic image encoding according to claim 1, characterized in that: The stereo image multi-scale feature decoding module is: Code stream Perform arithmetic decoding to obtain the reconstructed joint compact representation and reconstruct the joint compact representation from Decode the reconstructed high-level stereo image features and the joint compact representation Using stereo decoding unit from joint compact representation Decode the reconstructed mid-level stereo image features and compact representation Using stereo decoding unit from compact representation Decode the reconstructed low-level stereo image features 4. The method for machine vision-oriented stereoscopic image encoding according to claim 1, characterized in that: The method also includes: using a rate-distortion loss function to perform end-to-end joint optimization on the proposed stereoscopic image coding network for machine vision.

5. The method for machine vision-oriented stereoscopic image encoding according to claim 1, characterized in that: The stereo image coding network for machine vision is: Using rate-distortion loss function L MVSFC The proposed stereo image coding network for machine vision is end-to-end jointly optimized. MVSFC It includes a distortion term and a bit rate consumption term, and takes the 3D visual task loss as the distortion term. The visual task loss D v Including: classification loss L based on Focal loss cls , regression loss L based on smooth L1 norm reg And the disparity loss L based on stereo focal loss dis Three parts: D v =L cls +L reg +L dis . Finally, the rate-distortion loss function L MVSFC The calculation formula is: L MVSFC =λD v +R R +R L , Among them, R R and R L denote the bit rate consumption of the right view and the left view respectively, and λ is a hyperparameter that controls the balance between bit rate and 3D vision task performance.

Images