Disparity map generation method and device, equipment, medium and product

Abstract

The invention discloses a disparity map generation method and device, equipment, a medium and a product, and relates to the technical field of stereoscopic vision, and the method comprises the steps: carrying out the cost calculation of a two-stage feature map through a cost aggregation network, and obtaining a first image cost volume; and performing parallax regression according to the first image cost volume to obtain a parallax image. Therefore, the two-stage feature map of the input image is extracted through the feature extractor, then the cost aggregation network is adopted to carry out cost calculation on the two-stage feature map to obtain the first image cost volume, and finally parallax regression is carried out according to the obtained first image cost volume to obtain a parallax map. The problem that the disparity map cannot be quickly and accurately obtained in a traditional stereo matching algorithm is solved, and the efficiency of generating the disparity map is improved.

Description

Technical Field

[0001] The present application relates to the field of stereoscopic vision technology, and in particular to a disparity map generation method, device, equipment, medium and product. Background Art

[0002] The stereo matching algorithm is a computer vision technology used to determine the association between corresponding points in multi-view images. It is usually used in stereo vision systems. Its main principle is to estimate the disparity between them by comparing the features of corresponding pixels in the left and right images, thereby determining the distance and depth information of the object.

[0003] However, traditional stereo matching algorithms may not be able to fully extract depth information in images, resulting in poor matching performance. In addition, pathological areas such as shadows and textureless areas in images may cause errors in the matching algorithm. In existing technologies, convolutional neural networks are usually used to handle these situations. Although CNNs are developing rapidly and network performance has been greatly improved, existing CNN-based fast binocular stereo matching algorithms often cannot achieve high-precision real-time reasoning on GPUs. The main problem they face is that lightweight algorithms often sacrifice the accuracy of the model, while using high-precision algorithms sacrifices the reasoning speed of the model.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a disparity map generation method, device, equipment, medium and product, aiming to solve the technical problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm.

[0006] To achieve the above objectives, the present application proposes a disparity map generation method, the method comprising: The input image is subjected to feature extraction by a feature extractor to obtain a two-stage feature map; Performing cost calculation on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume; Perform disparity regression according to the first image cost volume to obtain a disparity map.

[0007] In one embodiment, the feature extractor includes an embedding layer and a two-dimensional convolution layer, and the step of extracting features from the input image by the feature extractor to obtain a two-stage feature map includes: Downsampling the input image through the embedding layer to obtain an initial feature atlas; Performing two-dimensional convolution on the initial feature atlas through the two-dimensional convolution layer to obtain a local feature atlas; The initial feature atlas and the local feature atlas are subjected to feature fusion to obtain a two-stage feature map.

[0008] In one embodiment, the two-stage feature map includes a left feature map and a right feature map, and the step of performing cost calculation on the two-stage feature map through a cost aggregation network to obtain a first image cost volume includes: Perform distance calculation according to the left feature map and the right feature map to obtain a second image cost volume; The second image cost volume is convolved through a cost aggregation network to obtain a first image cost.

[0009] In one embodiment, the step of performing distance calculation according to the left feature map and the right feature map to obtain the second image cost volume includes: Splitting the left feature map into a first comparison image block set; Performing geometric segmentation on the right feature map according to the first contrast image block set to obtain a second contrast image block set; Distance calculation is performed according to the first comparison image block set and the second comparison image block set to obtain a second image cost volume.

[0010] In one embodiment, the cost aggregation network includes a three-dimensional convolution layer and a multi-resolution geometric information fusion attention module, and the step of performing convolution processing on the second image cost volume through the cost aggregation network to obtain the first image cost includes: Regularizing the second image cost volume through the three-dimensional convolution layer to obtain a third image cost volume; Performing multi-head attention calculation on the third image cost to obtain key feature information; According to the key feature information, the third image cost volume and the second image cost volume are weightedly fused through the attention module to obtain a first image cost volume.

[0011] In one embodiment, the step of performing disparity regression according to the first image cost volume to obtain a disparity map includes: Performing two-dimensional cost aggregation on the first image cost volume to obtain an image disparity value; The disparity map is obtained by performing disparity regression on the disparity values ​​of the image through a neural network model.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a disparity map generating device, the device comprising: An extraction module, used for extracting features from an input image through a feature extractor to obtain a two-stage feature map; A calculation module, used for performing cost calculation on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume; The disparity regression module is used to perform disparity regression according to the first image cost volume to obtain a disparity map.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a disparity map generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the disparity map generation method as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the disparity map generation method described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the disparity map generation method as described above are implemented.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The embodiment of the present application proposes a disparity map generation method, device, equipment, storage medium and computer program product, which extracts features from an input image through a feature extractor to obtain a two-stage feature map; performs cost calculation on the two-stage feature map through a cost aggregation network to obtain a first image cost volume; and performs disparity regression based on the first image cost volume to obtain a disparity map. Thus, the two-stage feature map of the input image is extracted through a feature extractor, and then the cost aggregation network is used to perform cost calculation on the two-stage feature map to obtain a first image cost volume, and finally, the disparity regression is performed based on the obtained first image cost volume to obtain a disparity map, which solves the problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm, and improves the efficiency of disparity map generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the disparity map generation method of the present application; Figure 2 This is a schematic diagram of a stereo matching algorithm involved in the disparity map generation method of the present application; Figure 3 A schematic diagram of a process flow provided for the second embodiment of the disparity map generation method of the present application; Figure 4 This is a schematic diagram of cost calculation of feature maps involved in the disparity map generation method of the present application; Figure 5 This is a schematic diagram of multi-level information fusion involved in the disparity map generation method of the present application; Figure 6 This is a schematic diagram of the module structure of the disparity map generating device according to an embodiment of the present application; Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the disparity map generation method in the embodiment of the present application.

[0020] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: down-sampling the input image through the embedding layer to obtain an initial feature atlas; performing two-dimensional convolution on the initial feature atlas through the two-dimensional convolution layer to obtain a local feature atlas; performing feature fusion on the initial feature atlas and the local feature atlas to obtain a two-stage feature map. Distance calculation is performed according to the left feature map and the right feature map to obtain a second image cost volume; convolution processing is performed on the second image cost volume through the cost aggregation network to obtain a first image cost. The left feature map is divided into a first contrast image block set; the right feature map is geometrically divided according to the first contrast image block set to obtain a second contrast image block set; distance calculation is performed according to the first contrast image block set and the second contrast image block set to obtain a second image cost volume. Regularizing the second image cost volume through the three-dimensional convolution layer to obtain a third image cost volume; performing multi-head attention calculation on the third image cost to obtain key feature information; according to the key feature information, weighted fusion is performed on the third image cost volume and the second image cost volume through the attention module to obtain the first image cost volume. Perform two-dimensional cost aggregation on the first image cost volume to obtain an image disparity value; perform disparity regression on the image disparity value through a neural network model to obtain a disparity map. This solves the problem that the disparity map cannot be quickly and accurately obtained in the traditional stereo matching algorithm, realizes the generation of the disparity map, and improves the efficiency of disparity map generation. Based on the solution of the present invention, starting from the problem that the existing CNN-based fast binocular stereo matching algorithm often cannot achieve high-precision real-time reasoning on the GPU, thus being inefficient, a disparity map generation method is designed, and the effectiveness of the disparity map generation method of the present invention is verified when the disparity map is generated. Finally, the efficiency of disparity map generation by the method of the present invention is significantly improved.

[0024] In this embodiment, for ease of description, the following description is made by taking the disparity map generating device as the execution subject.

[0025] Since the traditional stereo matching algorithm in the existing technology may not be able to fully extract the depth information in the image, resulting in poor matching performance, in addition, pathological areas such as shadows and textureless areas in the image may cause errors in the matching algorithm. Although CNN is developing rapidly and the network performance has been greatly improved, the existing CNN-based fast binocular stereo matching algorithm often sacrifices the accuracy of the model due to the lightweight algorithm, and the high-precision algorithm sacrifices the reasoning speed of the model, which makes it impossible to achieve high-precision real-time reasoning on the GPU, which leads to a decrease in the efficiency of disparity map generation.

[0026] The present application provides a solution, which uses a feature extractor to extract features from an input image to obtain a two-stage feature map, uses a cost aggregation network to calculate the cost of the two-stage feature map to obtain the cost volume of the image, and finally performs disparity regression through the cost volume to obtain a disparity map, thereby providing users with better quality services.

[0027] From the above embodiments, it can be seen that the present application uses a feature extractor to extract features from an input image to obtain a two-stage feature map; uses a cost aggregation network to perform cost calculation on the two-stage feature map to obtain a first image cost volume; and performs disparity regression based on the first image cost volume to obtain a disparity map. Thus, the two-stage feature map of the input image is extracted by a feature extractor, and then the cost aggregation network is used to perform cost calculation on the two-stage feature map to obtain a first image cost volume, and finally, disparity regression is performed based on the obtained first image cost volume to obtain a disparity map, which solves the problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm, and improves the efficiency of disparity map generation.

[0028] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a disparity map generating device, etc. The following takes the disparity map generating device as an example to illustrate this embodiment and the following embodiments.

[0029] Based on this, the present application embodiment provides a disparity map generation method, referring to Figure 1 , Figure 1 This is a schematic diagram of the process of the first embodiment of the disparity map generation method of the present application.

[0030] In this embodiment, the disparity map generation method includes steps S01 to S03: Step S01, extracting features from an input image using a feature extractor to obtain a two-stage feature map; Before starting to explain this embodiment, it should be clear that stereo matching is a key task in computer vision. It aims to estimate the depth information of each pixel in the scene through two images taken from different perspectives (called a stereo image pair). The core goal of stereo matching is to find the corresponding point in the other image for each pixel and calculate the depth information through disparity (disparity refers to the horizontal difference in the position of the same scene point in two images). However, traditional stereo matching algorithms may not be able to fully extract the depth information in the image, resulting in poor matching performance. In addition, pathological areas such as shadows and textureless areas in the image may cause errors in the matching algorithm. In the prior art, convolutional neural networks are usually used to handle these situations. Although CNN is developing very fast and the network performance has been greatly improved, the existing fast binocular stereo matching algorithms based on CNN often cannot achieve high-precision real-time reasoning on GPUs. The main problem it faces is that lightweight algorithms often sacrifice the accuracy of the model, while the use of high-precision algorithms sacrifices the reasoning speed of the model. Stereo matching algorithms in the prior art such as Figure 2 As shown in the figure, the stereo matching algorithm is one of the key technologies for obtaining depth information from the left and right images. The basic idea is to find the corresponding points with the same name by comparing the pixels in the left and right images, and calculate the disparity between them. Specifically, the depth information is obtained by calculating the matching cost, the aggregation cost, the disparity and the optimization of the disparity.

[0031] In this embodiment, in order to achieve more accurate disparity map acquisition, the feature extractor is used to extract features from the input image to obtain a two-stage feature map. The feature extractor in this embodiment adopts an improved U-Net feature extractor, which uses a patch embedded feature extractor. The input image takes the automatic driving of a vehicle as an example, and two corresponding images are obtained as the left eye image input and the right eye image input. The two-stage feature is a technology that uses multi-stage feature extraction and processing to improve model performance in tasks such as target detection and image segmentation. In such tasks, the multi-stage processing of feature maps is to extract hierarchical and rich feature information from the image.

[0032] Step S02, performing cost calculation on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume; After obtaining the two-stage feature maps, the cost aggregation network is used to perform cost calculation to obtain the first image cost volume, where the cost volume is usually calculated automatically in a convolutional neural network (CNN) in a stereo matching method based on deep learning. A deep learning model (such as a neural network) can learn how to extract features from the original image and construct a cost volume during the learning process. It should be noted that this cost volume is no longer calculated manually, but is automatically optimized through network training.

[0033] Step S03: performing disparity regression according to the first image cost volume to obtain a disparity map.

[0034] In computer vision, disparity regression is a method that directly predicts a disparity map through a regression model. It is usually used for stereo matching problems. The disparity map represents the horizontal displacement of each pixel in the image between two views. It is usually used to represent the depth information of the scene. After obtaining the cost of the image, disparity regression can be performed based on the cost volume. The first image usually refers to the left or right image. The cost volume is a key data structure that represents the matching cost of pixels in the left and right images under different disparity assumptions. By using a deep learning model, the cost volume can be used to regress the disparity map.

[0035] Specifically, in the above step S01, the step of extracting features from the input image by a feature extractor to obtain a two-stage feature map includes: Step S011, downsampling the input image through the embedding layer to obtain an initial feature atlas; Step S012, performing two-dimensional convolution on the initial feature atlas through the two-dimensional convolution layer to obtain a local feature atlas; Step S013, performing feature fusion on the initial feature atlas and the local feature atlas to obtain a two-stage feature map.

[0036] In this embodiment, the U-Net feature extractor is improved, and a feature extractor using patch embedding is proposed. It should be clear that U-Net is a convolutional neural network (CNN) architecture commonly used in image segmentation tasks. It was originally designed for medical image segmentation. The uniqueness of U-Net lies in its symmetrical encoder-decoder structure, especially its skip connections, which can effectively extract and fuse feature information of different scales. More specifically, in U-Net, the feature extractor usually refers to the encoder part of the network, that is, the "downsampling" part, which is responsible for extracting multi-level features from the input image. In this process, the network will gradually reduce the spatial size of the image and increase the depth of the feature map, thereby extracting more and more abstract and advanced image features. In order to obtain more refined features, in this embodiment, a feature extractor using patch embedding is used, which is widely used in architectures such as vision transformer (ViT). Unlike traditional convolutional neural networks (CNN), patch embedding extracts features by dividing the image into small blocks (i.e., "patches") and linearly mapping each patch, which can achieve better and more refined feature extraction.

[0037] The specific methods of feature extractor based on patch embedding include: (1) Downsample the image through the embedding layer to generate a smaller feature map, i.e., the initial feature map set; (2) Then, three layers of 2D convolution are performed on each resolution feature map to obtain the local feature set of the initial feature map set; (3) Finally, the multi-resolution features sampled from the bilinear interpolation and convolutional layers are fused to generate the final two-stage feature map. Bilinear interpolation is a commonly used interpolation method, usually used for data point interpolation in two-dimensional space. It calculates the value of the new point by linear interpolation of known data points in the horizontal and vertical directions. Bilinear interpolation is a common method in image processing, especially for image scaling, rotation and other operations.

[0038] This improvement improves the representation ability of feature maps, enhances the capture of pixel-to-pixel correlations, and enriches contextual information.

[0039] Let’s take a specific example to illustrate: assuming that the resolution of the input image is 1024×1024, it is downsampled through the embedding layer to first obtain a 512×512 feature map, then local features are extracted through three layers of 2D convolution, and finally the upsampled 256×256 feature map is fused with the 512×512 feature map to output a high-quality feature map.

[0040] By extracting features from the input image through the improved feature extractor, the representation ability of the feature map can be improved, the capture of pixel correlations can be enhanced, and the contextual information can be enriched, thereby achieving more refined disparity map generation.

[0041] More specifically, in the above step S03, the step of performing disparity regression according to the first image cost volume to obtain a disparity map includes: Step S031, performing two-dimensional cost aggregation on the first image cost volume to obtain an image disparity value: Step S032: performing disparity regression on the image disparity values ​​through a neural network model to obtain a disparity map.

[0042] In this example, a lightweight 2D cost aggregation is used. The disparity map obtained in the first stage is enlarged to 1 / 8 resolution and compared with the left feature map with 1 / 8 resolution. Figure 1 The cost volume is constructed to form a four-dimensional 1 / 4 resolution cost volume, which is then input into a lightweight 2D cost aggregation network. This network is designed based on the U-Net structure, which reduces convolution calculations, improves inference speed, and can more effectively process low-resolution texture structures.

[0043] For example, suppose the disparity map obtained in the first stage has a resolution of 128×128. Before entering the lightweight network, it is enlarged to a resolution of 512×512 to ensure that more details can be processed in the subsequent 2D aggregation. More specifically, we should be clear about the concepts of cost volume and cost aggregation. In stereo matching, the cost volume is a three-dimensional array, in which the two dimensions represent the pixel positions in the image, and the third dimension represents the different disparity values ​​of each pixel. The cost of each pixel at different disparities is usually calculated by the grayscale difference, color difference, or other matching metrics of the pixel. The size of the cost volume is: H × W × D, where H is the height of the image, W is the width of the image, and D is the maximum possible value of the disparity (i.e., the disparity range). Cost aggregation is to improve the matching accuracy and reduce the impact of noise. On the basis of the initially calculated cost volume, the cost is smoothed, and the relationship between adjacent pixels is considered to obtain a more accurate disparity value. Aggregation methods usually utilize spatial neighborhood information, such as convolution operations, weighted averaging, etc. to avoid affecting the accuracy of the overall disparity map due to incorrect matching of a single pixel.

[0044] The two-dimensional cost aggregation of the cost volume generally refers to the spatial smoothing of the cost volume to reduce noise and improve matching accuracy. Cost aggregation usually processes the spatial dimensions of the cost volume (i.e., the height and width of the image) rather than operating on the disparity dimension. A common aggregation method is to use a sliding window (such as a convolution kernel) to smooth the cost volume. By weighted averaging the costs of adjacent pixels, the impact of local matching errors can be reduced. Bilateral filtering or convolutional neural network (CNN) can be used for cost aggregation. The convolution operation can smooth the cost volume in space to ensure that the cost value remains consistent within the neighborhood, thereby eliminating noise.

[0045] The two-dimensional aggregation process of the cost volume can be expressed as:

[0046] Among them, C(i, j, d) is the initially calculated cost volume, C′(i, j, d) is the aggregated cost, N(i, j) is the neighborhood of pixel (i, j), and w(i′, j) is a weighting function, which is usually weighted according to the similarity or distance of pixels.

[0047] Disparity map calculation: After completing the cost aggregation, the final disparity value can be obtained by selecting the minimum cost of each pixel at different disparities. The specific method is: for each pixel (i, j), select the disparity d so that the cost C′(i, j, d) is minimized:

[0048] The obtained disparity map d(i, j) is the final disparity value of each pixel in the image, representing the depth information from the pixel to the camera.

[0049] Finally, this embodiment uses disparity regression to predict the horizontal displacement of each pixel based on the pixel pairs in the image, so as to find the correspondence between the left and right images. Disparity regression is usually implemented through neural networks or other machine learning algorithms. The network model will learn to infer the optimal disparity value from the cost map. Different from the traditional discrete disparity calculation method, disparity regression can provide more delicate continuous disparity values, which helps to improve the accuracy and effect of stereo matching.

[0050] For example: For the generated cost map, the network regresses the disparity value of each pixel to accurately predict the depth of objects in the image.

[0051] This embodiment uses the above scheme to extract features from the input image through a feature extractor to obtain a two-stage feature map; the cost aggregation network is used to calculate the cost of the two-stage feature map to obtain a first image cost volume; and the disparity regression is performed based on the first image cost volume to obtain a disparity map. Thus, the two-stage feature map of the input image is extracted by the feature extractor, and then the cost aggregation network is used to calculate the cost of the two-stage feature map to obtain the first image cost volume, and finally the disparity regression is performed based on the obtained first image cost volume to obtain the disparity map, which solves the problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm, and improves the efficiency of disparity map generation.

[0052] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 In step S02, cost calculation is performed on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume. The method further includes steps S021-S022: Step S021, performing distance calculation according to the left feature map and the right feature map to obtain a second image cost volume; Step S022: performing convolution processing on the second image cost volume through a cost aggregation network to obtain a first image cost.

[0053] It should be clear that the cost volume is a data structure used for stereo matching in computer vision, and is widely used in the method of calculating the disparity map through two images of different perspectives. It is a three-dimensional array used to represent the matching cost of each pixel under different disparity values. Therefore, the two-stage feature map obtained above includes at least a left feature map and a right feature map, which are respectively used as the input left eye image and right eye image. Then, the distance is calculated through the left feature map and the right feature map to obtain the image cost volume before the two feature maps (i.e., the second image cost volume).

[0054] In order to reduce noise and mismatch, the second image cost volume needs to be smoothed. In this embodiment, cost aggregation is used to perform convolution on the second image cost volume to obtain a more refined first image cost volume.

[0055] In this embodiment, the cost volume is calculated for the obtained feature map, and the obtained initial cost volume is convolved to obtain a more refined cost volume, thereby improving the matching accuracy and obtaining a more efficient disparity map.

[0056] Specifically, in the above step S021, the step of performing distance calculation according to the left feature map and the right feature map to obtain the second image cost volume includes: Step S0211, dividing the left feature map into a first comparison image block set; Step S0212, performing geometric segmentation on the right feature map according to the first comparison image block set to obtain a second comparison image block set; Step S0213: performing distance calculation based on the first comparison image block set and the second comparison image block set to obtain a second image cost volume.

[0057] This embodiment uses a cost volume method based on grouped L1 distance to solve the problem in the cost matching stage. This method first divides the left and right images into image blocks of equal size or extracts image blocks of fixed size as the basic unit of comparison. Then, each left image block is compared with all possible right image blocks in the same row of the right image, and the L1 distance between them (that is, the sum of the absolute values ​​of the differences between corresponding elements) is calculated. Finally, these L1 distances are filled into the corresponding positions of the cost volume as cost values ​​to construct a complete cost volume.

[0058] This not only retains the multi-channel matching information, but also reduces the impact of noise on matching, and improves the accuracy and robustness of depth estimation. The L1 distance cost volume is defined as the distance between two vectors, which can be expressed as: .

[0059] like Figure 4 As shown in the figure, it is assumed that the left and right images are divided into 8×8 image blocks respectively. For each left image block, the L1 distance between it and all possible blocks of the right image is calculated, and the minimum L1 distance is used as the cost value to fill in the cost volume.

[0060] More specifically, in the above step S022, the step of performing convolution processing on the second image cost volume through the cost aggregation network to obtain the first image cost includes: Step S0221, regularizing the second image cost volume through the three-dimensional convolution layer to obtain a third image cost volume; Step S0222, performing multi-head attention calculation on the third image cost to obtain key feature information; Step S0223: According to the key feature information, weighted fusion is performed on the third image cost volume and the second image cost volume through the attention module to obtain a first image cost volume.

[0061] This embodiment proposes a multi-scale cost aggregation network to regularize the cost volume, thereby improving the matching accuracy and reliability of the disparity map, as shown in Table 1 below. First, a large-core 3D convolution layer is used to increase the number of channels and enhance the network's expressiveness to better handle complex matching relationships and patterns and reduce the risk of overfitting. Then, a three-layer residual 3D convolution is used for further regularization, and a separable multi-head attention module is used to extract key feature information. Finally, a 3D convolution is used to compress the number of channels and improve network efficiency.

[0062] Table 1

[0063] This embodiment uses a residual 3D convolutional layer and a separable multi-head attention module to perform multi-scale cost aggregation as shown in Table 1 above, to improve the model accuracy and enhance the ability to integrate information of different scales. By cascading 1 / 4 and 1 / 8 resolution cost volumes, this paper effectively combines details with global information, thereby improving the clarity and reasoning accuracy of the disparity map.

[0064] In this embodiment, multi-branch multi-head attention is proposed, and the specific steps are as follows: (1) Multi-head attention in parallax dimension: ① Channel division: divide the number of channels into C, C and 1 channels respectively , and ; ② Input branch compression: First, , and The branches are compressed in the disparity dimension to reduce computational cost; ③ Calculate the attention score of the parallax dimension: conduct Operation, get the attention score of the parallax dimension:

[0065] ④ Generate disparity context vector: and After element-by-element multiplication, sum it over the disparity dimension to get the context vector:

[0066] ⑤ Weighted V branch: The result after ReLU activation and the context vector Multiply element-wise to weight V:

[0067] (2) Channel dimension multi-head attention: ① Calculate the channel dimension attention score: conduct Operation to get the attention score of the channel dimension:

[0068] ② Generate channel context vector: and After element-by-element multiplication, sum it up in the channel dimension to get the context vector:

[0069] ③Weighting Branch: go through The result after activation and the context vector are After element-wise multiplication we get:

[0070] ④Integrate parallax and channel attention mechanism:

[0071] This embodiment proposes a multi-resolution geometric information fusion attention module. This module integrates multi-scale feature maps and cost volumes to achieve multi-level information fusion, such as Figure 5 As shown in the figure, first, it interacts the geometric information of the feature map from the left with the cost volume, so that the model can more effectively focus on the key information in the feature map, thereby improving the prediction accuracy. Then, the module uses a multi-branch multi-head attention mechanism to further optimize the feature attention and improve the model performance. Finally, in order to ensure the stability of the training process, the module downsamples the low-resolution attention cost volume, splices it with the high-resolution cost volume, and prevents the gradient from exploding or disappearing through the residual connection.

[0072] For example, for the input cost volume, multi-head attention calculation is first performed in the disparity dimension and channel dimension respectively. After extracting useful features, it is processed through residual convolution, and then aggregated using the attention module, and finally a clear disparity map is output.

[0073] This embodiment adopts the above scheme, specifically by calculating the distance according to the left feature map and the right feature map, to obtain the second image cost volume; and by convolution processing the second image cost volume through the cost aggregation network, to obtain the first image cost. Thus, the two-stage feature map of the input image is extracted by the feature extractor, and then the cost aggregation network is used to calculate the cost of the two-stage feature map to obtain the first image cost volume, and finally the disparity regression is performed according to the obtained first image cost volume to obtain the disparity map, which solves the problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm, and improves the efficiency of disparity map generation.

[0074] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the disparity map generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0075] This application also provides a disparity map generation device, please refer to Figure 6 , the disparity map generating device comprises: An extraction module 10 is used to extract features from an input image through a feature extractor to obtain a two-stage feature map; A calculation module 20, configured to perform cost calculation on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume; The disparity regression module 30 is configured to perform disparity regression according to the first image cost volume to obtain a disparity map.

[0076] The disparity map generation device provided by the present application adopts the disparity map generation method in the above embodiment, which can solve the technical problem that the disparity map cannot be quickly and accurately obtained in the traditional stereo matching algorithm. Compared with the prior art, the beneficial effects of the disparity map generation device provided by the present application are the same as the beneficial effects of the disparity map generation method provided by the above embodiment, and the other technical features in the disparity map generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0077] The present application provides a disparity map generating device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the disparity map generating method in the above-mentioned embodiment 1.

[0078] Reference below Figure 7 , which shows a schematic diagram of the structure of a disparity map generating device suitable for implementing the embodiment of the present application. The disparity map generating device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The disparity map generating device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 7 As shown, the disparity map generating device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the disparity map generating device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the disparity map generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a disparity map generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0080] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0081] The disparity map generation device provided by the present application adopts the disparity map generation method in the above embodiment, which can solve the technical problem that the disparity map cannot be quickly and accurately obtained in the traditional stereo matching algorithm. Compared with the prior art, the beneficial effects of the disparity map generation device provided by the present application are the same as the beneficial effects of the disparity map generation method provided by the above embodiment, and the other technical features in the disparity map generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0082] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0083] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0084] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the disparity map generation method in the above-mentioned embodiment.

[0085] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0086] The computer-readable storage medium may be included in the disparity map generating device; or may exist independently without being assembled into the disparity map generating device.

[0087] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the disparity map generating device, the disparity map generating device: extracts features from the input image through a feature extractor to obtain a two-stage feature map; calculates the cost of the two-stage feature map through a cost aggregation network to obtain a first image cost volume; and performs disparity regression based on the first image cost volume to obtain a disparity map.

[0088] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0090] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0091] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned disparity map generation method, and can solve the technical problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the disparity map generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0092] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned disparity map generation method when executed by a processor.

[0093] The computer program product provided by the present application can solve the technical problem that the disparity map cannot be obtained quickly and accurately in the traditional stereo matching algorithm. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the disparity map generation method provided by the above embodiment, which will not be repeated here.

[0094] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A disparity map generation method, characterized in that: The method comprises: The input image is subjected to feature extraction by a feature extractor to obtain a two-stage feature map; Performing cost calculation on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume; Perform disparity regression according to the first image cost volume to obtain a disparity map.

2. The method according to claim 1, characterized in that The feature extractor includes an embedding layer and a two-dimensional convolution layer. The step of extracting features from the input image by the feature extractor to obtain a two-stage feature map includes: Downsampling the input image through the embedding layer to obtain an initial feature atlas; Performing two-dimensional convolution on the initial feature atlas through the two-dimensional convolution layer to obtain a local feature atlas; Feature fusion is performed on the initial feature atlas and the local feature atlas to obtain a two-stage feature map.

3. The method according to claim 1, characterized in that The two-stage feature map includes a left feature map and a right feature map, and the step of performing cost calculation on the two-stage feature map through a cost aggregation network to obtain a first image cost volume includes: Perform distance calculation according to the left feature map and the right feature map to obtain a second image cost volume; The second image cost volume is convolved through a cost aggregation network to obtain a first image cost.

4. The method according to claim 3, characterized in that The step of performing distance calculation according to the left feature map and the right feature map to obtain a second image cost volume comprises: Splitting the left feature map into a first comparison image block set; Performing geometric segmentation on the right feature map according to the first contrast image block set to obtain a second contrast image block set; Distance calculation is performed according to the first comparison image block set and the second comparison image block set to obtain a second image cost volume.

5. The method according to claim 3, characterized in that The cost aggregation network includes a three-dimensional convolution layer and a multi-resolution geometric information fusion attention module. The step of performing convolution processing on the second image cost volume through the cost aggregation network to obtain the first image cost includes: Regularizing the second image cost volume through the three-dimensional convolution layer to obtain a third image cost volume; Performing multi-head attention calculation on the third image cost to obtain key feature information; According to the key feature information, the third image cost volume and the second image cost volume are weightedly fused through the attention module to obtain a first image cost volume.

6. The method according to claim 1, characterized in that The step of performing disparity regression according to the first image cost volume to obtain a disparity map comprises: Performing two-dimensional cost aggregation on the first image cost volume to obtain an image disparity value; The disparity map is obtained by performing disparity regression on the disparity values ​​of the image through a neural network model.

7. A disparity map generating device, characterized in that: The device comprises: An extraction module, used for extracting features from an input image through a feature extractor to obtain a two-stage feature map; A calculation module, used for performing cost calculation on the two-stage feature maps through a cost aggregation network to obtain a first image cost volume; The disparity regression module is used to perform disparity regression according to the first image cost volume to obtain a disparity map.

8. A disparity map generating device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the disparity map generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the disparity map generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the disparity map generation method according to any one of claims 1 to 6 are implemented.

Images