Disparity map generation method and device based on image multi-scale features

Abstract

The embodiment of the present application discloses a disparity map generation method and device based on image multi-scale features, which can be used in the financial field or other technical fields, and the method comprises the following steps: inputting a left image and a right image into a backbone feature extraction module respectively to obtain multi-scale feature maps corresponding to the left image and the right image respectively; generating a multi-scale matching cost volume according to the multi-scale feature maps corresponding to the left image and the right image respectively; fusing the multi-scale matching cost volume from the smallest scale to the upper level to obtain an initial matching cost volume; superimposing the left image and the right image in the channel to obtain a matching cost volume, then inputting the matching cost volume into a smoothing feature extraction module to obtain a smoothing feature map; fusing the smoothing feature map and the initial matching cost volume to obtain a fusion cost volume; and generating a disparity map according to the fusion cost volume. The present application helps to improve the accuracy of disparity prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a disparity map generation method and device based on image multi-scale features. BACKGROUND

[0002] In computer vision, it is crucial to obtain accurate depth information of a scene. Depth perception based on laser radar is costly and not easy to deploy, and three-dimensional reconstruction based on structured light relies heavily on the environment and is difficult to be widely applied. The stereo matching technology has high flexibility and low cost in obtaining depth information, and is more widely used in practical products. It has been applied in the fields of autonomous driving, face recognition, three-dimensional reconstruction, etc.

[0003] At present, stereo matching algorithms are mainly divided into two categories: deep learning algorithms and traditional algorithms. Traditional algorithms have high computing speed and can be used on most devices, but they rely only on the disparity between different viewpoints for calculation, resulting in low accuracy. The top algorithms on the public stereo matching algorithm ranking list are almost all based on deep learning methods. Compared with traditional algorithms, their accuracy and speed have been greatly improved, but they are highly dependent on hardware resources and sample size.

[0004] Although binocular stereo vision has been widely used in practical applications, and there are many effective algorithms in academia and industry, there are still many uncontrollable factors that affect the accuracy of stereo matching algorithms, such as occlusion between different objects, weak texture regions, and discontinuous disparity regions. How to more effectively use image feature information to improve the accuracy of disparity prediction is a technical problem that needs to be solved in the prior art. SUMMARY

[0005] In order to solve at least one of the technical problems in the above background art, the present application proposes a disparity map generation method and device based on image multi-scale features.

[0006] In order to achieve the above purpose, according to one aspect of the present application, a disparity map generation method based on image multi-scale features is provided, which comprises:

[0007] inputting a left image and a right image into a backbone feature extraction module to obtain multi-scale feature maps corresponding to the left image and the right image respectively;

[0008] generating a multi-scale matching cost volume according to the multi-scale feature maps corresponding to the left image and the right image respectively;

[0009] fusing the multi-scale matching cost volume from the smallest scale to the upper level to obtain an initial matching cost volume;

[0010] The left image and the right image are channel superimposed to obtain a matching cost volume, and then the matching cost volume is input into a smooth feature extraction module to obtain a smooth feature map, wherein the smooth feature extraction module is used to extract bottom information in an image.

[0011] The smooth feature map is fused with the initial matching cost volume to obtain a fused cost volume.

[0012] A disparity map is generated according to the fused cost volume.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a disparity map generation device based on image multi-scale features is provided, which comprises:

[0014] A backbone feature extraction unit is configured to input a left image and a right image into a backbone feature extraction module to obtain multi-scale feature maps corresponding to the left image and the right image respectively, wherein the multi-scale feature maps specifically comprise a feature map corresponding to each scale;

[0015] A multi-scale matching cost volume generation unit is configured to generate a multi-scale matching cost volume according to the multi-scale feature maps corresponding to the left image and the right image respectively;

[0016] An initial matching cost volume generation unit is configured to fuse the multi-scale matching cost volume from a smallest scale to a largest scale to obtain an initial matching cost volume;

[0017] A smooth feature extraction unit is configured to perform channel superposition on the left image and the right image to obtain a matching cost volume, and then input the matching cost volume into a smooth feature extraction module to obtain a smooth feature map, wherein the smooth feature extraction module is used to extract bottom information in an image;

[0018] A fused cost volume generation unit is configured to fuse the smooth feature map with the initial matching cost volume to obtain a fused cost volume;

[0019] A disparity map generation unit is configured to generate a disparity map according to the fused cost volume.

[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer device is also provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned disparity map generation method based on image multi-scale features when executing the computer program.

[0021] In order to achieve the above object, according to another aspect of the present application, there is further provided a computer readable storage medium having stored thereon computer programs / instructions which, when executed by a processor, implement the steps of the above-mentioned disparity map generation method based on multi-scale features of images.

[0022] In order to achieve the above object, according to another aspect of the present application, there is further provided a computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the above-mentioned disparity map generation method based on multi-scale features of images.

[0023] The present application has the following beneficial effects:

[0024] The present application is directed to the problem that weak texture and special structure regions are difficult to match in a stereo matching algorithm, and innovatively proposes a multi-scale feature extraction module, uses a backbone feature extraction module to extract more feature information, and calculates left and right feature maps to form a multi-scale matching cost volume, fuses the cost volumes of deep and shallow layers to obtain rich semantic information, and uses a smoothing feature extraction module to obtain low-level structure information to correct mis-matching values, so as to achieve the effect of smoothing disparity, thereby achieving higher matching precision and generating a more fine disparity map. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor. In the drawings:

[0026] Figure 1 is a flowchart of the disparity map generation method based on multi-scale features of images according to an embodiment of the present application;

[0027] Figure 2 is a flowchart of the generation of a multi-scale matching cost volume according to an embodiment of the present application;

[0028] Figure 3 is a flowchart of the smoothing of features according to an embodiment of the present application;

[0029] Figure 4 is a flowchart of the generation of a disparity map according to a fused cost volume according to an embodiment of the present application;

[0030] Figure 5 is a model structure diagram of an end-to-end stereo matching network based on multi-scale information according to an embodiment of the present application;

[0031] Figure 6 is a structure diagram of a residual module ResBlock according to an embodiment of the present application;

[0032] Figure 7 is a structural schematic diagram of a feature extraction module of the present application;

[0033] Figure 8 is a structural schematic diagram of a feature extraction module of the present application;

[0034] Figure 9 is a structural block diagram of a disparity map generation device based on image multi-scale features according to an embodiment of the present application;

[0035] Figure 10 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the personnel in the technical field better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0037] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0038] It should be noted that the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0039] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0040] It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of national laws and regulations.

[0041] It should be noted that the disparity map generation method and device based on image multi-scale features can be used in the financial field, and can also be used in any field other than the financial field, and the application field of the disparity map generation method and device based on image multi-scale features is not limited.

[0042] The application proposes a stereo matching algorithm to improve the accuracy of disparity prediction, and the application mainly comprises four parts of a main feature extraction module (Pyramidal Feature Hierarchy, PFH), a pyramid matching cost volume construction module, a feature extraction module (Featurized Pyramid Single, FPS), and a disparity estimation module. Figure 5 As shown in the overall structure of the network.

[0043] The application first inputs the original RGB left and right images into the weight-shared initial feature extraction module, learns the feature information used to calculate the stereo matching cost, that is, obtains the initial feature representation of the output image, and in this application, a PFH (Pyramidal Feature Hierarchy) backbone feature extraction module based on a residual structure is proposed, which can extract multi-level global context information. By calculating the L1 distance of each disparity level of the corresponding pixels in the left and right images, the feature maps of the left and right images are connected to form a 4D (width x height x disparity x channel number) multi-scale matching cost volume, which is called a pyramid matching cost volume (PCV), and then the matching cost volumes of multiple scales are fused upward to form an initial matching cost volume. Then the FPS (Featurized Pyramid Single) smoothing feature extraction module is proposed, which is used to extract the bottom layer information in the left and right images, and then the feature representation is fused with the initial matching cost, which can correct the initial error matching on the one hand, and expand the original pixel-level features on the other hand, so that the matching cost of the entire network utilizes more rich context information to smooth the disparity. Then 3D convolution and interlayer skip connection are used to fuse the bottom layer fine features and high layer abstract features to realize the process of cost aggregation. Finally, disparity regression is performed to obtain the final disparity map.

[0044] Figure 1 The application embodiment based on image multi-scale feature disparity map generation method is shown in the flowchart as Figure 1 In an embodiment of the application, the disparity map generation method based on image multi-scale features comprises steps S101 to S106.

[0045] In step S101, the left image and the right image are input into the backbone feature extraction module to obtain the multi-scale feature maps corresponding to the left image and the right image respectively.

[0046] Step S102, generating a multi-scale matching cost volume according to the multi-scale feature maps corresponding to the left image and the right image respectively.

[0047] Step S103, fusing the multi-scale matching cost volume from the smallest scale to the largest scale to obtain an initial matching cost volume.

[0048] Step S104, performing channel superposition on the left image and the right image to obtain a matching cost volume, and then inputting the matching cost volume into a smoothing feature extraction module to obtain a smoothing feature map, wherein the smoothing feature extraction module is used to extract bottom information in an image.

[0049] Step S105, fusing the smoothing feature map and the initial matching cost volume to obtain a fused cost volume.

[0050] Step S106, generating a disparity map according to the fused cost volume.

[0051] In an embodiment of the present application, the backbone feature extraction module in step S101 described above comprises: a plurality of combination modules connected in sequence; each of the combination modules comprises: a residual module and a pooling layer; and each of the combination modules is used to output a feature map of one scale.

[0052] Figure 6 is a structural schematic diagram of the residual module ResBlock of the present application. In deep learning, with the increase of the network layers, the feature information contained in the feature map will be reduced layer by layer, in order to make the feature information contained in the deep network more than that in the shallow layer, the present application uses the residual neural network Res-Net, introduces the front information, reduces the gradient disappearance, and improves the accuracy of the algorithm. The residual network is composed of a series of residual modules. A residual module can be represented as:

[0053] y l =h(x l )+F(x l ,W l )

[0054] x l+1 =f(y l )

[0055] The residual module can be divided into a direct mapping part and a residual part. h(x l ) is the direct mapping, F(x l ,W l ) is the residual part; x l is the input of the lth residual unit; y l is the output of the lth residual unit; W l is the parameter in the convolution operation, and f(.) is the activation function.

[0056] In order to solve the problem that special areas such as weak texture and repetitive texture in the image to be matched are prone to cause matching errors, the application proposes a PFH backbone feature extraction module based on a residual module, which provides more abundant context information for subsequent processing, and secondly, only the different parts need to be learned and output in the residual module, thereby simplifying the learning difficulty of the feature extraction module. For the left and right images input into the network, a parameter-shared PFH backbone feature extraction module is respectively arranged. The structure of the backbone feature extraction module is as shown in Figure 7 .

[0057] Generally, in a stereo matching network, the output features of the last layer of the network are used to calculate the matching cost after feature extraction, thereby ignoring the semantic information contained in other low-level networks. Since the residual network extracts features in a down-sampling manner, multiple down-samplings will cause serious information loss in the extracted features. Therefore, in the entire feature extraction process, the application can obtain multiple feature maps of different sizes through the residual module and the pooling layer, for example, 4 feature maps of different sizes, whose sizes are 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the size of the input image, respectively, to gradually construct four-scale features. And the residual edges across multiple layers are used to fuse the shallow features into the high-level features, and an output is generated at each scale from the second scale.

[0058] In an embodiment of the application, the plurality of combination modules specifically include a first combination module, a second combination module, a third combination module and a fourth combination module; the input of the first combination module is an input image, and the input image is the left image or the right image; the input of the second combination module is the fusion of the feature map output by the first combination module and the input image; the input of the third combination module is the feature map output by the second combination module; the input of the fourth combination module is the feature map output by the third combination module; and the multi-scale feature map output by the backbone feature extraction module is specifically the different-scale feature maps output by the second combination module, the third combination module and the fourth combination module.

[0059] As Figure 7As shown, in a specific embodiment of the present invention, the network structure of the backbone feature extraction module in step S101 mainly consists of residual modules and pooling. Here, the structure composed of every two residual modules and MaxPool2d is referred to as a combined module. Assuming the size of the left and right images is (3, H, W), they are input into the network. They first undergo a 3×3 convolution and a downsampling operation of size 2×2 with a stride of 2 in the first combined module to obtain an output feature map of size (32, H / 2, W / 2). This feature map is concatenated with the input image and used as the input of the second combined module. The above process is repeated to obtain a feature map of size (64, H / 4, W / 4) from the output of the second combined module. This output is used as the input of the third combined module and as the first output out1 of the entire PFH backbone feature extraction module. The third combined module obtains an output of size (128, H / 8, W / 8), which is used as the input of the last combined module and as the second output out2 of the PFH module. After convolution and pooling operations in the last combining module, a feature map with a spatial resolution of (32, H / 16, W / 16) is obtained, which is used as the third output out3 of the PFH module. The entire backbone feature extraction module generates a total of 3 matching cost outputs (i.e., feature maps) at different scales, thus enabling the extraction of object features at different scales in the image.

[0060] Let I be the input reference image (i.e., the left image) for the PFH backbone feature extraction module. l Target image (i.e., right image) I r The feature pyramids constructed after the backbone feature extraction module are represented as follows: and Where S represents the number of scales and s represents the index number of the scale. In this PFH module, a feature pyramid with three scales is constructed, i.e., S=3, which are features with resolutions of {1 / 4, 1 / 8, 1 / 16}.

[0061] like Figure 2 As shown, in one embodiment of the present invention, the step S102 of generating a multi-scale matching cost body based on the multi-scale feature maps corresponding to the left image and the right image respectively specifically includes steps S201 and S202.

[0062] Step S201: For each scale, the features of the left image and the right image are linked together based on the L1 distance between the image features of the left image and the right image to form a matching cost body corresponding to each scale.

[0063] Step S202: Obtain the multi-scale matching cost body according to the matching cost body corresponding to each scale.

[0064] In this invention, the matching cost body stores the matching degree between pixels from the reference image and corresponding pixels in the target image, containing data representing the matching cost value. Most stereo matching algorithms construct the complete matching cost body at a single scale, which is computationally expensive and results in the loss of some feature information. In contrast, the algorithm proposed in this invention, after passing through the PFH backbone feature extraction module, constructs partial matching costs at multiple scale levels by calculating L1 distance. Then, it fuses the cost bodies from each scale layer by layer from the smallest scale to the largest scale, thus forming a complete matching cost body. This structure is called a pyramid matching cost body. Figure 5 The part within the dashed box. To improve the real-time performance of the algorithm, this invention employs a distance metric, calculating the L1 distance between the left feature map and its corresponding right feature map at each disparity level, thereby forming a 4D cost volume.

[0065] As can be seen from the above embodiments, the feature maps generated after the left and right input images pass through the initial feature extraction module are: and Then, by calculating the L1 distance between the left and right image features at corresponding scales, the left and right features are linked together to form a multi-scale matching cost volume. The process of constructing the multi-scale matching cost volume is as follows:

[0066]

[0067]

[0068] In the formula: denoted as the matching cost at position (x,y) when the disparity is d; s represents the nth scale, with values ​​ranging from s∈[1,2,3], corresponding to resolutions {1 / 4, 1 / 8, 1 / 16} respectively; d represents the possible disparity search range; p represents any pixel in image I.

[0069] In one embodiment of the present invention, in step S103 above, the present invention targets the multi-scale matching cost body, starting from the smallest scale and fusing it step by step upward at different scales through convolution and upsampling operations to obtain the final cost body. The process can be represented as follows:

[0070]

[0071] In the formula: It is the matching cost body after cross-scale cost aggregation; f s It is a general function that can achieve adaptive combination of matching cost volumes at each scale, consisting of transposed convolution and upsampling.

[0072] like Figure 3As shown, in one embodiment of the present application, the inputting the channel superposition map into the smooth feature extraction module in the step S104 to obtain a smooth feature map specifically includes steps S301 to S304.

[0073] In step S301, the channel superposition map is down-sampled by a combination module to obtain a first feature map, wherein the combination module includes a residual module and a pooling layer.

[0074] In step S302, the first feature map is up-sampled by the residual module to obtain a second feature map.

[0075] In step S303, the first feature map and the second feature map are fused to obtain a third feature map.

[0076] In step S304, the third feature map is used to generate the smooth feature map.

[0077] In order to obtain more accurate disparity values of the image, a second multi-scale feature extraction module FPS smooth feature extraction module of the present application is proposed to extract the bottom layer information in the image pair and fuse it with the initial matching cost value to correct the error matching in the initial matching, so as to achieve the effect of smoothing the disparity, and the network structure is as shown in Figure 8

[0078] In one embodiment of the present application, the network of the smooth feature extraction module of the present application is mainly composed of a residual module and a MaxPool2d pooling layer. For the left and right input pictures of the channel superposition, first, the combination of two residual modules and MaxPool2d is used to obtain a feature map with a resolution of (16, H / 2, W / 2) from an input image with a size of (6, H, W), the above process is repeatedly executed to obtain a feature map with a spatial resolution of (24, H / 4, W / 4), and the convolution and pooling operations are repeated again to obtain a feature map with a size of (32, H / 8, W / 8), and then the up-sampling operation is performed again after the residual module, and the deconvolution operation with a size of 2x2 and a step of 2 is used in the up-sampling to expand the resolution of the feature map to (16, H / 4, W / 4), the result obtained after the up-sampling is subjected to the concat operation with the feature map with a resolution of (24, H / 4, W / 4) in the down-sampling process to increase the detailed features. Finally, two ResBlock modules are used to obtain the output with a shape of (16, H / 4, W / 4).

[0079] ​It can be seen that the FPS module and the PFH module are both feature extraction modules, but they have different structures. The FPS is similar to a top-down feature fusion manner, and the shallow structure has a higher resolution, which can better extract features of small targets in a complex scene by using high resolution. The specific operation is to first perform traditional top-down feature convolution, then fuse the feature map corresponding to the last scale picture and the feature map corresponding to the picture of the last scale, and use up-sampling operation and cross-layer fusion to make the output have bottom visual information and high-level semantic information. The low-level feature map is not rich in semantics and needs to interact with the deep feature, so the lateral connection and the top-down connection are combined to obtain a feature containing rich information. The PFH module is to obtain a feature pyramid of different scales by gradually down-sampling the original image. The large-scale feature map corresponds to the global feature of the image, and the small-scale is applied to the small feature in the image. Each scale corresponds to an output, so the features of each scale can contain rich semantic information, and the deeper the network, the more features the feature map has. The application adopts three scale feature maps, and finally corresponds to three different scale cost outputs.

[0080] In an embodiment of the application, the step S105 of fusing the smoothed feature map and the initial matching cost volume to obtain a fused cost volume comprises the following steps:

[0081] The size of the smoothed feature map and the initial matching cost volume is first adjusted to be consistent, and then the fused cost volume is obtained by fusing.

[0082] As shown in the figure, Figure 4 In an embodiment of the application, the step S106 of generating a disparity map according to the fused cost volume comprises steps S401 and S402.

[0083] In step S401, the fused cost volume is regularized by 3D convolution to obtain a regularized cost volume.

[0084] In step S402, a disparity regression operation is performed according to the regularized cost volume to generate the disparity map.

[0085] In an embodiment of the application, the step S401 of regularizing the fused cost volume by 3D convolution comprises the following steps: the fused cost volume is regularized in three dimensions by 3D convolution with four convolution kernels of 3x3x3.

[0086] Generally, in the stereo matching network based on deep learning, after constructing the matching cost volume, the network learns a function to regularize the matching cost volume. Most algorithms use multiple 3D convolution and 3D deconvolution to form a coding-decoding structure to perform the cost aggregation step, but 3D convolution and 3D deconvolution are high in calculation cost. Considering the real-time performance of the algorithm in practical application, the algorithm does not use a complex 3D convolution coding-decoding structure, but simply uses 3D convolution to regularize the matching cost volume, thereby reducing the calculation overhead. In this application, only four 3D convolution kernels with a size of 3*3*3 are used to regularize in three dimensions to generate the final normalized cost volume, which reduces the calculation time at the expense of a certain degree of accuracy loss. The entire cost aggregation process is trained in an end-to-end manner. After the network regularizes the matching cost volume, the disparity regression operation is performed to predict the disparity map.

[0087] In the end-to-end method, after cost regularization, the matching cost volume is still a 4D or 5D structure. The loss of deep learning is usually compared between the predicted value and the true disparity value, and then the weights are continuously corrected to obtain the optimal parameters. The application uses a softargmin function that is differentiable and can regress disparity estimation to calculate the optimal disparity of each pixel point to obtain the final disparity map.

[0088] First, the predicted matching cost value c d is taken as a negative number to convert the matching cost value into a probability representation. If the matching cost is higher, the probability value of the true disparity value is lower. Then, the probability value of each disparity is calculated on the probability volume in the disparity dimension using the softmax operation σ(x), and the expression is:

[0089]

[0090] In the formula, i is the i-th disparity map in the matching cost volume; v i represents the disparity value of the pixel point in the i-th disparity map; D max is the maximum disparity value.

[0091] After the above formula, the matching cost volume is converted into a probability representation, and the probability that the predicted disparity is the true value is σ(-c d ), that is, the softmax is used to normalize the probability value in the disparity dimension. Finally, each candidate disparity d is weighted and summed according to the normalized probability, and the result is taken as the final predicted disparity value of the network model. In summary, the mathematical model of disparity regression is:

[0092]

[0093] Since the softargmin disparity regression method is used in the application to obtain the final predicted disparity from the matching cost volume, in order to continuously optimize the model, the application compares the predicted value with the true disparity value provided in the official platform of the data set, calculates the loss value of the network, updates the parameters in the network through the back propagation of the neural network model, and until the training of the entire network model is completed. The application selects the Smooth L1 loss which is not sensitive to outliers, which is widely used in the boundary box regression of target detection, because it has robustness and low sensitivity to outliers, and the gradient change is relatively small, and can solve the problem of function roughness. The mathematical expression is shown in the following formula. Through continuous training of the network, the loss value gradually decreases until it tends to a relatively stable state.

[0094]

[0095]

[0096] In the formula, N is the total number of pixel points in the image; d i is the true disparity map; is the network predicted disparity value; i represents a certain pixel point.

[0097] The algorithm proposed in the application trains the entire network end-to-end from randomly initialized parameters, performs supervised learning using binocular images, and trains and tests on three data sets of ScenceFlow, KITTI2015 and KITTI2012, and evaluates using the evaluation indicators of each data set. Ablation experiments are performed on ScenceFlow and KITTI, and the results of the algorithm proposed in the application are compared with other stereo matching algorithms on the KITTI ranking list, and the effectiveness of the algorithm in stereo matching is obtained.

[0098] The application analyzes and researches some problems existing in the binocular stereo matching task and makes improvements, and the main advantages are as follows:

[0099] 1. The algorithm is a deep learning algorithm, which solves the problem that the traditional stereo matching algorithm only relies on the constraint conditions between different viewpoints and is difficult to work in weak texture and repetitive texture areas.

[0100] 2. In view of the problem that weak texture and special structure regions are difficult to match in a stereo matching algorithm, the application innovatively proposes a multi-scale feature extraction module, uses a PFH backbone feature extraction module to extract more feature information, and calculates left and right feature maps to form a multi-scale matching cost volume, fuses the cost volumes of deep and shallow layers to obtain rich semantic information, and uses an FPS smoothing feature extraction module to obtain low-level structure information to correct the mismatching value, thereby achieving the effect of smoothing the disparity.

[0101] 3. The algorithm discards the step of using multiple 3D convolution and 3D deconvolution to form a coding and decoding structure for cost aggregation in most algorithms, has a relatively simple network structure, reduces the complexity of the algorithm, and has a certain real-time performance.

[0102] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0103] Based on the same inventive concept, the embodiments of the application also provide a disparity map generation device based on image multi-scale features, which can be used to implement the disparity map generation method based on image multi-scale features described in the above embodiments, as described in the following embodiments. Since the principle of solving problems of the disparity map generation device based on image multi-scale features is similar to that of the disparity map generation method based on image multi-scale features, the embodiments of the disparity map generation device based on image multi-scale features can refer to the embodiments of the disparity map generation method based on image multi-scale features, and the repeated parts will not be described herein. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is conceived.

[0104] Figure 9 is a structural block diagram of the disparity map generation device based on image multi-scale features of the embodiments of the application, as shown in Figure 9 In one embodiment of the application, the disparity map generation device based on image multi-scale features of the application comprises:

[0105] The backbone feature extraction unit 1 is configured to input the left image and the right image into the backbone feature extraction module respectively to obtain the multi-scale feature maps corresponding to the left image and the right image respectively, wherein the multi-scale feature maps specifically comprise: a feature map corresponding to each scale.

[0106] a multi-scale matching cost volume generating unit 2 configured to generate a multi-scale matching cost volume according to multi-scale feature maps corresponding to the left image and the right image respectively;

[0107] an initial matching cost volume generating unit 3 configured to fuse the multi-scale matching cost volume from a smallest scale to an upper level to obtain an initial matching cost volume;

[0108] a smoothing feature extraction unit 4 configured to perform channel superposition on the left image and the right image to obtain a matching cost volume, and then input the matching cost volume into a smoothing feature extraction module to obtain a smoothing feature map, wherein the smoothing feature extraction module is configured to extract bottom information in an image;

[0109] a fused cost volume generating unit 5 configured to fuse the smoothing feature map and the initial matching cost volume to obtain a fused cost volume;

[0110] a disparity map generating unit 6 configured to generate a disparity map according to the fused cost volume.

[0111] In an embodiment of the present application, the multi-scale matching cost volume generating unit 2 specifically comprises:

[0112] an image feature association module configured to associate features of the left image and the right image according to L1 distance of image features of the left image and the right image to form a matching cost volume corresponding to each scale respectively;

[0113] a first generating module configured to obtain the multi-scale matching cost volume according to the matching cost volume corresponding to each scale respectively.

[0114] In an embodiment of the present application, the smoothing feature extraction unit 4 specifically comprises:

[0115] a down-sampling operation module configured to perform down-sampling operation on the channel superposition map through a combination module to obtain a first feature map, wherein the combination module comprises a residual module and a pooling layer;

[0116] an up-sampling operation module configured to perform up-sampling operation on the first feature map through a residual module to obtain a second feature map;

[0117] a fusion module configured to fuse the first feature map and the second feature map to obtain a third feature map;

[0118] a second generating module configured to generate the smoothing feature map according to the third feature map.

[0119] In an embodiment of the present application, the fusion cost volume generating unit 5 is specifically configured to first adjust the sizes of the smoothed feature map and the initial matching cost volume to be consistent, and then perform fusion to obtain the fusion cost volume.

[0120] In an embodiment of the present application, the disparity map generating unit 6 specifically comprises:

[0121] The regularization module is configured to regularize the fusion cost volume by using 3D convolution to obtain a regularized cost volume.

[0122] The disparity regression operation module is configured to perform disparity regression operation according to the regularized cost volume to generate the disparity map.

[0123] In an embodiment of the present application, the regularization module is specifically configured to regularize the fusion cost volume in three dimensions by using 3D convolution with four convolution kernels of 3x3x3.

[0124] In order to achieve the above-mentioned purposes, according to another aspect of the present application, a computer device is also provided. As shown in the accompanying drawings, the computer device comprises a memory, a processor, a communication interface and a communication bus, and a computer program executable on the processor is stored on the memory, and the processor implements the steps in the above-mentioned embodiment method when executing the computer program. Figure 10

[0125] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or combinations thereof.

[0126] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above-mentioned method embodiments of the present application. The processor performs various functional applications and work data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the method in the above-mentioned method embodiments.

[0127] ​The memory can include a program storage area and a data storage area. The program storage area can store an operating system and applications required by at least one function. The data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some embodiments, the memory can optionally include a memory that is remotely located with respect to the processor, which can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The one or more units are stored in the memory and, when executed by the processor, perform the method in the above embodiments.

[0129] The above computer device specific details can be understood in correspondence with the relevant description and effects in the above embodiments, which will not be described here again.

[0130] To achieve the above object, according to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program, when executed in a computer processor, implements the steps of the above disparity map generation method based on image multi-scale features. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0131] To achieve the above object, according to another aspect of the present application, a computer program product is provided, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of the above disparity map generation method based on image multi-scale features are implemented.

[0132] It is apparent that those skilled in the art should understand that the modules or steps of the present application described above can be realized by using general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by using program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, or they can be respectively manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the present application is not limited to any specific combination of hardware and software.

[0133] The preferred embodiments of the present application are described above, but the present application is not limited to the above. Various modifications and changes can be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a disparity map based on image multi-scale features, characterized in that, The method comprises the following steps: inputting a left image and a right image into a backbone feature extraction module to obtain a plurality of scale feature maps corresponding to the left image and the right image respectively; generating a multi-scale matching cost volume according to the plurality of scale feature maps corresponding to the left image and the right image respectively; fusing the multi-scale matching cost volume from the smallest scale to the largest scale to obtain an initial matching cost volume; stacking the left image and the right image in the channel to obtain a matching cost volume, and then inputting the matching cost volume into a smoothing feature extraction module to obtain a smoothing feature map, wherein the smoothing feature extraction module is used to extract the bottom information in the image; fusing the smoothing feature map and the initial matching cost volume to obtain a fused cost volume; generating a disparity map according to the fused cost volume. 2.The method of claim 1, wherein, The backbone feature extraction module comprises a plurality of combination modules connected in sequence; each combination module comprises a residual module and a pooling layer; and each combination module is used to output a feature map of a scale. 3.The method of claim 2, wherein, The plurality of combination modules specifically comprise a first combination module, a second combination module, a third combination module and a fourth combination module; the input of the first combination module is an input image, and the input image is the left image or the right image; the input of the second combination module is the fusion of the feature map output by the first combination module and the input image; the input of the third combination module is the feature map output by the second combination module; the input of the fourth combination module is the feature map output by the third combination module; and the multi-scale feature maps output by the backbone feature extraction module are specifically different scale feature maps output by the second combination module, the third combination module and the fourth combination module. 4.The method of claim 1, wherein, The method of generating a multi-scale matching cost volume according to the plurality of scale feature maps corresponding to the left image and the right image respectively specifically comprises the following steps: for each scale, the features of the left image and the right image are connected according to the L1 distance of the image features of the left image and the right image to form a matching cost volume corresponding to each scale respectively; the multi-scale matching cost volume is obtained according to the matching cost volume corresponding to each scale respectively. 5.The method of claim 1, wherein, The method of inputting the matching cost volume into a smoothing feature extraction module to obtain a smoothing feature map specifically comprises the following steps: performing a down-sampling operation on the channel stacked image through a combination module to obtain a first feature map, wherein the combination module comprises a residual module and a pooling layer; performing an up-sampling operation on the first feature map through a residual module to obtain a second feature map; fusing the first feature map and the second feature map to obtain a third feature map; generating the smoothing feature map according to the third feature map. 6.The method of claim 1, wherein, The method of fusing the smoothing feature map and the initial matching cost volume to obtain a fused cost volume specifically comprises the following steps: firstly, adjusting the sizes of the smoothing feature map and the initial matching cost volume to be consistent, and then fusing them to obtain the fused cost volume. 7.The method of claim 1, wherein, The method of generating a disparity map according to the fused cost volume specifically comprises the following steps: regularizing the fused cost volume by using 3D convolution to obtain a regularized cost volume; Disparity regression is performed according to the regularized cost volume to generate the disparity map.

8. The method of claim 7, wherein, The fusion cost volume is regularized by using 3D convolution, and specifically comprising: The fusion cost volume is regularized in three dimensions by using 3D convolution with four convolution kernels of 3*3*3.

9. An apparatus for generating a disparity map based on image multi-scale features, the apparatus comprising: Comprising: The backbone feature extraction unit is configured to input the left image and the right image into a backbone feature extraction module to obtain a plurality of scale feature maps corresponding to the left image and the right image respectively, wherein each scale feature map comprises a feature map corresponding to each scale; The multi-scale matching cost volume generation unit is configured to generate a multi-scale matching cost volume according to the plurality of scale feature maps corresponding to the left image and the right image respectively; The initial matching cost volume generation unit is configured to fuse the multi-scale matching cost volume from the smallest scale to the largest scale to obtain an initial matching cost volume; The smoothing feature extraction unit is configured to stack channels of the left image and the right image to obtain a matching cost volume, and then input the matching cost volume into a smoothing feature extraction module to obtain a smoothing feature map, wherein the smoothing feature extraction module is configured to extract low-level information in the image; The fusion cost volume generation unit is configured to fuse the smoothing feature map and the initial matching cost volume to obtain a fusion cost volume; The disparity map generation unit is configured to generate a disparity map according to the fusion cost volume.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method in any one of claims 1 to 8.

12. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the method in any one of claims 1 to 8.

Images