Binocular stereo matching method, device and storage medium

Abstract

The application discloses a binocular stereo matching method and device and a storage medium, and the method comprises the following steps: acquiring a first view and a second view; inputting the first view and the second view into two feature extraction modules respectively to obtain a first feature map and a second feature map; inputting the first feature map and the second feature map into an effective attention module to obtain a second large-scale reorganization map, and the effective attention module is used for reallocating the content weight of the second feature map according to the first feature map; constructing a hybrid cost volume according to the first feature map and the second large-scale reorganization map; and inputting the hybrid cost volume into a disparity regression module to obtain a disparity map. According to the application, the effective attention module is arranged, the content weight of the second feature map is reallocated according to the first feature map, the hybrid cost volume obtained according to the first feature map and the second large-scale reorganization map is high in accuracy, and the accuracy of the disparity map can be improved.

Description

Technical Field

[0001] This invention relates to the field of image data processing, and in particular to a binocular stereo matching method, device, and storage medium. Background Technology

[0002] Binocular stereo matching is a computer vision task, also known as binocular disparity estimation or binocular depth estimation. It typically uses a cost volume constructed from left and right feature maps to obtain a disparity map or depth map, and then estimates disparity and depth. It is widely used in fields such as 3D reconstruction, autonomous driving, and robot navigation.

[0003] In related technologies, stereo matching involves extracting corresponding feature maps from the left and right images respectively, and constructing a cost volume based on the left and right feature maps. During the construction process, due to the mismatch between some regions between the two views, the accuracy of the constructed cost volume can be easily affected, resulting in insufficient accuracy of the obtained disparity map. Summary of the Invention

[0004] The present invention aims to at least solve one of the technical problems existing in the prior art. To this end, the present invention provides a binocular stereo matching method, device, and storage medium, which can improve the accuracy of the obtained stereo and the accuracy of the obtained disparity map.

[0005] A first aspect of the present invention provides a binocular stereo matching method, comprising the following steps:

[0006] Get the first view and the second view;

[0007] The first view and the second view are respectively input into two feature extraction modules to obtain the first feature map and the second feature map;

[0008] The first feature map and the second feature map are input into the effective attention module to obtain the second large-scale reconstructed map. The effective attention module is used to reallocate the content weights of the second feature map according to the first feature map.

[0009] Based on the first feature map and the second large-scale reconstruction map, a hybrid cost body is constructed;

[0010] Input the hybrid cost volume into the disparity regression module to obtain the disparity map.

[0011] According to the above embodiments of the present invention, at least the following beneficial effects are achieved: by setting an effective attention module, the content weights of the second feature map are redistributed according to the first feature map, so that useful information of the second feature map is emphasized and useless information is suppressed, thereby improving the similarity between the obtained second large-scale reconstructed map and the first feature map. That is, the second large-scale reconstructed map can focus on displaying the image information corresponding to each region of the first feature map, thereby improving the matching performance of the two. The accuracy of the hybrid cost body constructed based on the first feature map and the second large-scale reconstructed map is high, thereby improving the accuracy of the obtained disparity map.

[0012] According to some embodiments of the first aspect of the present invention, a first view and a second view are respectively input into two feature extraction modules to obtain a first feature map and a second feature map, including:

[0013] The first view is input into the first feature extraction module to obtain the first initial feature map at multiple scales;

[0014] The second view is input into the second feature extraction module to obtain a second initial feature map at multiple scales;

[0015] Each initial feature map is sampled to the same scale and then fused to obtain the first feature map;

[0016] Each second initial feature map is sampled to the same scale and then fused to obtain the second feature map.

[0017] According to some embodiments of the first aspect of the present invention, each first initial feature map is sampled to the same scale and then fused to obtain a first feature map, including:

[0018] Each initial feature map is sampled at two different scales to obtain two sets of first sampled maps at different scales, wherein each first sampled map in the first sampled map set has the same scale.

[0019] The two sets of first sampled images are fused to obtain a first large-scale feature map and a first small-scale feature map. Both the first large-scale feature map and the first small-scale feature map are first feature maps, and the scale of the first large-scale feature map is larger than that of the first small-scale feature map.

[0020] According to some embodiments of the first aspect of the present invention, each second initial feature map is sampled to the same scale and then fused to obtain a second feature map, including:

[0021] Each second initial feature map is sampled at two different scales to obtain two sets of second sampled map groups at different scales, wherein each second sampled map in the second sampled map group has the same scale;

[0022] The two sets of second sampling maps are fused to obtain the second large-scale feature map and the second small-scale feature map. Both the second large-scale feature map and the second small-scale feature map are second feature maps, and the scale of the second large-scale feature map is larger than that of the second small-scale feature map.

[0023] According to some embodiments of the first aspect of the present invention, the scale of the first large-scale feature map is the same as the scale of the second large-scale feature map, and the scale of the first small-scale feature map is the same as the scale of the second small-scale feature map.

[0024] The first and second feature maps are input into the effective attention module to obtain the second large-scale reconstructed map, including:

[0025] The first and second small-scale feature maps are input into the first effective attention module to obtain the second small-scale reconstructed map.

[0026] The first small-scale feature map and the second small-scale reconstructed map are merged and then input into the thinning module for feature thinning to obtain the feature thinning map.

[0027] The refined feature map and the second-largest-scale feature map are input into the warping module to obtain the warped feature map;

[0028] The first large-scale feature map and the distorted feature map are input into the second effective attention module to obtain the second large-scale reconstructed map.

[0029] According to some embodiments of the first aspect of the present invention, inputting a first small-scale feature map and a second small-scale feature map into a first effective attention module to obtain a second small-scale reconstructed map includes: inputting the first small-scale feature map and the second small-scale feature map into the first effective attention module, and multiplying the first small-scale feature map by the second small-scale feature map after performing max pooling, two-dimensional convolution and activation sequentially through the first effective attention module to obtain a second small-scale reconstructed map;

[0030] The first large-scale feature map and the distorted feature map are input into the second effective attention module to obtain the second large-scale reconstructed map. This includes: inputting the first large-scale feature map and the distorted feature map into the second effective attention module, and then multiplying the first large-scale feature map by the second effective attention module after performing max pooling, two-dimensional convolution and activation on the first large-scale feature map in sequence, so as to obtain the second large-scale reconstructed map.

[0031] According to some embodiments of the first aspect of the present invention, a hybrid cost body is constructed based on a first feature map and a second large-scale reconstruction map, comprising:

[0032] Based on the first feature map and the second large-scale reconstruction map, subtraction-type cost bodies, group-related cost bodies, and connection-merging cost bodies are constructed respectively.

[0033] The subtractive cost body, the grouped cost body, and the link-merging cost body are fused to obtain the hybrid cost body.

[0034] According to some embodiments of the first aspect of the present invention, a hybrid cost volume is input into a disparity regression module to obtain a disparity map, including:

[0035] Input the hybrid cost volume into the aggregation module to obtain the cost aggregation result;

[0036] Input the cost aggregation result into the disparity regression module to obtain the disparity map.

[0037] A second aspect of the present invention provides an electronic device, comprising:

[0038] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the binocular stereo matching method of any one of the first aspects.

[0039] Since the electronic device of the second aspect embodiment applies the binocular stereo matching method of any one of the first aspects, it has all the beneficial effects of the first aspect of the present invention.

[0040] According to a third aspect of the present invention, a computer storage medium is provided, which stores computer-executable instructions for executing the binocular stereo matching method of any one of the first aspects.

[0041] Since the computer storage medium of the third aspect embodiment can perform the binocular stereo matching method of any one of the first aspects, it has all the beneficial effects of the first aspect of the present invention.

[0042] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0043] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0044] Figure 1 This is a diagram showing the main steps of the binocular stereo matching method according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating the working principle of the binocular stereo matching method according to an embodiment of the present invention;

[0046] Figure 3 yes Figure 2 A schematic diagram illustrating the working principle of the multi-scale feature extraction and fusion module;

[0047] Figure 4 yes Figure 2 A schematic diagram illustrating the working principle of the effective attention module in the middle;

[0048] Figure 5 yes Figure 2 A schematic diagram illustrating the working principle of the construction process of the hybrid cost body. Detailed Implementation

[0049] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly. Those skilled in the art can reasonably determine the specific meaning of these terms in this invention based on the specific content of the technical solution. In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. Furthermore, features specified as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0050] Stereo matching is essentially a depth estimation method, belonging to computer vision tasks. Also known as disparity estimation or binocular depth estimation, it typically uses a cost volume constructed from left and right feature maps to estimate disparity and depth. It is widely used in 3D reconstruction, autonomous driving, and robot navigation. It usually obtains the disparity by calculating the total cost from the input left and right images, and then calculates the depth D using the formula D = B × f / d. That is, the depth D can be calculated from the baseline B, the focal length f, and the estimated disparity d, where the baseline B refers to the distance between the input left and right images.

[0051] On the one hand, since deep learning was introduced into the field of machine vision, many vision tasks have achieved excellent results. Significant progress has been made in image classification, object detection, and object recognition. Taking PSMNet, FADNet, and StereoNet as examples, PSMNet represents a major breakthrough by integrating global contextual information into the construction of the cost volume to address the ill-posed region problem. FADNet achieves this through correlation layers based on two-dimensional convolutions and maintains faster computation speed with a multi-scale weight training strategy. StereoNet uses a low-resolution cost volume to accelerate runtime while employing an upsampling function with edge sensing capabilities to preserve edge details. On the other hand, some learning-based vision tasks, such as instance segmentation, scene segmentation, and image super-resolution, also perform well by utilizing attention algorithms commonly used in natural language processing to focus on regions of interest. Binocular stereo matching is no exception. For example, MCANet uses it to refine parallax, and NLCANet uses it to leverage global contextual information.

[0052] In related technologies, stereo matching involves extracting corresponding feature maps from the left and right images respectively, and constructing a cost volume based on the left and right feature maps. During the construction process, the inappropriate regions between the two views affect the accuracy of the constructed cost volume, resulting in insufficient accuracy of the obtained disparity.

[0053] When convolutional neural networks and attention algorithms are introduced into binocular stereo matching, the resulting disparity maps still suffer from insufficient accuracy.

[0054] The following reference Figures 1 to 5 The binocular stereo matching method, apparatus, and storage medium of the present invention improve the accuracy of the obtained cost volume and the accuracy of the obtained disparity map by improving the matching of the two feature maps used to construct the cost volume.

[0055] refer to Figure 1 As shown, a binocular stereo matching method according to a first aspect embodiment of the present invention includes at least the following steps:

[0056] S100. Obtain the first view and the second view, wherein the first view and the second view are respectively the left image and the right image, or the right image and the left image;

[0057] S200: Input the first view and the second view into two feature extraction modules respectively to obtain the first feature map and the second feature map;

[0058] S300. Input the first feature map and the second feature map into the effective attention module to obtain the second large-scale reconstructed map. The effective attention module is used to redistribute the content weights of the second feature map according to the first feature map to obtain the second large-scale reconstructed map.

[0059] S400. Construct a hybrid cost body based on the first feature map and the second large-scale reconstruction map;

[0060] S500: Input the hybrid cost volume into the disparity regression module to obtain the disparity map.

[0061] By setting an effective attention module, the content weights of the second feature map are redistributed according to the first feature map, emphasizing the useful information of the second feature map and suppressing the useless information. This improves the similarity between the obtained second-scale reconstructed map and the first feature map, allowing the second-scale reconstructed map to highlight the image information corresponding to each region of the first feature map. This improves the matching performance between the obtained second-scale reconstructed map and the first feature map. The accuracy of the hybrid cost body constructed based on the first feature map and the second-scale reconstructed map is high, which in turn improves the accuracy of the obtained disparity map.

[0062] In related technologies, convolutional neural networks and attention algorithms have been introduced into binocular stereo matching. Although these methods have achieved convincing performance in terms of accuracy or efficiency, they still have some unsuitable areas, such as surface reflection, thin structure, pattern repetition, texture loss and region occlusion. Existing binocular stereo matching methods do not acquire complete information, making it difficult for them to meet the requirements of high-precision applications.

[0063] To mitigate the impact of the aforementioned issues, it is understood that step S200 involves inputting the first view and the second view into two feature extraction modules respectively to obtain a first feature map and a second feature map, including:

[0064] S210. Input the first view into the first feature extraction module to obtain a first initial feature map at multiple scales;

[0065] S220. Input the second view into the second feature extraction module to obtain a second initial feature map at multiple scales;

[0066] S230. After sampling each initial feature map to the same scale, they are fused to obtain the first feature map;

[0067] S240. After sampling each second initial feature map to the same scale, they are fused to obtain the second feature map.

[0068] It is understandable that both the first feature extraction module and the second feature extraction module are MobileNetV2 feature extraction modules, which are lightweight. They construct corresponding fusion modules through upsampling and downsampling of the U-Net network to implement the fusion operations in steps S230 and S240.

[0069] It is understandable that step S230 specifically involves: sampling each first initial feature map to the same scale, fusing them, and then convolving them to obtain the first feature map. Step S240 specifically involves: sampling each second initial feature map to the same scale, fusing them, and then convolving them to obtain the second feature map. The convolution operation effectively improves the performance of the obtained first and second feature maps, thereby increasing the reliability of subsequent steps.

[0070] It is understandable that S230, after sampling each initial feature map to the same scale, fuses them to obtain the first feature map, including:

[0071] S231. Each first initial feature map is sampled to two different scales to obtain two sets of first sampled map groups with different scales, wherein each first sampled map in the same first sampled map group has the same scale.

[0072] S232. The two sets of first sampling images are fused to obtain a first large-scale feature map and a first small-scale feature map. Both the first large-scale feature map and the first small-scale feature map are first feature maps. The scale of the first large-scale feature map is larger than the scale of the first small-scale feature map. During fusion, all first sampling images in the same set of first sampling images are fused.

[0073] It is understandable that each second initial feature map is sampled to the same scale and then fused to obtain a second feature map, including:

[0074] S241. Each second initial feature map is sampled to two different scales to obtain two sets of second sampled map groups with different scales, wherein each second sampled map in the same second sampled map group has the same scale.

[0075] S242. The two sets of second sampling maps are fused to obtain a second large-scale feature map and a second small-scale feature map. The second large-scale feature map and the second small-scale feature map are both second feature maps. The scale of the second large-scale feature map is larger than the scale of the second small-scale feature map. During fusion, all second sampling maps in the same set of second sampling maps are fused.

[0076] It is understandable that the scale of the first large-scale feature map is the same as that of the second large-scale feature map, and the scale of the first small-scale feature map is the same as that of the second small-scale feature map.

[0077] Step S300: Input the first feature map and the second feature map into the effective attention module to obtain the second large-scale reconstruction map, including:

[0078] S310. Input the first small-scale feature map and the second small-scale feature map into the first effective attention module to obtain the second small-scale reconstructed map;

[0079] S320. The first small-scale feature map and the second small-scale reconstructed map are merged and then input into the thinning module for feature thinning to obtain a feature thinning map. Feature thinning refers to the convolution thinning operation of features. The scale of the feature thinning map obtained after feature thinning is the same as the scale of the second large-scale feature map.

[0080] S330. Input the feature refinement map and the second large-scale feature map into the distortion module to perform affine distortion to obtain a distorted feature map. The distorted feature map is obtained by distorting the feature refinement map in the view direction of the second large-scale feature map.

[0081] S340. Input the first large-scale feature map and the distorted feature map into the second effective attention module to obtain the second large-scale reconstructed map.

[0082] The second small-scale reconstructed map and the second large-scale reconstructed map satisfy the following: in, This indicates a reconstructed image at the second smallest scale or the second largest scale. This represents the first small-scale feature map or the first large-scale feature map. It is the second smallest scale reconstruction or the second largest scale reconstruction, where i represents the i-th scale.

[0083] Understandably, step S310, which inputs the first small-scale feature map and the second small-scale feature map into the first effective attention module to obtain the second small-scale reconstructed map, includes: inputting the first small-scale feature map and the second small-scale feature map into the first effective attention module, and multiplying the first small-scale feature map by the second small-scale feature map after performing max pooling, two-dimensional convolution and sigmoid function activation sequentially through the first effective attention module to obtain the second small-scale reconstructed map;

[0084] Step S340: Input the first large-scale feature map and the distorted feature map into the second effective attention module to obtain the second large-scale reconstructed map, including: inputting the first large-scale feature map and the distorted feature map into the second effective attention module, and multiplying the first large-scale feature map by the second effective attention module after performing max pooling, two-dimensional convolution and sigmoid function activation on the first large-scale feature map in sequence to obtain the second large-scale reconstructed map.

[0085] A single cost volume is insufficient to provide the model with enough feature information. For example, a subtractive cost volume utilizes useful information differences between input pairs of images, enabling the model to obtain results faster; a grouped correlation cost volume can group and store the average information of the input pairs of image channels, stabilizing the results within a certain range; and a connection-merging cost volume provides more comprehensive image information for training the model.

[0086] To make fuller use of feature information and integrate the characteristics of various cost bodies, it is understood that step S400, based on the first feature map and the second large-scale reconstruction map, constructs a hybrid cost body, including:

[0087] Step S410: Based on the first feature map and the second large-scale reconstruction map, construct the subtraction cost body, the group correlation cost body, and the connection merging cost body, respectively;

[0088] Step S420: The subtractive cost body, the grouped correlation cost body, and the connection merging cost body are fused to obtain a hybrid cost body.

[0089] The fusion process of hybrid cost bodies satisfies: C mix =C sub ||C gwc ||C con, where C sub For subtractive cost entities, C gwc For group-related cost bodies, C con For join-and-merge cost body, || represents join-and-merge operation.

[0090] The subtraction-type cost body, the group-related cost body, and the link-merging cost body are obtained through the following methods:

[0091]

[0092]

[0093]

[0094] in, and This represents the extracted first feature map. and N represents the second-largest reconstructed graph. c N represents the number of channels in the extracted feature map. g The number of groups is represented by <*,*>, which represents the inner product operation, and || represents the join / merge operation.

[0095] In stereo matching, the specific cost volume is a crucial step. A novel aggregation module is designed to leverage the hybrid cost volume. This can be understood as inputting the hybrid cost volume into the disparity regression module to obtain a disparity map, including:

[0096] S510. Input the hybrid cost body into the aggregation module to obtain the cost aggregation result;

[0097] S520. Input the cost aggregation result into the disparity regression module to obtain the disparity map.

[0098] The operation of the aggregation module satisfies: C mix′ =NCAM(C mix ), C mix′ C represents the cost aggregation result. mix This indicates a hybrid cost body.

[0099] The binocular stereo matching method of the first aspect of the present invention will be described in detail below with a specific embodiment. It is worth understanding that the following description is merely illustrative and not intended to limit the invention.

[0100] Obtain the first view and the second view, where the first view and the second view are the left image and the right image, respectively.

[0101] The first view is input into the first feature extraction module to obtain first initial feature maps at multiple scales, namely 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Each first initial feature map is sampled at two different scales to obtain two sets of first sampled map groups at different scales. Within the same first sampled map group, each first sampled map has the same scale. The two sets of first sampled map groups are then fused to obtain a first large-scale feature map and a first small-scale feature map. Both the first large-scale and first small-scale feature maps are first feature maps; the scale of the first large-scale feature map is 1 / 4, and the scale of the first small-scale feature map is 1 / 8. During fusion, all first sampled maps within the same first sampled map group are fused. The above execution process is described in... Figure 2 The multi-scale feature extraction and fusion module is performed after the first view in the middle. The execution process of the multi-scale feature extraction and fusion module in the left image is shown in the reference section. Figure 3 As shown in the upper middle part.

[0102] The second view is input into the second feature extraction module to obtain second initial feature maps at multiple scales, namely 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32. The second feature extraction module shares weights with the first feature extraction module. Each second initial feature map is sampled at two different scales, resulting in two sets of second sampled map groups at different scales. Within the same second sampled map group, each second sampled map has the same scale. The two sets of second sampled map groups are then fused to obtain a second large-scale feature map and a second small-scale feature map. Both the second large-scale and the second small-scale feature maps are second feature maps; the scale of the second large-scale feature map is 1 / 4, and the scale of the second small-scale feature map is 1 / 8. During fusion, all second sampled maps within the same set of second sampled map groups are fused. The above execution process is described in... Figure 2 The multi-scale feature extraction and fusion module is performed after the second view. The execution process of the multi-scale feature extraction and fusion module is described in reference [reference needed]. Figure 3 The lower half is shown.

[0103] The first and second small-scale feature maps are input into the first effective attention module. The first small-scale feature map is then subjected to max pooling, 2D convolution, and sigmoid activation, followed by multiplication with the second small-scale feature map to obtain the second small-scale reconstructed map. The above execution process is described in... Figure 2 The effective attention module on the left side is executed; the execution process of the corresponding effective attention module is referenced. Figure 4 As shown, Figure 4 The left feature map shown is the first small-scale feature map. Figure 4 The (distorted) right feature map shown is the second small-scale feature map, and the new (distorted) right feature map is the second small-scale reconstructed map.

[0104] The first small-scale feature map and the second small-scale reconstructed map are fused and then input into the thinning module for feature thinning, resulting in a thinned feature map. Feature thinning refers to the convolutional thinning operation on the features. The scale of the thinned feature map is the same as that of the second large-scale feature map, both being 1 / 4. The above execution process is described in... Figure 2 This is done in the detailed module.

[0105] The refined feature map and the second-largest-scale feature map are input into the warping module for affine warping to obtain a warped feature map. This warped feature map is obtained by warping the refined feature map towards the viewpoint of the second-largest-scale feature map. The above execution process is as follows: Figure 2 This is done in the twisting module.

[0106] The first large-scale feature map and the distorted feature map are input into the second effective attention module. The first large-scale feature map is then subjected to max pooling, 2D convolution, and sigmoid activation, followed by multiplication with the second small-scale feature map to obtain the second large-scale reconstructed map. The above execution process is described in... Figure 2 The effective attention module on the right side is executed, and the execution process of the corresponding effective attention module is referenced. Figure 4 As shown, Figure 4 The left feature map shown is the first large-scale feature map. Figure 4 The (distorted) right feature map shown is a distorted feature map, and the new (distorted) right feature map is a second-largest-scale reconstructed map.

[0107] Based on the first-scale feature map and the second-scale reconstruction map, subtractive cost bodies, grouped correlation cost bodies, and connection-merging cost bodies are constructed respectively. These three cost bodies are then fused to obtain a hybrid cost body. The above execution process is described in... Figure 2 The process involves constructing a hybrid cost body, with the construction process of the hybrid cost body referring to... Figure 5 As shown.

[0108] The hybrid cost body is input into the aggregation module to obtain the cost aggregation result. The above execution process is as follows: Figure 2 The new cost body aggregation module is used.

[0109] The cost aggregation result is input into the disparity regression module to obtain the disparity map. The above execution process is as follows: Figure 2 The parallax regression module is used.

[0110] Each of the above processing modules can be obtained through neural network training.

[0111] In addition, a second aspect of the present invention provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0112] The processor and memory can be connected via a bus or other means.

[0113] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The non-transient software program and instructions required to implement the binocular stereo matching method of the first aspect embodiment described above are stored in a memory. When executed by a processor, the binocular stereo matching method in the above embodiment is executed, for example, the method steps S100 to S500, S210 to S240, S231 and S232, S241 and S242, S310 to S340, S410 to S420, and S510 to S520 described above are executed.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, a third aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described device embodiment, causing the processor to perform the binocular stereo matching method in the above-described embodiment, for example, performing the method steps S100 to S500, S210 to S240, S231 and S232, S241 and S242, S310 to S340, S410 to S420, and S510 to S520 described above.

[0117] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0119] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A binocular stereo matching method, characterized in that, Includes the following steps: Get the first view and the second view; The first view and the second view are respectively input into two feature extraction modules to obtain a first feature map and a second feature map; The first feature map and the second feature map are input into the effective attention module to obtain the second large-scale reconstruction map, wherein the effective attention module is used to reallocate the content weights of the second feature map according to the first feature map; Based on the first feature map and the second large-scale reconstruction map, a hybrid cost body is constructed; The hybrid cost volume is input into the disparity regression module to obtain the disparity map; The step of constructing a hybrid cost body based on the first feature map and the second large-scale reconstruction map includes: Based on the first feature map and the second large-scale reconstruction map, subtraction-type cost bodies, group-related cost bodies, and connection-merging cost bodies are constructed respectively. The subtraction-type cost body, the group correlation-type cost body, and the connection merging-type cost body are fused to obtain a hybrid cost body.

2. The binocular stereo matching method according to claim 1, characterized in that, The step of inputting the first view and the second view into two feature extraction modules respectively to obtain a first feature map and a second feature map includes: The first view is input into the first feature extraction module to obtain a first initial feature map at multiple scales; The second view is input into the second feature extraction module to obtain a second initial feature map at multiple scales; Each of the first initial feature maps is sampled to the same scale and then fused to obtain the first feature map; Each of the second initial feature maps is sampled to the same scale and then fused to obtain the second feature map.

3. The binocular stereo matching method according to claim 2, characterized in that, The step of sampling each of the first initial feature maps to the same scale and then fusing them to obtain the first feature map includes: Each of the first initial feature maps is sampled at two different scales to obtain two sets of first sample maps at different scales, wherein each first sample map in the first sample map set has the same scale. The two sets of the first sampling image groups are fused to obtain a first large-scale feature map and a first small-scale feature map, wherein the first large-scale feature map and the first small-scale feature map are both first feature maps, and the scale of the first large-scale feature map is larger than the scale of the first small-scale feature map.

4. The binocular stereo matching method according to claim 3, characterized in that, The step of sampling each of the second initial feature maps to the same scale and then fusing them to obtain the second feature map includes: Each of the second initial feature maps is sampled at two different scales to obtain two sets of second sample maps at different scales, wherein each second sample map in the second sample map set has the same scale; The two sets of the second sampling map are fused to obtain a second large-scale feature map and a second small-scale feature map. Both the second large-scale feature map and the second small-scale feature map are second feature maps, and the scale of the second large-scale feature map is larger than the scale of the second small-scale feature map.

5. The binocular stereo matching method according to claim 4, characterized in that, The scale of the first large-scale feature map is the same as the scale of the second large-scale feature map, and the scale of the first small-scale feature map is the same as the scale of the second small-scale feature map. The step of inputting the first feature map and the second feature map into the effective attention module to obtain the second large-scale reconstructed map includes: The first small-scale feature map and the second small-scale feature map are input into the first effective attention module to obtain the second small-scale reconstructed map. The first small-scale feature map and the second small-scale reconstructed map are fused and then input into the thinning module for feature thinning to obtain a feature thinning map. The refined feature map and the second large-scale feature map are input into the warping module to obtain a warped feature map; The first large-scale feature map and the distorted feature map are input into the second effective attention module to obtain the second large-scale reconstructed map.

6. The binocular stereo matching method according to claim 5, characterized in that, The step of inputting the first small-scale feature map and the second small-scale feature map into the first effective attention module to obtain the second small-scale reconstructed map includes: inputting the first small-scale feature map and the second small-scale feature map into the first effective attention module, and multiplying the first small-scale feature map by the second small-scale feature map after performing max pooling, two-dimensional convolution and activation sequentially through the first effective attention module to obtain the second small-scale reconstructed map; The step of inputting the first large-scale feature map and the distorted feature map into the second effective attention module to obtain the second large-scale reconstructed map includes: inputting the first large-scale feature map and the distorted feature map into the second effective attention module, and then multiplying the first large-scale feature map by the second effective attention module after performing max pooling, two-dimensional convolution and activation sequentially, to obtain the second large-scale reconstructed map.

7. The binocular stereo matching method according to claim 1, characterized in that, The step of inputting the hybrid cost volume into the disparity regression module to obtain a disparity map includes: The hybrid cost body is input into the aggregation module to obtain the cost aggregation result; The cost aggregation result is input into the disparity regression module to obtain the disparity map.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a binocular stereo matching method as described in any one of claims 1 to 7.

9. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing a binocular stereo matching method as described in any one of claims 1 to 7.

Images