Binocular image matching methods, devices, equipment and media

Abstract

This application provides a binocular image matching method, apparatus, device, and medium. The method includes acquiring a binocular image; performing feature extraction processing on the binocular image to obtain multiple feature maps of different scales; fusing the multiple feature maps to obtain multiple fused feature maps; generating multiple cost bodies of different types based on the multiple fused feature maps; aggregating the cost bodies using an hourglass module to obtain a target cost body; merging the multiple cost bodies to obtain a source cost body; generating a first target cost body using an attention module based on the source cost body and the target cost body; aggregating the first target cost body using an hourglass module to obtain a second target cost body; and performing visual regression processing based on the second target cost body to obtain a disparity map. This method can focus on useful information from the cost bodies during binocular image matching, improving the efficiency and quality of disparity map generation.

Description

Technical Field

[0001] The embodiments of this application relate to, but are not limited to, the field of image processing, and particularly to binocular image matching methods, apparatuses, devices, and media. Background Technology

[0002] Stereo image matching, also known as disparity estimation, typically aims to obtain the disparity between left and right input image pairs and plays a role in visual processing and robotics applications. Achieving high-precision results usually requires significant computational resources. Similarly, repeatedly aggregating feature-rich hybrid cost volumes can help obtain relatively accurate disparities, but this consumes substantial computational resources and slows down the process, hindering stereo image matching models from obtaining results in a more efficient manner. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This application provides a binocular image matching method, apparatus, device, and medium that can focus on useful information of the cost volume during the binocular image matching process and improve the quality of the disparity map.

[0005] An embodiment of the first aspect of this application provides a binocular image matching method, comprising:

[0006] Acquire binocular images;

[0007] The binocular image is subjected to feature extraction processing to obtain multiple feature maps of different scales, and the multiple feature maps are fused to obtain multiple fused feature maps;

[0008] Multiple cost bodies of different types are generated based on the multiple fused feature maps;

[0009] The target cost body is obtained by aggregating the cost body using the hourglass module;

[0010] The source cost body is obtained by merging multiple cost bodies.

[0011] The attention module generates a first target cost body based on the source cost body and the target cost body;

[0012] The hourglass module aggregates the first target cost body to obtain the second target cost body;

[0013] A disparity map is obtained by performing visual regression processing based on the second target cost volume.

[0014] In certain embodiments of the first aspect of this application, the cost body includes a subtraction cost body, a join-and-merge cost body, and a grouping correlation cost body; the step of generating multiple cost bodies of different types based on multiple fused feature maps includes:

[0015] The subtractive cost body is generated by the first function based on multiple fused feature maps;

[0016] The second function generates the connection-merging cost body based on multiple fused feature maps;

[0017] The grouped correlation cost body is generated by the third function based on multiple fused feature maps.

[0018] In certain embodiments of the first aspect of this application, the process of merging multiple cost bodies to obtain a source cost body includes:

[0019] The source cost body, which is of the mixed type, is obtained by merging the subtractive cost body, the connection-merging cost body, and the grouping correlation cost body.

[0020] In some embodiments of the first aspect of this application, the attention module includes a 3D convolutional layer, an activation function layer, and a multiplication layer; the generation of the first target cost body by the attention module based on the source cost body and the target cost body specifically involves: the result of the source cost body processed by the 3D convolutional layer and the activation function layer and the target cost body being input into the multiplication layer for multiplication processing to generate the first target cost body.

[0021] In certain embodiments of the first aspect of this application, the hourglass module includes nine 3D convolutional blocks, two 3D deconvolutional blocks, two attention modules, and two activation function layers; the output of the first 3D convolutional block serves as the input to the second 3D convolutional block and the first right branch 3D convolutional block; the output of the second 3D convolutional block serves as the input to the third 3D convolutional block and the first left branch 3D convolutional block; the result of concatenating and merging the outputs of the third 3D convolutional block and the first right branch 3D convolutional block serves as the input to the first attention module; the output of the first attention module serves as the input to the fourth 3D convolutional block, the second right branch 3D convolutional block, and the second left branch 3D convolutional block. The input of the first 3D convolutional block is the output of the fourth 3D convolutional block. The output of the fifth 3D convolutional block and the output of the second right branch 3D convolutional block are concatenated and merged as the input of the second attention module. The output of the second attention module is the input of the first 3D deconvolutional block. The output of the first 3D deconvolutional block and the output of the second left branch 3D convolutional block are concatenated and merged as the input of the first activation function layer. The output of the first activation function layer is the input of the second 3D deconvolutional block. The output of the second 3D deconvolutional block and the output of the first left branch 3D convolutional block are concatenated and merged as the input of the second activation function.

[0022] In some embodiments of the first aspect of this application, the 3D convolutional block includes a 3D convolutional layer, a batch normalization layer, and an activation function layer.

[0023] In some embodiments of the first aspect of this application, the 3D deconvolution block includes a 3D deconvolution layer and a batch normalization layer.

[0024] According to a second aspect of this application, a binocular image matching device includes:

[0025] Image acquisition unit, used to acquire stereo images;

[0026] The feature extraction unit is used to perform feature extraction processing on the binocular image to obtain multiple feature maps of different scales, and to perform fusion processing on the multiple feature maps to obtain multiple fused feature maps;

[0027] The cost body generation unit is used to generate multiple cost bodies of different types based on the multiple fused feature maps;

[0028] The target cost body generation unit is used to aggregate the cost body by the hourglass module to obtain the target cost body;

[0029] A source cost body generation unit is used to obtain a source cost body by merging multiple cost bodies;

[0030] The first target cost body generation unit is used to generate a first target cost body by the attention module based on the source cost body and the target cost body;

[0031] The second target cost body generation unit is used to aggregate the second target cost body from the first target cost body by the hourglass module;

[0032] The disparity map generation unit is used to perform visual regression processing based on the second target cost volume to obtain a disparity map.

[0033] According to a third aspect of this application, an electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the binocular image matching method as described above.

[0034] According to a fourth aspect of this application, a computer-readable storage medium stores computer-executable instructions for causing a computer to perform the binocular image matching method as described above.

[0035] The above scheme has at least the following beneficial effects: by emphasizing and focusing on the required information and suppressing useless information through the attention module, the computational amount of the aggregation cost body process is reduced, while retaining the advantages of the hybrid cost body, so as to generate a better first target cost body; by adding the attention module to the hourglass module, more useful information data can be provided and stored to the subtractive cost body with fewer feature messages than other types of cost bodies during the iteration process, emphasizing the useful information of the subtractive cost body in the hourglass module and suppressing the propagation of harmful messages, so as to generate a better second target cost body, thereby improving the disparity map generation efficiency and the quality of the disparity map. Attached Figure Description

[0036] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0037] Figure 1 This is a flowchart illustrating the steps of the binocular image matching method provided in an embodiment of this application;

[0038] Figure 2 This is a structural diagram of the attention module;

[0039] Figure 3 This is a structural diagram of the hourglass module;

[0040] Figure 4 This is a schematic diagram of the network structure corresponding to the binocular image matching method. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0043] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0044] An embodiment of this application provides a binocular image matching method, and a schematic diagram of the network structure corresponding to the binocular image matching method is shown below. Figure 4 As shown.

[0045] Reference Figure 1 The binocular image matching method includes, but is not limited to, the following steps:

[0046] Step S100: Acquire stereo images;

[0047] Step S200: Perform feature extraction processing on the binocular image to obtain multiple feature maps of different scales, and perform fusion processing on the multiple feature maps to obtain multiple fused feature maps;

[0048] Step S300: Generate multiple cost bodies of different types based on multiple fused feature maps;

[0049] Step S400: The hourglass module aggregates the cost body to obtain the target cost body;

[0050] Step S500: The source cost body is obtained by merging multiple cost bodies;

[0051] Step S600: The attention module generates the first target cost body based on the source cost body and the target cost body;

[0052] Step S700: The hourglass module aggregates the first target cost body to obtain the second target cost body;

[0053] Step S800: Perform visual regression processing based on the second target cost volume to obtain a disparity map.

[0054] In step S100, a binocular image is obtained by capturing images using a binocular camera. The binocular camera includes at least two cameras, and the binocular image captured by the binocular camera typically includes a left image and a right image.

[0055] During the network training phase, the training set composed of stereo images is input into the matching network to train the matching network; during the network application phase, the stereo images to be matched are input into the already trained matching network.

[0056] In step S200, within the matching network, the feature extraction module performs feature extraction processing on the binocular image to obtain multiple feature maps of different scales. Then, the fusion module performs fusion processing on these multiple feature maps to obtain multiple fused feature maps. For example, the left image yields two 1 / 4 scale left feature maps, and the right image yields two 1 / 4 scale right feature maps.

[0057] For step S300, the cost body generation module generates multiple cost bodies of different types based on multiple fused feature maps, including but not limited to the following steps:

[0058] The first function generates a subtractive cost body based on multiple fused feature maps;

[0059] The second function generates a connection-merged cost body based on multiple fused feature maps;

[0060] The third function generates the grouped correlation cost body based on multiple fused feature maps.

[0061] Understandably, the first function is used to generate a subtractive cost body, the second function is used to generate a connection-merging cost body, and the third function is used to generate a grouped correlation cost body.

[0062] The cost body is the total cost of feature matching between the left and right images calculated from the input left and right images.

[0063] For step S400, the hourglass module aggregates the cost bodies to obtain the target cost body. It should be noted that step S400 can be completed during the network training phase. The hourglass module aggregates the subtractive cost bodies to obtain another subtractive cost body as the target cost body, and saves the target cost body.

[0064] For step S500, a source cost body is obtained by merging multiple cost bodies, specifically including: a source cost body of type hybrid cost body obtained by merging subtractive cost bodies, concatenated cost bodies, and grouped correlation cost bodies.

[0065] For step S600, the attention module generates a first target cost body based on the source cost body and the target cost body. Specifically, the first target cost body of type subtractive cost body is generated based on the target cost body of type subtractive cost body and the source cost body of type hybrid cost body.

[0066] Reference Figure 2 The attention module includes 3D convolutional layers, activation function layers, and multiplication layers. The source cost volume, after being processed by the 3D convolutional layers and activation function layers, and the target cost volume are input into the multiplication layer for multiplication to generate the first target cost volume.

[0067] While directly aggregating hybrid cost bodies with rich feature information can achieve good results, its heavy computational burden leads to slower speed and longer runtime. To reduce the computational cost of the cost body aggregation process and retain the advantages of hybrid cost bodies, an attention module is used to emphasize and focus on the required information while suppressing useless information.

[0068] When the source cost volume and the target cost volume are input into the attention module, the target cost volume will be generated by taking the source cost volume as a reference and then being processed by 3D convolution, softmax activation function and multiplication operation.

[0069] We select the hybrid cost body as the source cost body and the subtractive cost body as the target cost body. We obtain useful clues from the hybrid cost body to filter information as much as possible, thereby reducing some unnecessary and time-consuming operations.

[0070] For step S700, the hourglass module aggregates the first target cost body to obtain the second target cost body. The type of the second target cost body is the same as that of the first target cost body, and the second target cost body is also a subtraction type cost body.

[0071] Reference Figure 3The hourglass module includes 9 3D convolutional blocks, 2 3D deconvolutional blocks, 2 attention modules, and 2 activation function layers. The output of the first 3D convolutional block serves as the input to the second 3D convolutional block and the first right branch 3D convolutional block. The output of the second 3D convolutional block serves as the input to the third 3D convolutional block and the first left branch 3D convolutional block. The concatenated output of the third 3D convolutional block and the first right branch 3D convolutional block serves as the input to the first attention module. The output of the first attention module serves as the input to the fourth 3D convolutional block, the second right branch 3D convolutional block, and the second left branch 3D convolutional block. The fourth... The output of the 3D convolutional block is used as the input of the fifth 3D convolutional block. The output of the fifth 3D convolutional block and the output of the second right branch 3D convolutional block are concatenated and merged as the input of the second attention module. The output of the second attention module is used as the input of the first 3D deconvolutional block. The output of the first 3D deconvolutional block and the output of the second left branch 3D convolutional block are concatenated and merged as the input of the first activation function layer. The output of the first activation function layer is used as the input of the second 3D deconvolutional block. The output of the second 3D deconvolutional block and the output of the first left branch 3D convolutional block are concatenated and merged as the input of the second activation function.

[0072] A 3D convolutional block consists of a 3D convolutional layer, a batch normalization layer, and an activation function layer.

[0073] A 3D deconvolution block consists of a 3D deconvolution layer and a batch normalized layer.

[0074] Applying a 3D convolutional block with a kernel size of 1 and a stride of 1×1×1 to the subtractive cost body before the aggregate cost body will improve the performance of the hourglass module to some extent.

[0075] The hourglass module typically iterates three times to utilize the feature information from the cost body. Better results are achieved when the hourglass module can provide and store more useful information data to the subtractive cost body, which contains fewer feature messages than other types of cost bodies, during iteration. To fully leverage the subtractive cost body, an attention module is used as an insertion component into the hourglass model to highlight useful information from the subtractive cost body within the hourglass module and suppress the propagation of harmful messages.

[0076] In addition, the hourglass module addresses the issues of gradient vanishing and feature information loss by performing two residual operations.

[0077] In step S800, a disparity map is obtained by performing visual regression processing based on the second target cost volume using the disparity regression module. Disparity is the directional difference produced when observing the same target from two points at a certain distance, and the disparity map reflects this directional difference.

[0078] By emphasizing and focusing on necessary information and suppressing useless information through the attention module, the computational cost of the aggregation cost body process is reduced, while retaining the advantages of the hybrid cost body, to generate a better first target cost body. The hourglass module with added attention can provide and store more useful information data to the subtractive cost body with fewer feature messages than other types of cost bodies during the iteration process, emphasizing the useful information of the subtractive cost body in the hourglass module and suppressing the propagation of harmful messages to generate a better second target cost body, thereby improving the disparity map generation efficiency and the quality of the disparity map.

[0079] Embodiments of this application also provide a binocular image matching device.

[0080] The binocular image matching device includes an image acquisition unit, a feature extraction unit, a cost volume generation unit, a target cost volume generation unit, a source cost volume generation unit, a first target cost volume generation unit, a second target cost volume generation unit, and a disparity map generation unit.

[0081] The system includes: an image acquisition unit for acquiring binocular images; a feature extraction unit for extracting features from the binocular images to obtain multiple feature maps of different scales, and fusing these feature maps to obtain multiple fused feature maps; a cost volume generation unit for generating multiple cost volumes of different types based on the fused feature maps; a target cost volume generation unit for aggregating cost volumes using the hourglass module to obtain a target cost volume; a source cost volume generation unit for merging multiple cost volumes to obtain a source cost volume; a first target cost volume generation unit for generating a first target cost volume using the attention module based on the source cost volume and the target cost volume; a second target cost volume generation unit for aggregating the first target cost volume using the hourglass module to obtain a second target cost volume; and a disparity map generation unit for performing visual regression processing based on the second target cost volume to obtain a disparity map.

[0082] It is understood that the content of the binocular image matching method embodiment is applicable to the binocular image matching device embodiment. The specific functions implemented by the binocular image matching device embodiment are the same as those of the binocular image matching method embodiment, and the beneficial effects achieved are also the same as those achieved by the binocular image matching method embodiment.

[0083] Embodiments of this application also provide an electronic device. The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the binocular image matching method as described above.

[0084] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the function constructions in the embodiments of the present invention described above. The processor implements the function constructions in the embodiments of the present invention by running the non-transitory software programs and programs stored in the memory.

[0085] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function. The data storage area may store data required for executing the functions described in the embodiments of the present invention. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] Embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the binocular image matching method described above.

[0087] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0088] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0089] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A binocular image matching method, characterized by, The method comprises: obtaining binocular images; performing feature extraction processing on the binocular images to obtain a plurality of feature maps of different scales, and performing fusion processing on the plurality of feature maps to obtain a plurality of fused feature maps; generating a plurality of cost volumes of different types according to the plurality of fused feature maps; aggregating the cost volumes by a sandglass module to obtain a target cost volume; merging the plurality of cost volumes to obtain a source cost volume; generating a first target cost volume by an attention module according to the source cost volume and the target cost volume; aggregating the first target cost volume by the sandglass module to obtain a second target cost volume; performing visual regression processing according to the second target cost volume to obtain a disparity map; The sandglass module comprises 9 3D convolution blocks, 2 3D deconvolution blocks, 2 attention modules, and 2 activation function layers; the output of the first 3D convolution block is used as the input of the second 3D convolution block and the 3D convolution block of the first right branch; the output of the second 3D convolution block is used as the input of the third 3D convolution block and the 3D convolution block of the first left branch; the output of the third 3D convolution block and the output of the 3D convolution block of the first right branch are connected and merged to obtain a result, which is used as the input of the first attention module; the output of the first attention module is used as the input of the fourth 3D convolution block, the 3D convolution block of the second right branch, and the 3D convolution block of the second left branch; the output of the fourth 3D convolution block is used as the input of the fifth 3D convolution block; the output of the fifth 3D convolution block and the output of the 3D convolution block of the second right branch are connected and merged to obtain a result, which is used as the input of the second attention module; the output of the second attention module is used as the input of the first 3D deconvolution block; the output of the first 3D deconvolution block and the output of the 3D convolution block of the second left branch are connected and merged to obtain a result, which is used as the input of the first activation function layer; the output of the first activation function layer is used as the input of the second 3D deconvolution block; the output of the second 3D deconvolution block and the output of the 3D convolution block of the first left branch are connected and merged to obtain a result, which is used as the input of the second activation function layer.

2. The binocular image matching method according to claim 1, wherein The cost volume comprises a subtraction type cost volume, a connection and merging type cost volume, and a grouping correlation cost volume; The method comprises: generating the subtraction type cost volume according to the plurality of fused feature maps by a first function; generating the connection and merging type cost volume according to the plurality of fused feature maps by a second function; generating the grouping correlation cost volume according to the plurality of fused feature maps by a third function.

3. The binocular image matching method of claim 2, wherein, The method comprises: merging the subtraction type cost volume, the connection and merging type cost volume, and the grouping correlation cost volume to obtain a source cost volume of a mixed type.

4. The binocular image matching method of claim 1, wherein, The attention module comprises a 3D convolution layer, an activation function layer, and a multiplication layer; the method comprises: multiplying the source cost volume after being processed by the 3D convolution layer and the activation function layer with the target cost volume to generate the first target cost volume.

5. The binocular image matching method of claim 1, wherein, The 3D convolution block comprises a 3D convolution layer, a batch normalization layer and an activation function layer.

6. The binocular image matching method of claim 1, wherein, The 3D deconvolution block comprises a 3D deconvolution layer and a batch normalization layer.

7. A binocular image matching apparatus characterized by comprising: It comprises: An image acquisition unit is configured to acquire binocular images. A feature extraction unit is configured to perform feature extraction on the binocular images to obtain a plurality of feature maps of different scales, and perform fusion processing on the plurality of feature maps to obtain a plurality of fused feature maps. A cost volume generation unit is configured to generate a plurality of cost volumes of different types according to the plurality of fused feature maps. A target cost volume generation unit is configured to generate a target cost volume according to the cost volumes by a sandglass module. A source cost volume generation unit is configured to merge the plurality of cost volumes to obtain a source cost volume. A first target cost volume generation unit is configured to generate a first target cost volume according to the source cost volume and the target cost volume by an attention module. A second target cost volume generation unit is configured to generate a second target cost volume according to the first target cost volume by a sandglass module. A disparity map generation unit is configured to perform visual regression processing on the second target cost volume to obtain a disparity map. The sandglass module comprises nine 3D convolution blocks, two 3D deconvolution blocks, two attention modules and two activation function layers.

8. An electronic device, comprising: The electronic device comprises a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the binocular image matching method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to execute the binocular image matching method according to any one of claims 1 to 6.

Images