Binocular stereo matching method and device for mixed cost volume and storage medium

Abstract

The application discloses a binocular stereo matching method of mixed cost volume, equipment and storage medium, the method comprises the following steps: acquiring left and right stereo image pairs; multi-scale feature extraction is carried out on the left and right stereo image pairs, and a feature map pair under multiple scales is obtained; 3D matching cost volume and 4D matching cost volume of the corresponding scale are constructed according to the feature map pair under each scale; the 4D matching cost volume is aggregated and compressed to be equal to the size of the 3D matching cost volume, and a 4D compressed cost volume is obtained; the 4D compressed cost volume of the corresponding scale is connected with the 3D matching cost volume, and a mixed matching cost volume under multiple scales is obtained; the mixed matching cost volume is input into a 2D cost aggregation model for disparity calculation, and a disparity map is obtained. The application constructs 3D matching cost volume and 4D matching cost volume of multiple scales, obtains mixed matching cost volume under multiple scales, has high estimation performance, and can effectively reduce the operation amount through the 2D cost aggregation model.

Description

Technical Field

[0001] This invention relates to the field of image data processing, and in particular to a binocular stereo matching method, device and storage medium for hybrid cost data. Background Technology

[0002] Binocular stereo matching is a branch of computer vision. It is also known as binocular disparity estimation or binocular depth estimation. It typically uses a cost volume constructed from left and right feature maps to obtain a disparity map or depth map, and then estimates disparity and depth. It is widely used in fields such as 3D reconstruction, autonomous driving and robot navigation.

[0003] Binocular stereo matching primarily utilizes two-view images of the same scene simultaneously captured by identical cameras, simulating human eyes. Depth can be directly recovered using the disparity between the two images. In related technologies, cost aggregation in binocular stereo matching is typically achieved using 2D convolutional neural networks to improve computational speed. However, this method leads to significant information loss, affecting the accuracy of the output disparity map. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention provides a binocular stereo matching method, device, and storage medium with hybrid cost volume, which can simultaneously ensure computing speed and the accuracy of the output disparity map.

[0005] A first aspect of the present invention provides a binocular stereo matching method for hybrid cost bodies, comprising:

[0006] Obtain left and right stereo image pairs;

[0007] Multi-scale feature extraction is performed on the left and right stereo image pairs to obtain feature map pairs at multiple scales;

[0008] Based on the feature map pairs at each scale, construct the 3D matching cost volume and 4D matching cost volume for the corresponding scale;

[0009] The 4D matching cost volume is aggregated and compressed to the same size as the 3D matching cost volume to obtain the 4D compressed cost volume;

[0010] By connecting the 4D compressed cost volume at the corresponding scale with the 3D matching cost volume, a hybrid matching cost volume at multiple scales is obtained.

[0011] The hybrid matching cost volume is input into the 2D cost aggregation model for disparity calculation to obtain the disparity map.

[0012] According to the above embodiments of the present invention, at least the following beneficial effects are achieved: by constructing multi-scale 3D matching cost bodies and 4D matching cost bodies, and by concatenating the 4D matching cost body with the 3D matching cost body after aggregation and compression, a hybrid matching cost body at multiple scales is obtained. This can simultaneously preserve the similarity information and contextual information of the image, thereby effectively improving the effectiveness of the information carried by the hybrid matching cost body, resulting in high estimation performance. Furthermore, by aggregating through the 2D cost body aggregation model, the amount of computation can be effectively reduced, resulting in low memory consumption and fast computation speed.

[0013] According to some embodiments of the first aspect of the present invention, multi-scale feature extraction is performed on left and right stereo image pairs to obtain feature map pairs at multiple scales, including:

[0014] The left and right stereo image pairs are downsampled to obtain the first sampled image pair;

[0015] Multi-scale feature extraction is performed on the first sampled image pair to obtain multiple initial feature image pairs;

[0016] Each initial feature map pair is concatenated to obtain the first scale feature map pair;

[0017] By extracting features from the first-scale feature map pair using residual blocks, we obtain the second-scale feature map pair and the third-scale feature map pair.

[0018] According to some embodiments of the first aspect of the present invention, constructing a 3D matching cost volume of a corresponding scale includes:

[0019] The view correlation is calculated based on the left and right stereo images;

[0020] Based on the view relevance and feature map pairs at each scale, a 3D matching cost volume for the corresponding scale is constructed. The size of the 3D matching cost volume is D×H×W, where D represents the disparity range and H×W represents the spatial size.

[0021] According to some embodiments of the first aspect of the present invention, constructing a 4D matching cost body of a corresponding scale includes:

[0022] The feature maps at each scale are concatenated to obtain the 4D matching cost volume for the corresponding scale. The size of the 4D matching cost volume is C×D×H×W, where C represents the number of channels after feature concatenation.

[0023] According to some embodiments of the first aspect of the present invention, a 4D matching cost body is polymerized and compressed to the same size as a 3D matching cost body to obtain a 4D compressed cost body, comprising:

[0024] The 4D matching cost volume is aggregated using a 3D convolution module to obtain a 4D aggregated cost volume;

[0025] The 4D aggregate cost body is compressed to the same size as the 3D matching cost body to obtain the 4D compressed cost body.

[0026] According to some embodiments of the first aspect of the present invention, concatenating 4D compressed cost volumes and 3D matching cost volumes at corresponding scales yields a hybrid matching cost volume at multiple scales, including:

[0027] The 4D compressed cost volume and the 3D matching cost volume at the first scale are connected to obtain the first hybrid matching cost volume corresponding to the first scale.

[0028] By connecting the 4D compressed cost volume and the 3D matching cost volume at the second scale, the second hybrid matching cost volume corresponding to the second scale is obtained.

[0029] By connecting the 4D compressed cost volume and the 3D matching cost volume at the third scale, the third hybrid matching cost volume corresponding to the third scale is obtained.

[0030] According to some embodiments of the first aspect of the present invention, a disparity map is obtained by inputting a hybrid matching cost volume into a 2D cost aggregation model for disparity calculation, including:

[0031] The first hybrid matching cost body is regularized by the first 2D convolution module in the 2D cost aggregation model to obtain the regularized cost body;

[0032] The regularized cost volume is downsampled using a 2D cost aggregation model to obtain the first sampled cost volume at the second scale.

[0033] The first sampled cost body and the second hybrid matching cost body are connected by a 2D cost aggregation model to obtain the first connection cost body;

[0034] The number of feature channels of the first connected cost volume is reduced to a specified value by the second 2D convolution module in the 2D cost aggregation model to obtain a low-channel cost volume.

[0035] The low-channel cost volume is downsampled using a 2D cost aggregation model to obtain the second sampled cost volume at the third scale.

[0036] The second sampling cost body is obtained by connecting the third hybrid matching cost body through a 2D cost aggregation model.

[0037] The second connection cost volume is upsampled by the third 2D convolution module in the 2D cost aggregation model to obtain the third sampled cost volume at the first scale.

[0038] The disparity is calculated on the third sampled cost volume using a 2D cost aggregation model to obtain a disparity map.

[0039] According to some embodiments of the first aspect of the present invention, disparity is calculated on a third sampled cost volume using a 2D cost aggregation model to obtain a disparity map, including:

[0040] The third sample cost volume is bilinearly interpolated and then input into the disparity regression module in the 2D cost aggregation model to obtain the disparity map.

[0041] A second aspect of the present invention provides an electronic device, comprising:

[0042] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a binocular stereo matching method for a hybrid cost body that is any of the first aspects.

[0043] Since the electronic device of the second aspect embodiment applies the binocular stereo matching method of any one of the first aspects of the invention, it has all the beneficial effects of the first aspect of the invention.

[0044] According to a third aspect of the present invention, a computer storage medium is provided, storing computer-executable instructions for executing the binocular stereo matching method of any one of the first aspects.

[0045] Since the computer storage medium of the third aspect embodiment can perform the binocular stereo matching method of any one of the first aspects of the invention, it has all the beneficial effects of the first aspect of the invention.

[0046] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0048] Figure 1 This is a diagram showing the main steps of the binocular stereo matching method for hybrid cost bodies according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of step S200 in the binocular stereo matching method for hybrid cost bodies according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of step S300 in the binocular stereo matching method for hybrid cost bodies according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of step S400 in the binocular stereo matching method for hybrid cost bodies according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of step S500 in the binocular stereo matching method for hybrid cost bodies according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of step S600 in the binocular stereo matching method for hybrid cost bodies according to an embodiment of the present invention;

[0054] Figure 7 This is a framework diagram of the network used in the binocular stereo matching method with hybrid cost body according to an embodiment of the present invention;

[0055] Figure 8 yes Figure 7 Framework diagram for constructing a hybrid cost body;

[0056] Figure 9 yes Figure 7 Framework diagram of the 2DCNN cost volume aggregation model. Detailed Implementation

[0057] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly. Those skilled in the art can reasonably determine the specific meaning of these terms in this invention based on the specific content of the technical solution. In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. Furthermore, features specified as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0058] After years of research, binocular stereo matching technology has developed a mature theoretical system and can be applied in practice. Due to its low cost, simple structure, and strong practicality, it has wide applications in 3D reconstruction, autonomous driving, medical diagnosis, robot navigation, and other fields.

[0059] Binocular stereo matching can be broadly divided into four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization. Early research on binocular stereo matching used traditional algorithms. Traditional stereo matching methods are affected by manually designed features during matching cost calculation, leading to a lack of context information and thus impacting performance. Furthermore, in disparity optimization, traditional algorithms use post-processing functions with empirical parameters, which significantly influence matching results, making them unsuitable for complex environments. With the significant achievements of deep learning in computer vision, research on binocular stereo matching algorithms has gradually incorporated convolutional neural networks (CNNs) for disparity estimation. Deep learning methods can achieve good disparity results without post-processing. Initially, CNNs were used to replace one or more of the four steps in traditional algorithms, forming a non-end-to-end stereo matching network. This gradually evolved to include convolutional layers adapted to the entire process, resulting in a high-quality, reasonably fast end-to-end network.

[0060] In related technologies, end-to-end stereo matching networks can be broadly classified into two categories based on the method of generating the matching cost volume: 2D convolutional neural network-based methods and 3D convolutional neural network-based methods. With the success of the stereo matching network DispNetC, a correlation layer representing the relationship between left and right images was introduced to construct the 3D matching cost volume. 2D convolution was used in the cost aggregation step, significantly improving estimation performance. However, while the 2D convolutional neural network-based method is relatively fast, it only extracts contextual information in the spatial dimension, ignoring information in the disparity dimension. The stereo matching network GCNet employs a novel approach: concatenating each unary feature map from the two images at each disparity level and packaging them into a 4D cost volume, then using 3D convolution for cost aggregation. Inspired by the stereo matching network GCNet, the stereo matching network PSMNet applies an SPP (Spatial Pyramid Pooling) module to extract multi-scale feature representations and uses a stacked 3D CNN hourglass structure to aggregate contextual features. Although the 3D CNN hourglass structure can achieve high accuracy, it has high computational cost and memory consumption.

[0061] The correlation layer used in existing 2D convolutional neural network-based methods can effectively calculate feature similarity, but because it only generates one feature channel for each disparity level, it loses a lot of detailed information, leading to large errors. Another approach, the 4D matching cost volume constructed using 3D convolutional neural networks, has richer contextual information but lacks a measure of feature similarity. Therefore, the subsequent cost aggregation network needs to use more parameters to learn the feature similarity function, resulting in higher memory consumption and longer runtime.

[0062] Based on this, this invention constructs a hybrid 3D-4D matching cost volume by integrating 3D correlation features and 4D concatenation features, while preserving similarity and contextual information. It uses a stacked 2D hourglass structure for cost aggregation, improving model speed without sacrificing accuracy. Furthermore, we introduce a method for fusing multi-scale cost volumes, aiming to extract global and structural features from all scales. The hybrid 3D-4D cost volume method constructed in this invention does not lose excessive information compared to 2D convolutional neural network-based methods, and its reduced parameter count leads to improved computational speed compared to 3D convolutional neural network-based methods.

[0063] The following reference Figures 1 to 9 The present invention describes a binocular stereo matching method, apparatus, and storage medium with hybrid cost data, which can simultaneously ensure computational speed and the accuracy of the output disparity map.

[0064] refer to Figure 1 As shown, the binocular stereo matching method for hybrid cost bodies according to the first aspect of the present invention includes:

[0065] S100. Obtain a pair of left and right stereo images, wherein the pair of left and right stereo images includes a left image and a right image;

[0066] S200. Perform multi-scale feature extraction on the left and right stereo image pairs to obtain feature map pairs at multiple scales;

[0067] S300. Based on the feature map pairs at each scale, construct the 3D matching cost volume and 4D matching cost volume for the corresponding scale.

[0068] S400: Aggregate and compress the 4D matching cost body to the same size as the 3D matching cost body to obtain the 4D compressed cost body;

[0069] S500: Connect the 4D compressed cost volume and the 3D matching cost volume at the corresponding scale to obtain a hybrid matching cost volume at multiple scales;

[0070] S600. Input the hybrid matching cost volume into the 2D cost aggregation model to calculate disparity and obtain the disparity map.

[0071] By constructing multi-scale 3D and 4D matching cost volumes, and concatenating the 4D matching cost volume with the 3D matching cost volume after aggregation and compression, a multi-scale hybrid matching cost volume is obtained. This can simultaneously preserve the similarity information and contextual information of the image, thereby effectively improving the effectiveness of the information carried by the hybrid matching cost volume. It has high estimation performance. By aggregating through the 2D cost volume aggregation model, the amount of computation can be effectively reduced, with low memory consumption and fast computation speed.

[0072] Understandably, reference Figure 2 As shown, in step S200, multi-scale feature extraction is performed on the left and right stereo image pairs to obtain feature map pairs at multiple scales, including:

[0073] S210. Downsample the left and right stereo image pairs to obtain the first sampled image pair;

[0074] S220. Perform multi-scale feature extraction on the first sampled image pair to obtain multiple initial feature image pairs;

[0075] S230. Connect each initial feature map pair to obtain the first scale feature map pair;

[0076] S240. Extract features from the first-scale feature map pair using residual blocks to obtain the second-scale feature map pair and the third-scale feature map pair.

[0077] Understandably, reference Figure 3 As shown, the 3D matching cost volume of the corresponding scale is constructed, including:

[0078] S310. Calculate the view correlation degree based on the left and right stereoscopic image pairs, whereby the view correlation degree is used to represent the relationship between the left and right images in the left and right stereoscopic image pairs;

[0079] S320. Based on the view relevance and feature map pairs at each scale, construct the 3D matching cost volume for the corresponding scale. The size of the 3D matching cost volume is D×H×W, where D represents the disparity range, H×W represents the spatial size, H represents the height of the 3D matching cost volume, and W represents the width of the 3D matching cost volume.

[0080] Understandably, reference Figure 3 As shown, the 4D matching cost volume of the corresponding scale is constructed, including:

[0081] S330. Connect the feature map pairs at each scale to obtain the 4D matching cost volume at the corresponding scale. The feature map pair has two feature maps, left and right. The size of the 4D matching cost volume is C×D×H×W, where C represents the number of channels after feature connection, D represents the disparity range, H represents the height of the 4D matching cost volume, and W represents the width of the 4D matching cost volume.

[0082] It is understandable that a feature map pair includes a left feature map and a right feature map, with the left feature map represented as follows: The right feature map is represented as s represents the scale level, with s values ​​of 1, 2, and 3 corresponding to scales of 1 / 4, 1 / 8, and 1 / 16, respectively.

[0083] 3D matching cost body C 3D It can be constructed using the following formula:

[0084]

[0085] 4D matching cost body C 4D It can be constructed using the following formula:

[0086] C 4D (x, y, d) = Concat{f L (x, y), f R (xd, y)}

[0087] Where d is the maximum disparity search range, d∈[0,192), and (x,y) is the pixel position.

[0088] Understandably, reference Figure 4 As shown, step S400 involves aggregating and compressing the 4D matching cost body to the same size as the 3D matching cost body to obtain a 4D compressed cost body, including:

[0089] S410. Aggregate the 4D matching cost volume using the 3D convolution module to obtain the 4D aggregated cost volume;

[0090] S420. Compress the 4D aggregate cost body to the same size as the 3D matching cost body to obtain the 4D compressed cost body.

[0091] Understandably, reference Figure 5 As shown, in step S500, the 4D compressed cost volume at the corresponding scale is connected to the 3D matching cost volume to obtain a hybrid matching cost volume at multiple scales, including:

[0092] S510. Connect the 4D compressed cost volume and the 3D matching cost volume under the first scale to obtain the first hybrid matching cost volume corresponding to the first scale.

[0093] S520. Connect the 4D compressed cost volume and the 3D matching cost volume at the second scale to obtain the second hybrid matching cost volume corresponding to the second scale.

[0094] S530. Connect the 4D compression cost volume and the 3D matching cost volume at the third scale to obtain the third hybrid matching cost volume corresponding to the third scale, wherein the first scale > the second scale > the third scale.

[0095] Among them, the hybrid matching cost body The calculation can be expressed as:

[0096]

[0097] Understandably, reference Figure 6 As shown, in step S600, the hybrid matching cost volume is input into the 2D cost aggregation model for disparity calculation to obtain a disparity map, including:

[0098] S610. The first hybrid matching cost body is regularized by the first 2D convolutional module in the 2D cost aggregation model to obtain a regularized cost body, wherein the first 2D convolutional module has a skip connection function.

[0099] S620. The regularized cost volume is downsampled using a 2D cost aggregation model to obtain the first sampled cost volume at the second scale.

[0100] S630. The first sampled cost body and the second hybrid matching cost body are connected through a 2D cost aggregation model to obtain the first connection cost body;

[0101] S640. The number of feature channels of the first connected cost body is reduced to a specified value by the second 2D convolution module in the 2D cost aggregation model to obtain a low-channel cost body, wherein the number of feature channels of the low-channel cost body is equal to the number of channels of the regularized cost body.

[0102] S650. The low-channel cost volume is downsampled using a 2D cost aggregation model to obtain the second sampled cost volume at the third scale.

[0103] S660. The second sampling cost body and the third hybrid matching cost body are connected through a 2D cost aggregation model to obtain the second connection cost body;

[0104] S670. Upsample the second connection cost volume by using the third 2D convolution module in the 2D cost aggregation model to obtain the third sampled cost volume at the first scale.

[0105] S680. The disparity is calculated on the third sampled cost volume using a 2D cost aggregation model to obtain a disparity map.

[0106] It is understandable that the framework diagram of the network used in the hybrid cost-volume binocular stereo matching method is referenced. Figure 7 As shown, the 3D matching cost body downsamples the left and right stereo images to half resolution using three 3×3 convolution operations. Then, it learns unary feature extraction through four residual blocks. The three extracted feature maps are concatenated to form a unary feature map with a resolution of 1 / 4 and 320 channels. Then, two residual blocks with a stride of 2 are used to operate on this unary feature map to form feature maps with resolutions of 1 / 8 and 1 / 16, with corresponding feature channels of 160 and 80, respectively. These two feature maps are used to construct the relevant cost body, i.e., the 3D matching cost body. The previous 320-channel unary feature map is compressed to 24 channels. Then, this 1 / 4 resolution, 24-channel unary feature map is used to generate 1 / 8 resolution and 1 / 16 resolution feature maps with feature channels of 12 and 6, respectively, through two residual blocks with a stride of 2. The above feature maps are concatenated to obtain the cascaded cost body, i.e., the 4D matching cost body.

[0107] refer to Figure 8 As shown, the 4D matching cost volume C is generated through a 3D convolution module. 4D Aggregation is performed to obtain a 4D aggregated cost body. The 4D aggregated cost body is then compressed to a size equal to that of the 3D matching cost body, resulting in a 4D compressed cost body, i.e., the compressed C. 4D The 4D compression cost volume and the 3D matching cost volume C 3D Perform a connection to obtain the hybrid matching cost body C. hybrid

[0108] The 2D cost aggregation model is a 2D CNN cost volume aggregation model, referenced. Figure 9As shown, the 2D CNN cost body aggregation model is trained by three 2D convolutional neural networks. Each 2D convolutional neural network can generate a disparity map and a loss function. The 2D CNN cost body aggregation model forms a stacked hourglass structure. The scales of the first mixed matching cost body, the second mixed matching cost body, and the third mixed matching cost body are 1 / 4, 1 / 8, and 1 / 16, respectively. In step 600, the working process of the 2D convolutional neural network in the 2D cost aggregation model is as follows: First, the first mixed matching cost body is regularized by four 2D convolutional layers with skip connections. The obtained regularized cost body is downsampled from the 1 / 4 scale to the 1 / 8 scale to obtain the first sampled cost body. The first sampled cost body is connected with the second mixed cost body to obtain the first connected cost body. Then, the feature channels of the connected cost body are reduced to a specified size by the 2D convolutional layer to obtain the low-channel cost body. Next, the same operation is used to downsample the low-channel cost body to the 1 / 16 scale to obtain the second downsampled cost body. The second downsampled cost body is connected with the third mixed matching cost body to obtain the second connected cost body. The second connected cost body is upsampled by the 2D transposed convolutional layer to obtain the third sampled cost body. During the training of the 2D CNN cost volume aggregation model, the total loss is calculated as a weighted sum of three losses. During the testing process, the last loss is used as the final loss to calculate the disparity map.

[0109] Understandably, by calculating the disparity of the third sampled cost volume using a 2D cost aggregation model, a disparity map is obtained, including:

[0110] The third sample cost volume is bilinearly interpolated and then input into the disparity regression module in the 2D cost aggregation model to obtain the disparity map.

[0111] The calculation of the disparity regression module can be expressed by the following formula:

[0112]

[0113] The probability of disparity d is derived from the predicted cost volume c through the Solftmax function σ(·). d The disparity prediction was calculated. It is the probability weighted sum of each disparity candidate.

[0114] The smoothL1 function is used to train the 2D CNN cost volume aggregation model. SmoothL1 is highly sensitive to outliers and noise, which helps prevent disparity discontinuities. The loss function of the 2D CNN cost volume aggregation model is defined as:

[0115]

[0116] Where N is the number of pixels in the disparity map, d iFor the true disparity value, This represents the predicted disparity value.

[0117] The smoothL1 function is defined as follows:

[0118]

[0119] In addition, a second aspect of the present invention provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0120] The processor and memory can be connected via a bus or other means.

[0121] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The non-transient software program and instructions required to implement the binocular stereo matching method of the hybrid cost body in the first aspect embodiment are stored in the memory. When executed by the processor, the binocular stereo matching method of the hybrid cost body in the above embodiment is executed. For example, the method steps S100 to S600, S210 to S240, S310 to S330, S410 to S420, S510 to S530, and S610 to S680 described above are executed.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, a third aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described device embodiment, causing the processor to perform the binocular stereo matching method of the hybrid cost body in the above-described embodiment, for example, performing the method steps S100 to S600, S210 to S240, S310 to S330, S410 to S420, S510 to S530, and S610 to S680 described above.

[0125] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0126] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A binocular stereo matching method for hybrid cost bodies, characterized in that, include: Obtain left and right stereo image pairs; Multi-scale feature extraction is performed on the left and right stereo image pairs to obtain feature map pairs at multiple scales; Based on the feature map pairs at each scale, construct the 3D matching cost volume and 4D matching cost volume for the corresponding scale; The 4D matching cost body is aggregated and compressed to the same size as the 3D matching cost body to obtain a 4D compressed cost body; By connecting the 4D compressed cost body at the corresponding scale with the 3D matching cost body, a hybrid matching cost body at multiple scales is obtained; The hybrid matching cost volume is input into a 2D cost aggregation model for disparity calculation to obtain a disparity map; The step of connecting the 4D compressed cost volume at the corresponding scale with the 3D matching cost volume to obtain a hybrid matching cost volume at multiple scales includes: The 4D compressed cost body at the first scale is connected to the 3D matching cost body to obtain the first hybrid matching cost body corresponding to the first scale. The 4D compressed cost body at the second scale is connected to the 3D matching cost body to obtain the second hybrid matching cost body corresponding to the second scale. The 4D compressed cost body at the third scale is connected to the 3D matching cost body to obtain the third hybrid matching cost body corresponding to the third scale; wherein, the first scale > the second scale > the third scale; The step of inputting the hybrid matching cost volume into a 2D cost aggregation model for disparity calculation to obtain a disparity map includes: The first hybrid matching cost body is regularized by the first 2D convolution module in the 2D cost aggregation model to obtain a regularized cost body; The regularized cost volume is downsampled using the 2D cost aggregation model to obtain the first sampled cost volume at the second scale. The first sampled cost body and the second hybrid matching cost body are connected through the 2D cost aggregation model to obtain the first connection cost body; The number of feature channels of the first connected cost body is reduced to a specified value by the second 2D convolution module in the 2D cost aggregation model to obtain a low-channel cost body. The low-channel cost volume is downsampled using the 2D cost aggregation model to obtain the second sampled cost volume at the third scale. The second sampling cost body is obtained by connecting the third hybrid matching cost body through the 2D cost aggregation model. The second connection cost volume is upsampled by the third 2D convolution module in the 2D cost aggregation model to obtain the third sampled cost volume at the first scale. The disparity map is obtained by calculating the disparity of the third sampled cost volume using the 2D cost aggregation model.

2. The binocular stereo matching method for hybrid cost bodies according to claim 1, characterized in that, The step of performing multi-scale feature extraction on the left and right stereo image pairs to obtain feature map pairs at multiple scales includes: The left and right stereoscopic image pairs are downsampled to obtain the first sampled image pair; Multi-scale feature extraction is performed on the first sampled image pair to obtain multiple initial feature image pairs; Each of the initial feature map pairs is concatenated to obtain a first-scale feature map pair; By extracting features from the first scale feature map pair using residual blocks, we obtain the second scale feature map pair and the third scale feature map pair.

3. The binocular stereo matching method for hybrid cost bodies according to claim 1, characterized in that, The construction of the 3D matching cost volume at the corresponding scale includes: The view correlation is calculated based on the left and right stereoscopic images; Based on the view relevance and the feature map pairs at each scale, a 3D matching cost body is constructed for the corresponding scale, wherein the size of the 3D matching cost body is [missing information]. , Indicates the parallax range. Indicates the size of the space.

4. The binocular stereo matching method for hybrid cost bodies according to claim 3, characterized in that, The construction of the 4D matching cost volume at the corresponding scale includes: The feature map pairs at each scale are concatenated to obtain the 4D matching cost body at the corresponding scale, wherein the size of the 4D matching cost body is [size missing]. , This indicates the number of channels after feature concatenation.

5. The binocular stereo matching method for hybrid cost bodies according to claim 4, characterized in that, The step of aggregating and compressing the 4D matching cost body to the same size as the 3D matching cost body to obtain a 4D compressed cost body includes: The 4D matching cost volume is aggregated using a 3D convolution module to obtain a 4D aggregated cost volume; The 4D aggregate cost body is compressed to the same size as the 3D matching cost body to obtain the 4D compressed cost body.

6. The binocular stereo matching method for hybrid cost bodies according to claim 1, characterized in that, The step of calculating disparity on the third sampled cost volume using the 2D cost aggregation model to obtain the disparity map includes: The third sampling cost volume is bilinearly interpolated and then input into the disparity regression module in the 2D cost aggregation model to obtain the disparity map.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a binocular stereo matching method for a hybrid cost body as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing a binocular stereo matching method for a hybrid cost body as described in any one of claims 1 to 5.

Images