A stereo matching method, device, equipment and computer readable storage medium

Abstract

The application discloses a stereo matching method, device and equipment and a computer readable storage medium. The hidden state is decoupled from the update matrix of the disparity map through a multi-scale decoupling LSTM network, more high-frequency information is retained, and further based on normalized refinement processing, information from the up-sampled disparity map, i.e. the original left and right images containing high-frequency information, is fully utilized to enhance edges and details, solving the problem that in the original GRU structure, the information of the update matrix for generating the disparity map is coupled with the hidden state transition value between iterations, so that high-frequency details in the hidden state are difficult to retain. Meanwhile, in order to balance the performance and the calculation speed, the resolution of the iteration stage is at most 1 / 4 of the original resolution, so that it is difficult to produce a disparity map with sharp edges and subtle details.

Description

Technical Field

[0001] This application relates to the field of binocular stereo matching technology, and in particular to a stereo matching method, apparatus, device and computer-readable storage medium. Background Technology

[0002] Stereo depth estimation is becoming the infrastructure for 3D applications. Accurate depth perception is crucial for autonomous driving, drone navigation, robotics, and other related fields. The key to this task is estimating a pixel-level displacement map, also known as parallax, which can be used to determine the depth of pixels in a scene.

[0003] Traditional stereo matching algorithms are mainly divided into two types: global methods and local methods. Both methods solve the optimization problem by minimizing an objective function that includes data and a smoothing term. The former considers global information, while the latter only considers local information. Therefore, each has its advantages in terms of accuracy and speed when solving optimization problems. Traditional methods have good generalization performance and robustness in different scenarios, but they perform poorly in regions with weak textures and repetitive textures.

[0004] With the development of convolutional neural networks, learning-based methods have recently shown good results in solving challenging region matching problems. Utilizing the strong regularization properties of 3D convolution and 4D cost convolution, methods using 3D convolution perform well. However, their practical applicability is limited by high computational costs. Subsequent methods have attempted to replace 3D convolution with multiple adaptive and guided aggregation 2D convolutions, reducing computational costs and achieving better performance.

[0005] The recent emergence of iterative methods has brought a new concept to stereo matching research. Iterative methods employ a coarse-to-fine iterative refinement approach. It first calculates the correlation between all pixel pairs, constructing a 3D correlation pyramid. Then, an update optimizer with a convolutional GRU as its core unit is used to retrieve features from the correlation pyramid and update the disparity map.

[0006] Most current iterative methods typically use the original GRU structure as their iteration unit. The problem is that in the original GRU structure, the information in the update matrix used to generate the disparity map is coupled with the hidden state transition values ​​between iterations, making it difficult to preserve high-frequency details in the hidden states. Furthermore, to balance performance and computational speed, the resolution of the iteration stage is at most only 1 / 4 of the original resolution, making it difficult to generate disparity maps with sharp edges and subtle details. Summary of the Invention

[0007] This application provides a stereo matching method, apparatus, device, and computer-readable storage medium, which solves the problem in the original GRU structure where the information of the update matrix used to generate the disparity map is coupled with the hidden state transition values ​​between iterations, making it difficult to preserve high-frequency details in the hidden state. Simultaneously, to balance performance and computational speed, the resolution in the iteration stage is at most only 1 / 4 of the original resolution, making it difficult to generate disparity maps with sharp edges and subtle details.

[0008] In view of the above, the first aspect of this application provides a stereo matching method, the method comprising:

[0009] S1, the left image I after epipolar correction l And right image I r The left image I is input into the feature extractor as an input image. l and the right image I r Feature F l F m and F h ;

[0010] S2, the left image I l Feature F l F m and F h The result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask.

[0011] S3. The first disparity map is normalized and refined using the upsampling mask to obtain the second disparity map.

[0012] Optionally, the feature processor is specifically a Channel-Attention Transformer feature extractor;

[0013] The Channel-Attention Transformer feature extractor includes: a PixelUnshuffle module for downsampling, a U-shaped structure composed of several CTBs, a Channel Self-Attention (CWSA) module, and a gated Dconv feedforward network (GDFN).

[0014] Optionally, the Pixel Unshuffle module specifically involves: transferring the left image I... l The original shape [C,H*r,W*r] is reshaped into [C*r] 2 [,H,W], where C is the number of channels in the original input image, H is the height of the reconstructed image, W is the width of the reconstructed image, and r is the pixel inverse reconstruction parameter.

[0015] Optionally, the multi-scale decoupled LSTM network includes:

[0016] An iterative module consisting of iterative sub-modules of three different scales of LSTM structure;

[0017] The iterative module will use the features F of the left image l F m and F h And the hidden state h derived from the feature map i C i Parallax map D updated in the previous stage i-1 As input, in updating matrix ΔD i When the value is 0, output the first disparity map D of the target resolution. lr And an upsampling mask (upmask), otherwise continue iterating.

[0018] Optionally, step S3 specifically includes:

[0019] S31, the first disparity map D lr The upsampling mask upmask is upsampled using learned upsampling to obtain a full-resolution disparity map D. fr ;

[0020] S32. Use the warp function to modify the right image I. r Converted to the left image I l And calculate the error map E. l ;

[0021] S33, the left image I l Using the pixel width as the denominator, based on the full-resolution disparity map D... fr and min(D) fr Calculate the maximum disparity map by the difference between the two values. Where min(D) fr ) is 0;

[0022] S34. Combine and process the maximum disparity map using an hourglass network. The error map E l and the left image I l The normalized, refined disparity map D is obtained. fr` ;

[0023] S35, refine the disparity map D fr` Perform inverse normalization to obtain the second disparity map D. refined .

[0024] Optionally, step S34 specifically includes:

[0025] Based on the error map El and the left image I l The left image error I was calculated. err I err =Conv 3×3 ([E l ,I l ]);

[0026] The maximum disparity map is combined and processed using an hourglass network. and the left image error I err The normalized, refined disparity map D is obtained. fr` ,

[0027] Optionally, it also includes:

[0028] S4. Supervise the feature extractor, the multi-scale decoupled LSTM network, and the normalization and thinning process using a preset set of formulas. Specifically, the preset set of formulas includes:

[0029]

[0030] L1=||d gt -d i ||1

[0031] L refine =||d gt -d refined ||1;

[0032] Where, d gt For true parallax, d i To predict parallax, d refined This is the refined predicted parallax.

[0033] A second aspect of this application provides a stereo matching device, the device comprising:

[0034] Image processing unit, used to process the left image I after epipolar correction. l And right image I r The left image I is input into the feature extractor as an input image. l and the right image I r Feature F l F m and F h ;

[0035] The decoupled convolution processing unit is used to process the left image I l Feature F l F m and F hThe result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask.

[0036] The normalization and refinement processing unit is used to perform normalization and refinement processing on the first disparity map using the upsampling mask to obtain the second disparity map.

[0037] A third aspect of this application provides a stereo matching device, the device comprising a processor and a memory:

[0038] The memory is used to store program code and transmit the program code to the processor;

[0039] The processor is configured to execute the steps of the stereo matching method as described in the first aspect above, according to the instructions in the program code.

[0040] A fourth aspect of this application provides a computer-readable storage medium for storing program code for performing the steps of the stereo matching method described in the first aspect.

[0041] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0042] This application provides a stereo matching method, apparatus, device, and computer-readable storage medium. It decouples the hidden state from the disparity map update matrix using a multi-scale decoupled LSTM network, preserving more high-frequency information. Furthermore, it utilizes normalized thinning processing to fully leverage information from the upsampled disparity map—that is, the original left and right images containing high-frequency information—to enhance edges and details. This solves the problem in the original GRU structure where the information in the update matrix used to generate the disparity map is coupled with the hidden state transition values ​​between iterations, making it difficult to preserve high-frequency details in the hidden state. Simultaneously, to balance performance and computational speed, the resolution in the iteration stage is at most 1 / 4 of the original resolution, making it difficult to generate disparity maps with sharp edges and subtle details. Attached Figure Description

[0043] Figure 1 This is a flowchart of the stereo matching method in the embodiments of this application;

[0044] Figure 2 This is a schematic diagram of the structure of the stereo matching device in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram of the structure of the stereo matching device in the embodiments of this application;

[0046] Figure 4 This is a schematic diagram of the overall model of the stereo matching method in the embodiments of this application;

[0047] Figure 5 This is a schematic diagram of the Channel-Attention Transformer feature extractor in the embodiments of this application;

[0048] Figure 6 This is a schematic diagram of the structure of the multi-scale decoupled LSTM network in the embodiments of this application;

[0049] Figure 7 This is a schematic diagram of the normalization and refinement processing module in an embodiment of this application. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0051] This application designs a stereo matching method, apparatus, device, and computer-readable storage medium, which solves the problem in the original GRU structure where the information of the update matrix used to generate the disparity map is coupled with the hidden state transition values ​​between iterations, making it difficult to preserve high-frequency details in the hidden state. Simultaneously, to balance performance and computational speed, the resolution in the iteration stage is at most only 1 / 4 of the original resolution, making it difficult to generate disparity maps with sharp edges and subtle details.

[0052] For easier understanding, please refer to Figure 1 , Figure 1 This is a flowchart of the stereo matching method in the embodiments of this application, such as... Figure 1 As shown, specifically:

[0053] S1, the left image I after epipolar correction l And right image I r The input image is fed into the feature extractor to obtain the left image I. l And right image I r Feature F l F m and F h ;

[0054] in:

[0055] The feature processor is specifically the Channel-Attention Transformer feature extractor;

[0056] The Channel-Attention Transformer feature extractor includes: a PixelUnshuffle module for downsampling, a U-shaped structure composed of several CTBs, a Channel Self-Attention (CWSA) module, and a gated Dconv feedforward network (GDFN).

[0057] The Pixel Unshuffle module specifically works by: transferring the left image I... l The original shape [C,H*r,W*r] is reshaped into [C*r] 2 [,H,W], where C is the number of channels in the original input image, H is the height of the reconstructed image, W is the width of the reconstructed image, and r is the pixel inverse reconstruction parameter.

[0058] It should be noted that, as Figure 4 and Figure 5 As shown, in the field of stereo matching, feature extractors have not seen significant improvements over the years. Most learning-based methods still use feature extractors similar to ResNet, and the performance of these feature extractors becomes a bottleneck when providing information for carefully designed later-stage structures.

[0059] To alleviate this problem, we propose the Channel-Attention Transformer feature extractor, which aims to capture long-range pixel dependencies and preserve high-frequency information.

[0060] In the Channel-Attention Transformer feature extractor, a series of CTBs are used to form a U-shaped structure, outputting multi-scale and multi-stage features; the Pixel Unshuffle module is used for downsampling without losing any high-frequency details.

[0061] To achieve sharp edges and better handle weakly textured regions, preserving high-frequency information throughout the model's lifecycle is crucial. The simplest and most intuitive approach is to use full resolution across the entire structure, but this incurs extremely high computational costs. Convolutions with spanning or downsampling using pooling mechanisms inevitably lead to information loss and performance degradation.

[0062] It employs Pixel Unshuffle technology, which can downsample images to 1 / 4 of their original size and expand channels without losing any high-frequency information.

[0063] Traditional self-attention mechanisms generate an attention map, which leads to quadratic complexity, making it impractical for vision tasks requiring high resolution. Therefore, we employ the CWSA module, which computes self-attention along the channel dimension with linear complexity.

[0064] S2, Move the left image I l Feature F l F m and F h The result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask.

[0065] Among them: such as Figure 6 As shown, the multi-scale decoupled LSTM network includes:

[0066] An iterative module consisting of iterative sub-modules of three different scales of LSTM structure;

[0067] The iterative module will use the features F of the left image l F m and F h And the hidden state h derived from the feature map i C i Parallax map D updated in the previous stage i-1 As input, in updating matrix ΔD i When the value is 0, output the first disparity map D of the target resolution. lr And an upsampling mask (upmask), otherwise continue iterating.

[0068] It's important to note that in stereo matching tasks, the patterns in textureless regions are weak, making it difficult to find corresponding pixels. Therefore, capturing the patterns of spatially adjacent pixels becomes a crucial part of the problem. We address this issue by designing our iterative module across multiple scales.

[0069] Specifically, the iterative module consists of three iterative sub-modules at different scales, corresponding to 1 / 4, 1 / 8, and 1 / 16 of the image resolution, respectively. Each module interacts with its adjacent resolution. The low-resolution branch has a larger equivalent receptive area and can better handle textureless regions, while the high-resolution branch captures more high-frequency details, adding more detail to the edges of the disparity map.

[0070] Furthermore, in the original GRU structure used in most iterative visual networks, the hidden state h is used to generate the disparity update matrix ΔD. i (The output of the GRU unit) is also the hidden state h of the GRU network (passing information to the next iteration). In our ablation experiments, this coupling problem was shown to have a significant impact on network performance.

[0071] An additional hidden state C is introduced, and the hidden state h is used to generate the update matrix ΔD from the disparity map. i The newly introduced hidden state C is used only for iterative information transmission. This design updates the matrix ΔD. iDecoupling from hidden states allows for the preservation of more effective semantic information during iteration.

[0072] S3. The first disparity map is normalized and refined using an upsampling mask to obtain the second disparity map;

[0073] Specifically, it includes:

[0074] S31, the first disparity map D lr The upsampling mask (upmask) is upsampled using learned upsampling to obtain the full-resolution disparity map (D). fr ;

[0075] S32. Use the warp function to transform the right image I r Convert to left image I l And calculate the error map E. l ;

[0076] S33, Move the left image I l Using the pixel width as the denominator, based on the full-resolution disparity map D... fr and min(D) fr Calculate the maximum disparity map by the difference between the two values. Where min(D) fr ) is 0;

[0077] S34. Combining and processing the maximum disparity map using an hourglass network. Error plot E l and left image I l The normalized, refined disparity map D is obtained. fr` ;

[0078] Specifically:

[0079] Based on error map E l and left image I l The left image error I was calculated. err I err =Conv 3×3 ([E l ,I l ]);

[0080] Maximum disparity map is combined and processed using hourglass networks. and left image error I err The normalized, refined disparity map D is obtained. fr` ,

[0081] S35, Refine the disparity map D fr` Perform inverse normalization to obtain the second disparity map D. refined .

[0082] It should be noted that because our model performs regularization at the downsampling resolution, high-frequency information cannot be fully preserved in this process. Therefore, we designed a method such as... Figure 7 The normalization and refinement module shown is designed to capture more details at full resolution.

[0083] In relatively independent modules, due to domain differences, the output of the feature maps may all be negative. After passing through the ReLU activation function, the feature maps all become zero, preventing the network from fine-tuning these parameters. The network can only pass the feature information to subsequent modules by skipping connections. This leads to some modules not being able to be fine-tuned well after network pre-training, and even encountering module failure issues when fine-tuning on other datasets.

[0084] The disparity map at 1 / 4 resolution is first upsampled using learned upsampling, then the right image is converted to the left image using the warp function, and the error map is calculated. Specifically:

[0085] D fr =learnedUpsample(D lr ,upMask)

[0086] I` l =warp(I r (disp)

[0087] E l =I` l -I r ;

[0088] Among them, D fr For a full-resolution disparity map, D lr This is the first disparity map before upsampling.

[0089] Typically, min(D) fr The maximum disparity value is obtained by using the pixel width of the left image as the denominator when the value is equal to 0.

[0090]

[0091] Then normalize the disparity map Error plot E l and left image I l This generates a normalized, refined disparity map D. fr` :

[0092]

[0093] Finally, inverse normalization is performed to generate the final second disparity map:

[0094] D refined=D fr` ×width(I l )+min(D fr` ).

[0095] Furthermore, it also includes:

[0096] S4. Supervise the feature extractor, multi-scale decoupled LSTM network, and normalization refinement process using a set of preset formulas. The preset formula set is as follows:

[0097]

[0098] L1=||d gt -d i ||1

[0099] L refine =||d gt -d refined ||1;

[0100] Where, d gt For true parallax, d i To predict parallax, d refined This is the refined predicted parallax.

[0101] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of the stereo matching device in the embodiments of this application, such as... Figure 2 As shown, specifically:

[0102] Image processing unit 201 is used to process the left image I after a pair of epipolar corrections. l And right image I r The input image is fed into the feature extractor to obtain the left image I. l And right image I r Feature F l F m and F h ;

[0103] Decoupling convolution processing unit 202 is used to process the left image I l Feature F l F m and F h The result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask.

[0104] The normalization and refinement processing unit 203 is used to perform normalization and refinement processing on the first disparity map through an upsampling mask to obtain the second disparity map.

[0105] This application also provides another stereo matching device, such as... Figure 3As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0106] Figure 3 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090, etc. Those skilled in the art will understand that... Figure 3 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0107] The following is combined with Figure 3 A detailed introduction to each component of a mobile phone:

[0108] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0109] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0110] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1080, and can also receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0111] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 1041. Further, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0112] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0113] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.

[0114] WiFi is a short-range wireless transmission technology. Through the WiFi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 3 The WiFi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0115] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020 and calls data stored in the memory 1020 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1080.

[0116] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0117] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0118] In this embodiment of the application, the processor 1080 included in the terminal also has the following functions:

[0119] S1, the left image I after epipolar correction l And right image I r The input image is fed into the feature extractor to obtain the left image I. l And right image I r Feature F l F m and F h ;

[0120] S2, Move the left image I l Feature F l F m and F h The result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask.

[0121] S3. The first disparity map is normalized and refined by using an upsampling mask to obtain the second disparity map.

[0122] This application also provides a computer-readable storage medium for storing program code that executes any one of the implementation methods of the stereo matching method described in the foregoing embodiments.

[0123] This application provides a stereo matching method, apparatus, device, and computer-readable storage medium. By decoupling the hidden state from the disparity map update matrix through a multi-scale decoupled LSTM network, more high-frequency information is preserved. Furthermore, normalization thinning is used to fully utilize information from the upsampled disparity map—that is, the original left and right images containing high-frequency information—to enhance edges and details. This solves the problem in the original GRU structure where the information in the update matrix used to generate the disparity map is coupled with the hidden state transition values ​​between iterations, making it difficult to preserve high-frequency details in the hidden state. Simultaneously, to balance performance and computational speed, the resolution in the iteration stage is at most 1 / 4 of the original resolution, making it difficult to generate disparity maps with sharp edges and subtle details.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0125] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0126] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0131] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A stereo matching method, characterized in that, include: S1, the left image after polar line correction. And the right image The left image is input into the feature extractor as an input image. and the right image Features , and ; S2, the left image Features , and The result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask. S3. The first disparity map is normalized and refined using the upsampling mask to obtain the second disparity map; The multi-scale decoupled LSTM network includes: An iterative module consisting of iterative sub-modules of three different scales of LSTM structure; The iterative module will use the features of the left image , and and the hidden states derived from feature maps , Parallax map updated in the previous phase As input, in updating the matrix Output the first disparity map of the target resolution when the value is 0. and upsampling mask Otherwise, continue iterating. Step S3 specifically includes: S31, the first disparity map and the upsampling mask Upsampling is performed using learned upsampling to obtain a full-resolution disparity map. ; S32. Use the warp function to process the right image. Converted to the left image And calculate the error graph. ; S33, the left image Using the pixel width as the denominator, based on the full-resolution disparity map. as well as Calculate the difference in the maximum disparity map ,in =0; S34. Combine and process the maximum disparity map using an hourglass network. The error diagram and the left image The normalized, refined disparity map is obtained. ; S35. The refined disparity map Perform inverse normalization to obtain the second disparity map. .

2. The stereo matching method according to claim 1, characterized in that, The feature extractor is specifically a Channel-Attention Transformer feature extractor; The Channel-Attention Transformer feature extractor includes: a PixelUnshuffle module for downsampling, a U-shaped structure composed of several CTBs, a Channel Self-Attention (CWSA) module, and a gated Dconv feedforward network (GDFN).

3. The stereo matching method according to claim 2, characterized in that, The Pixel Unshuffle module specifically performs the following function: to unshuffle the left image... original shape Remodeling ,in, The number of channels in the original input image. The height of the reconstructed image. The width of the reshaped image. These are the parameters for pixel inverse reconstruction.

4. The stereo matching method according to claim 1, characterized in that, Step S34 specifically includes: Based on the error map and the left image The error of the left image was calculated. , ; The maximum disparity map is combined and processed using an hourglass network. and the left image error The normalized, refined disparity map is obtained. , .

5. The stereo matching method according to claim 1, characterized in that, Also includes: S4. Supervise the feature extractor, the multi-scale decoupled LSTM network, and the normalization and thinning process using a preset set of formulas. Specifically, the preset set of formulas includes: ； in, For true parallax, To predict parallax, This is the refined predicted parallax.

6. A stereo matching device, characterized in that, include: Image processing unit for processing a pair of epipolar-corrected left images And the right image The left image is input into the feature extractor as an input image. and the right image Features , and ; The decoupled convolution processing unit is used to process the left image. Features , and The result is passed to a multi-scale decoupled LSTM network to obtain a first disparity map of the target resolution and an upsampling mask. The normalization and refinement processing unit is used to perform normalization and refinement processing on the first disparity map using the upsampling mask to obtain the second disparity map; The multi-scale decoupled LSTM network includes: An iterative module consisting of iterative sub-modules of three different scales of LSTM structure; The iterative module will use the features of the left image , and and the hidden states derived from feature maps , Parallax map updated in the previous phase As input, in updating the matrix Output the first disparity map of the target resolution when the value is 0. and upsampling mask Otherwise, continue iterating. The normalization and refinement processing unit is specifically used for: The first disparity map and the upsampling mask Upsampling is performed using learned upsampling to obtain a full-resolution disparity map. ; The right image is processed using the warp function. Converted to the left image And calculate the error graph. ; The left image Using the pixel width as the denominator, based on the full-resolution disparity map. as well as Calculate the difference in the maximum disparity map ,in =0; The maximum disparity map is combined and processed using an hourglass network. The error diagram and the left image The normalized, refined disparity map is obtained. ; For the refined disparity map Inverse normalization is performed to obtain the second disparity map.

7. A stereo matching device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the stereo matching method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the stereo matching method according to any one of claims 1-5.

Images