Stereo matching method and device

Abstract

The present invention discloses a stereo matching method and device, belonging to the field of image processing technology. The method comprises: extracting features from an input image to obtain a multi-scale feature map and a context feature map; fusing high-frequency and low-frequency features from the multi-scale feature map to obtain a fused feature map; dividing the fused feature map into multiple feature groups to obtain an intra-group correlation body of each feature group and a mean correlation body of the fused feature map; obtaining an initial cost volume according to the intra-group correlation body and the mean correlation body; calculating the similarity between each pixel in the cost volume and the neighboring pixel information of the pixel to obtain an affinity relationship, updating the initial cost volume using the affinity relationship to obtain a cost volume with an affinity relationship constraint; inputting the context feature map, the initial cost volume and the cost volume with an affinity relationship constraint into a gated recurrent unit for iterative updating to obtain a final disparity map. The present invention can improve the accuracy of disparity estimation.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular, relates to a stereo matching method and device. Background Art

[0002] Stereo matching is a fundamental research area in computer vision, which seeks to determine the disparity between corresponding pixels in a pair of images, thereby reconstructing the depth information of a stereoscopic scene. Its applications span factory automation, robot navigation, and autonomous driving, demonstrating its key role in the advancement of modern technology.

[0003] In the stereo matching methods of related technologies, it is difficult to characterize stereo matching features in complex scenes, and correlation information is not fully utilized, which makes it challenging to deal with local ambiguity in the neighborhood, resulting in errors in matching edge details and affecting the accuracy of disparity estimation. Summary of the invention

[0004] The purpose of the present invention is to provide a stereo matching method and device to achieve the technical purpose of improving the accuracy of disparity estimation.

[0005] In order to achieve the above-mentioned purpose of the invention, the stereo matching method provided by the present invention is implemented by adopting the following technical scheme:

[0006] A stereo matching method, comprising:

[0007] Extract features from the input left and right images using residual blocks and downsampling to obtain a multi-scale feature map; re-extract the multi-scale feature map to obtain a context feature map; fuse high- and low-frequency features of the multi-scale feature map to obtain a fused feature map;

[0008] The fused feature map is divided into a plurality of feature groups along the channel dimension, and in each feature group, an inner product is used to obtain an intra-group correlation body; the product mean of the feature image pairs in the fused feature map on all channels is calculated to obtain a mean correlation body; the intra-group correlation body and the mean correlation body are spliced ​​to obtain an initial cost volume; a neighborhood search is performed on each pixel in the initial cost volume to obtain neighborhood pixel information of each pixel; the similarity between each pixel and the neighborhood pixel information of the pixel is calculated to obtain an affinity relationship, and the initial cost volume is updated using the affinity relationship to obtain a cost volume with affinity relationship constraints;

[0009] The context feature map, the initial cost volume, and the cost volume with affinity relationship constraints are input into a gated recurrent unit for iterative updating to obtain a final disparity map.

[0010] In some embodiments of the present application, calculating the similarity between each pixel and the neighboring pixel information of the pixel includes:

[0011] A structured attention model is used to calculate the attention weight of each neighboring pixel of each pixel;

[0012] The similarity between each pixel and the neighborhood pixel information of the pixel is calculated based on the neighborhood pixel information and the attention weight of each neighborhood pixel.

[0013] In some embodiments of the present application, the initial cost volume is updated using the affinity relationship to obtain a cost volume with affinity relationship constraints, including:

[0014] The initial cost volume is updated using the weight matrix and the affinity relationship to obtain a cost volume with affinity relationship constraints.

[0015] In some embodiments of the present application, the intra-group correlation body and the mean correlation body are concatenated to obtain an initial cost volume, including:

[0016] Performing pooling operations on the intra-group correlation entity and the mean correlation entity respectively to obtain a pooled intra-group correlation entity and a pooled mean correlation entity;

[0017] The pooled intra-group correlation body and the pooled mean correlation body are concatenated to obtain the initial cost volume.

[0018] In some embodiments of the present application, the context feature map, the initial cost volume, and the cost volume with affinity constraint are input into a gated recurrent unit for iterative update to obtain a final disparity map, including:

[0019] Initializing a first hidden state of the gated recurrent unit using the context feature map of the image, the initial disparity estimate, and the initial cost volume, and initializing a second hidden state of the gated recurrent unit using the cost volume with affinity relationship constraints;

[0020] The gated recurrent unit performs the following iterative updates based on the initialized first hidden state and the initialized second hidden state: in each iteration, the current first hidden state is used as the input of the gated recurrent unit, and the current first hidden state is updated by controlling the update gate and the reset gate of the gated recurrent unit to obtain an updated first hidden state; a weight matrix is ​​generated based on the second hidden state, and the disparity estimate in the updated first hidden state is adjusted by using the weight matrix to obtain an adjusted disparity estimate; and the disparity estimate in the updated first hidden state is replaced by the adjusted disparity estimate to obtain a first hidden state of the next iteration;

[0021] After the iterative update is completed, the final disparity map is obtained.

[0022] In some embodiments of the present application, feature extraction is performed on the input left and right images to obtain a multi-scale feature map, including:

[0023] The input left and right images are respectively normalized to obtain the normalized images;

[0024] The standardized image is subjected to feature extraction using a residual block and downsampling to obtain the multi-scale feature map.

[0025] In some embodiments of the present application, high-frequency and low-frequency features are fused on the multi-scale feature map to obtain a fused feature map, including:

[0026] A high-pass filter is used to filter the multi-scale feature map to obtain high-frequency features; a low-pass filter is used to filter the multi-scale feature map to obtain low-frequency features;

[0027] The high-frequency features and the low-frequency features are weightedly fused using an adaptive weight sliding window to obtain the fused feature map.

[0028] In order to achieve the above-mentioned object of the invention, the stereo matching device provided by the present invention adopts the following scheme to achieve:

[0029] A stereo matching device, comprising:

[0030] A multi-scale feature map acquisition unit is used to extract features from the input left and right images using residual blocks and downsampling to obtain a multi-scale feature map;

[0031] A context feature map acquisition unit, used to re-extract the multi-scale feature map to obtain a context feature map;

[0032] A fused feature map acquisition unit, used for fusing high-frequency and low-frequency features of the multi-scale feature map to obtain a fused feature map;

[0033] The cost volume acquisition unit is used to divide the fused feature map into multiple feature groups along the channel dimension, and in each feature group, use the inner product to obtain the intra-group correlation body; further calculate the product mean of the feature image pair in the fused feature map on all channels to obtain the mean correlation body; splice the intra-group correlation body and the mean correlation body to obtain the initial cost volume; perform a neighborhood search on each pixel in the initial cost volume to obtain the neighborhood pixel information of each pixel; calculate the similarity between each pixel and the neighborhood pixel information of the pixel to obtain the affinity relationship, and use the affinity relationship to update the initial cost volume to obtain the cost volume with the affinity relationship constraint;

[0034] The iterative updating unit is used to input the context feature map, the initial cost volume and the cost volume with affinity relationship constraint into the gated recurrent unit for iterative updating to obtain a final disparity map.

[0035] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the above-mentioned stereo matching method is implemented.

[0036] Compared with the prior art, the advantages and positive effects of the present invention are:

[0037] The stereo matching method and device provided by the present invention extract features from an input image to obtain a multi-scale feature map and a context feature map, which can capture global context information; then perform high-frequency and low-frequency feature fusion on the multi-scale feature map to enhance feature extraction, which can capture clearer and more detailed image edge features and eliminate unnecessary noise; for the acquired fused feature map, calculate the intra-group correlation body and the mean correlation body, then combine the two correlation bodies to form an initial cost volume, and obtain an affinity relationship by calculating the similarity between a pixel and its neighborhood pixel information, and use the affinity relationship to update the initial cost volume to obtain a cost volume with an affinity relationship constraint, thereby introducing an affinity relationship constraint into the cost volume and more accurately determining the cost of edge matching; then use the acquired context feature map, the initial cost volume and the cost volume with the affinity relationship constraint as inputs of a gated recurrent unit for iterative updating, which can not only retain the required historical information to stabilize the disparity estimation, but also use the neighborhood pixel information to adjust the disparity update process, so that new information is processed preferentially in key areas, generate a disparity map that captures fine image details, and gradually improve the accuracy of disparity estimation. Therefore, the present invention effectively solves the problem of difficulty in characterizing stereo matching features in complex scenes and improves the accuracy of disparity estimation.

[0038] After reading the specific embodiments of the present invention in conjunction with the accompanying drawings, other features and advantages of the present invention will become more clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of an embodiment of a stereo matching method provided by the present invention;

[0041] Figure 2 A structural block diagram of an embodiment of a stereo matching device provided by the present invention;

[0042] Figure 3 An original image of an embodiment;

[0043] Figure 4 For Figure 3 A disparity map obtained by using the stereo matching method provided by the present invention;

[0044] Figure 5 The original images of other embodiments and the disparity maps obtained by applying the stereo matching method provided by the present invention to the original images. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0047] Figure 1 Shown is a flow chart of an embodiment of the stereo matching method provided by the present invention.

[0048] like Figure 1 As shown, this embodiment adopts the following process to obtain a disparity map and then implement stereo matching.

[0049] S11: Use residual blocks and downsampling to extract features from the input left and right images to obtain multi-scale feature maps.

[0050] The method of extracting features from an input image to obtain a multi-scale feature map may be implemented using relevant technologies, which is not limited in this embodiment.

[0051] S12: Perform feature re-extraction and feature fusion on the multi-scale feature map to obtain a context feature map and a fused feature map respectively.

[0052] The method of extracting features from a multi-scale feature map to obtain a context feature map may be implemented using relevant technologies, which is not limited in this embodiment.

[0053] The context feature map obtained by feature extraction can capture the global context information in the image, generate a more fine-grained context feature map, and assist in enhancing the semantic expression ability of multi-scale features.

[0054] High-frequency features and low-frequency features are also extracted according to the multi-scale feature map, and the high-frequency features and the low-frequency features are fused to obtain a fused feature map.

[0055] The method for extracting and fusing high- and low-frequency features from a multi-scale feature map may be implemented using relevant technologies, which are not limited in this embodiment.

[0056] S13: Obtain the cost volume based on the fused feature map.

[0057] In this step, the fused feature map obtained in the above step is first divided into multiple feature groups along the channel dimension, and each feature group contains multiple channels.

[0058] Then, correlation calculation is performed.

[0059] The correlation calculation includes two parts: one is the intra-group correlation body, which represents the correlation by calculating the correlation body group by group; the other is the mean correlation body, which represents the correlation by the mean in the channel dimension.

[0060] Specifically, within each feature group, the inner product is used to obtain the intra-group correlation entity.

[0061] For the feature image pairs in the fused feature map, the product mean of the image pairs on all channels is calculated to obtain the mean correlation volume.

[0062] Then, the intra-group correlation volume and the mean correlation volume are concatenated to obtain the initial cost volume, which contains multi-scale information of intra-group correlation and global mean correlation and has better recognition ability.

[0063] Then, the affinity features are introduced into the cost volume to form a cost volume with affinity relationship constraints.

[0064] Specifically, a neighborhood search is performed on each pixel in the initial cost volume to obtain the neighborhood pixel information of each pixel. The method for obtaining the neighborhood pixel information can be implemented using relevant technologies, for example, based on an adaptive search window, all pixels in the window are traversed to obtain the neighborhood pixel information. Then, each pixel is taken as the center pixel, and the similarity between the center pixel and its neighborhood pixel information is calculated to obtain an affinity relationship. The similarity between the center pixel and its neighborhood pixel information can be the square difference between the two pixel values. Finally, the affinity relationship is used to update the initial cost volume to obtain a cost volume with affinity relationship constraints.

[0065] The obtained cost volume with affinity constraint contains multi-scale correlation information and relative features, enhances the connection between adjacent pixels, can more accurately determine the cost of edge matching, and can obtain more accurate disparity estimation when used for subsequent stereo matching.

[0066] In some other embodiments, the intra-group correlation volume and the mean correlation volume are concatenated to obtain an initial cost volume, including:

[0067] The intra-group correlation entity and the mean correlation entity are pooled respectively to obtain the pooled intra-group correlation entity and the pooled mean correlation entity;

[0068] Then, the pooled intra-group correlation body and the pooled mean correlation body are concatenated to obtain an initial cost volume, which is a multi-scale cost volume.

[0069] In some other embodiments, the following method is used to calculate the similarity between each pixel and its neighboring pixel information:

[0070] A structured attention model is used to calculate the attention weight of each neighboring pixel of each pixel;

[0071] Based on the neighborhood pixel information and the attention weight of each neighborhood pixel, the similarity between each pixel and its neighborhood pixel information is calculated, thereby further strengthening the relationship between adjacent pixels.

[0072] In some other embodiments, the initial cost volume is updated using the weight matrix and the affinity relationship between each pixel and its neighboring pixel information to obtain a cost volume with affinity relationship constraints. The weight matrix is ​​applied to the update of the initial cost volume through a convolution operation to further enhance the expression of correlation information. The weight matrix is ​​a known matrix or a matrix obtained through learning.

[0073] S14: Input the context feature map and the cost volume into the gated recurrent unit for iterative update to obtain the final disparity map.

[0074] The implementation method of this process may adopt relevant technologies, which is not limited in this embodiment.

[0075] Stereo matching is performed in the manner of the above embodiment, and features are extracted from the input image to obtain a multi-scale feature map and a context feature map, which can capture global context information; high- and low-frequency features are then fused on the multi-scale feature map to enhance feature extraction, which can capture clearer and more detailed image edge features while eliminating unnecessary noise; for the extracted fused feature map, the intra-group correlation body and the mean correlation body are calculated, and then the two correlation bodies are combined to form an initial cost volume, and the affinity relationship is obtained by calculating the similarity between the pixel and its neighborhood pixel information, and the affinity relationship is used to update the initial cost volume to obtain a cost volume with affinity relationship constraints, so that the affinity relationship constraints are introduced into the cost volume, and the cost of edge matching can be determined more accurately. The obtained context feature map, the initial cost volume, and the cost volume with affinity relationship constraints are then used as inputs of the gated recurrent unit for iterative update, which can not only retain the required historical information to stabilize the disparity estimation, but also use the neighborhood pixel information to adjust the disparity update process, so that the new information is processed first in the key area, and a disparity map that captures fine image details is generated, and the accuracy of the disparity estimation is gradually improved. Therefore, the above embodiment effectively solves the problem of difficulty in characterizing stereo matching features in complex scenes and improves the accuracy of disparity estimation.

[0076] In some other embodiments, the context feature map, the initial cost volume, and the cost volume with affinity constraint are input into a gated recurrent unit for iterative update to obtain a final disparity map, including:

[0077] The first hidden state of the gated recurrent unit is initialized with the context feature map of the image, the initial disparity estimate and the initial cost volume, and the second hidden state of the gated recurrent unit is initialized with the cost volume with affinity constraints. The initial disparity estimate can be set empirically or directly estimated through a pre-trained stereo matching network.

[0078] The gated recurrent unit performs the following iterative updates based on the initialized first hidden state and the initialized second hidden state:

[0079] In each iteration, the current first hidden state is used as the input of the gated cyclic unit, and the current first hidden state is updated by controlling the update gate and reset gate of the gated cyclic unit to obtain the updated first hidden state. The first hidden state includes a context feature map, a current disparity estimate, and a current cost volume. A weight matrix is ​​generated based on the second hidden state, and the disparity estimate in the updated first hidden state is adjusted by the weight matrix to obtain the adjusted disparity estimate; the disparity estimate in the updated first hidden state is replaced by the adjusted disparity estimate, thereby obtaining the first hidden state of the next iteration. The implementation method of updating the current first hidden state according to the update gate and reset gate of the gated cyclic unit to obtain the updated first hidden state can adopt relevant technologies, which are not limited in this embodiment. The method of generating a weight matrix based on the second hidden state can be implemented by relevant technologies, which are not limited in this embodiment. In some possible embodiments, the weight matrix can be generated based on the second hidden state using a softmax activation function.

[0080] After the iterative update is completed, a final disparity map is obtained. The iterative update may be completed when the number of iterative updates is reached, or when the obtained disparity map converges.

[0081] When the above method is used for iterative updating, the context feature map, the current disparity estimate and the current cost volume are used as the first hidden state of the gated recurrent unit to provide key information, and the disparity is gradually optimized through iterative updating. The first hidden state is updated by controlling the update gate and reset gate of the gated recurrent unit, so that the key information in the first hidden state is updated and used as the hidden state for the next iterative update, which not only retains the required historical information to stabilize the disparity estimate, but also allows new information to be processed preferentially in the key area. The cost volume with affinity relationship constraints is used as the second hidden state. The second hidden state is used as the supervisory information in the iterative update process in the form of an additional hidden state to adjust the disparity estimate in the first hidden state output by the gated recurrent unit, so that the relative features of the center pixel and the domain pixel play a role in the update process, thereby obtaining a more accurate disparity estimate. Through the iterative update of the key information in the hidden state, the disparity estimate gradually converges, and after the iterative update is completed, the final accurate disparity map is obtained. In addition, by introducing the second hidden state to adjust the disparity estimate of the gated recurrent unit, which is used as the current disparity estimate in the first hidden state for the next iteration, the context feature map and the current cost volume in the first hidden state for the next iteration are only updated by the update gate and the reset gate of the gated recurrent unit, thereby achieving decoupling of the disparity estimation output, avoiding potential coupling problems, improving the performance of the network, and facilitating the retention of more effective semantic information in iterations.

[0082] In some other embodiments, feature extraction is performed on the input left and right images to obtain a multi-scale feature map, including:

[0083] The input left and right images are respectively normalized to obtain a normalized image. The pixel values ​​of the normalized image are normalized to a set range. The method of normalizing an image can be implemented by using relevant technologies.

[0084] The normalized image is subjected to feature extraction using residual blocks and downsampling to obtain a multi-scale feature map.

[0085] In some other embodiments, high-frequency and low-frequency features are fused on the multi-scale feature map to obtain a fused feature map, including:

[0086] A high-pass filter is used to filter the multi-scale feature map to obtain high-frequency features containing high-frequency information to enhance the edges and details of the image. A low-pass filter is used to filter the multi-scale feature map to obtain low-frequency features containing low-frequency information to smooth the image and reduce noise.

[0087] Then, an adaptive weight sliding window is used to perform weighted fusion of high-frequency features and low-frequency features to obtain a fused feature map. By using an adaptive weight sliding window for weighted fusion, the adaptive weight is optimized through back-propagation training to adjust the weight based on the gradient loss to achieve the optimal fusion effect.

[0088] Figure 2 Shown is a structural block diagram of an embodiment of a stereo matching device provided by the present invention.

[0089] like Figure 2 As shown, the structural units included in the device of this embodiment, the functions of the structural units and the relationships between them are as follows:

[0090] The device includes:

[0091] The multi-scale feature map acquisition unit 21 is used to extract features from the input left and right images using residual blocks and downsampling to obtain a multi-scale feature map.

[0092] The context feature map acquisition unit 22 is used to re-extract the multi-scale feature map acquired by the multi-scale feature map acquisition unit 21 to obtain a context feature map.

[0093] The fused feature map acquisition unit 23 is used to fuse the high-frequency and low-frequency features of the multi-scale feature map acquired by the multi-scale feature map acquisition unit 21 to obtain a fused feature map.

[0094] The cost volume acquisition unit 24 is used to divide the fused feature map obtained by the fused feature map acquisition unit 23 into multiple feature groups along the channel dimension, and in each feature group, use the inner product to obtain the intra-group correlation body; also calculate the product mean of the feature image pairs in the fused feature map on all channels to obtain the mean correlation body; splice the intra-group correlation body and the mean correlation body to obtain the initial cost volume; perform a neighborhood search on each pixel in the cost volume to obtain the neighborhood pixel information of each pixel; calculate the similarity between each pixel and the neighborhood pixel information of the pixel to obtain the affinity relationship, and use the affinity relationship to update the initial cost volume to obtain the cost volume with the affinity relationship constraint.

[0095] The iterative updating unit 25 is used to input the context feature map obtained by the context feature map obtaining unit 22, the initial cost volume obtained by the cost volume obtaining unit 24, and the cost volume with affinity relationship constraints into the gated recurrent unit for iterative updating to obtain a final disparity map.

[0096] The device of the above structure runs the corresponding software program and performs the corresponding function according to Figure 1 The process of the stereo matching method embodiment and other embodiments realizes stereo matching, achieving Figure 1 The corresponding technical effects of the embodiment and other embodiments.

[0097] Figure 3 and Figure 4 The figure shows an original image of an embodiment and a disparity map obtained by using the stereo matching method provided by the present invention. Figure 3 is an original image, specifically, an original image of a road intersection; Figure 4 For using the stereo matching method provided by the present invention Figure 3 The disparity map is obtained by processing the original image of the road intersection. Figure 4 It can be seen that the disparity map obtained by the stereo matching method of the present invention can capture fine image details.

[0098] Figure 5 The original images of other embodiments and the disparity maps obtained by applying the stereo matching method provided by the present invention to the original images are shown. Figure 5 In the four groups of images (a), (b), (c), and (d), the left image is the original image, and the right image is the disparity map obtained by processing the original image on the left. Figure 5 It can be seen that the disparity map obtained by the stereo matching method of the present invention has higher detail and edge preservation quality.

[0099] Another embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, Figure 1The stereo matching method of the embodiment and other embodiments, and realizes the technical effects of the corresponding embodiments.

[0100] The above-mentioned computer storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Computer storage media can be any available storage media that can be accessed by general or special-purpose computers.

[0101] In some embodiments, a computer storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist in the device as discrete components.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.

Claims

1. A stereo matching method, characterized in that: The method comprises: Extract features from the input left and right images using residual blocks and downsampling to obtain a multi-scale feature map; re-extract the multi-scale feature map to obtain a context feature map; fuse high- and low-frequency features of the multi-scale feature map to obtain a fused feature map; The fused feature map is divided into a plurality of feature groups along the channel dimension, and in each feature group, an inner product is used to obtain an intra-group correlation body; the product mean of the feature image pairs in the fused feature map on all channels is calculated to obtain a mean correlation body; the intra-group correlation body and the mean correlation body are spliced ​​to obtain an initial cost volume; a neighborhood search is performed on each pixel in the initial cost volume to obtain neighborhood pixel information of each pixel; the similarity between each pixel and the neighborhood pixel information of the pixel is calculated to obtain an affinity relationship, and the initial cost volume is updated using the affinity relationship to obtain a cost volume with affinity relationship constraints; The context feature map, the initial cost volume and the cost volume with affinity relationship constraints are input into a gated recurrent unit for iterative updating to obtain a final disparity map.

2. The stereo matching method according to claim 1, characterized in that: Calculate the similarity between each pixel and its neighboring pixel information, including: A structured attention model is used to calculate the attention weight of each neighboring pixel of each pixel; The similarity between each pixel and the neighborhood pixel information of the pixel is calculated based on the neighborhood pixel information and the attention weight of each neighborhood pixel.

3. The stereo matching method according to claim 1, characterized in that: The initial cost volume is updated by using the affinity relationship to obtain a cost volume with affinity relationship constraints, including: The initial cost volume is updated using the weight matrix and the affinity relationship to obtain a cost volume with affinity relationship constraints.

4. The stereo matching method according to claim 1, characterized in that: The intra-group correlation body and the mean correlation body are concatenated to obtain an initial cost volume, including: Performing pooling operations on the intra-group correlation entity and the mean correlation entity respectively to obtain a pooled intra-group correlation entity and a pooled mean correlation entity; The pooled intra-group correlation body and the pooled mean correlation body are concatenated to obtain the initial cost volume.

5. The stereo matching method according to any one of claims 1 to 4, characterized in that: Inputting the context feature map, the initial cost volume, and the cost volume with affinity relationship constraints into a gated recurrent unit for iterative updating to obtain a final disparity map, including: Initializing a first hidden state of the gated recurrent unit using the context feature map of the image, the initial disparity estimate, and the initial cost volume, and initializing a second hidden state of the gated recurrent unit using the cost volume with affinity relationship constraints; The gated recurrent unit performs the following iterative updates based on the initialized first hidden state and the initialized second hidden state: in each iteration, the current first hidden state is used as the input of the gated recurrent unit, and the current first hidden state is updated by controlling the update gate and the reset gate of the gated recurrent unit to obtain an updated first hidden state; a weight matrix is ​​generated based on the second hidden state, and the disparity estimate in the updated first hidden state is adjusted by using the weight matrix to obtain an adjusted disparity estimate; and the disparity estimate in the updated first hidden state is replaced by the adjusted disparity estimate to obtain a first hidden state of the next iteration; After the iterative update is completed, the final disparity map is obtained.

6. The stereo matching method according to any one of claims 1 to 4, characterized in that: Perform feature extraction on the input left and right images to obtain multi-scale feature maps, including: The input left and right images are respectively normalized to obtain the normalized images; The standardized image is subjected to feature extraction using a residual block and downsampling to obtain the multi-scale feature map.

7. The stereo matching method according to any one of claims 1 to 4, characterized in that: The high-frequency and low-frequency features of the multi-scale feature map are fused to obtain a fused feature map, including: A high-pass filter is used to filter the multi-scale feature map to obtain high-frequency features; a low-pass filter is used to filter the multi-scale feature map to obtain low-frequency features; The high-frequency features and the low-frequency features are weightedly fused using an adaptive weight sliding window to obtain the fused feature map.

8. A stereo matching device, characterized in that: The device comprises: A multi-scale feature map acquisition unit is used to extract features from the input left and right images using residual blocks and downsampling to obtain a multi-scale feature map; A context feature map acquisition unit, used to re-extract the multi-scale feature map to obtain a context feature map; A fused feature map acquisition unit, used for fusing high-frequency and low-frequency features of the multi-scale feature map to obtain a fused feature map; The cost volume acquisition unit is used to divide the fused feature map into multiple feature groups along the channel dimension, and in each feature group, use the inner product to obtain the intra-group correlation body; further calculate the product mean of the feature image pair in the fused feature map on all channels to obtain the mean correlation body; splice the intra-group correlation body and the mean correlation body to obtain the initial cost volume; perform a neighborhood search on each pixel in the initial cost volume to obtain the neighborhood pixel information of each pixel; calculate the similarity between each pixel and the neighborhood pixel information of the pixel to obtain the affinity relationship, and use the affinity relationship to update the initial cost volume to obtain the cost volume with the affinity relationship constraint; The iterative updating unit is used to input the context feature map, the initial cost volume and the cost volume with affinity relationship constraint into the gated recurrent unit for iterative updating to obtain a final disparity map.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the stereo matching method according to any one of claims 1 to 7 is implemented.

Images