Binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement

Abstract

The application discloses a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, and belongs to the field of disparity calculation, comprising the following steps: determining at least two first image features of a first image and at least one second image feature of a second image; performing geometric enhancement on a target first image feature according to all the first image features except the target first image feature and all the second image features, so as to obtain enhanced features of the first image; and determining a disparity map of the first image to the second image based on the enhanced features. The application can fully utilize structural information provided by monocular prior in a weak texture area and fully utilize local boundary information in a structural edge area, so that the stability and detail keeping ability of the disparity map in a complex scene are improved, and the probability of phenomena such as unstable disparity estimation, blurred disparity and unclear boundary is reduced.

Description

Technical Field

[0001] This application belongs to the field of disparity calculation, specifically relating to a binocular stereo matching method, device, storage medium, equipment, and computer program product based on monocular disparity distribution prior and gradient feature enhancement. Background Technology

[0002] With the rapid development of technologies such as autonomous driving, robot navigation, 3D reconstruction, and augmented reality, acquiring accurate depth information of objects in a scene has become a fundamental capability for many visual perception tasks. Binocular stereo matching, as a technique for inferring scene depth by utilizing disparity information between the left and right eyes, is widely used in these fields due to its advantages such as not requiring additional depth sensors, wide applicability, and low cost. In practical applications, binocular stereo matching systems typically need to maintain stable disparity estimation capabilities in scenes with sparse textures, large lighting variations, severe occlusion, or complex structures. Therefore, higher demands are placed on the accuracy of the disparity map, its detail preservation capabilities, and its robustness in weakly textured regions.

[0003] To improve the accuracy of binocular stereo matching, various deep learning-based stereo matching methods have been proposed in existing technologies. For example, ... Figure 1 As shown, in the Iterative Geometry Encoding Volume for Stereo Matching (IGEV) network... Under Stereo, by adjusting the left eye view ( Figure 1 - (a) column) and right view ( Figure 1 Feature extraction is performed on column (b), and a matching cost body for the left and right images under different disparity assumptions is constructed. The initial disparity is then updated multiple times using an iterative optimization network to generate the final disparity map. Figure 1 - (c) column). This type of method typically relies on deep convolutional features or cost volume construction strategies to enhance the matching relationship between the left and right images, thereby achieving high disparity estimation accuracy in most scenarios.

[0004] However, existing methods are still prone to instability in disparity estimation in weak texture regions, repetitive texture regions, or structural edge regions, which can easily lead to disparity blurring or unclear boundaries. Summary of the Invention

[0005] This application aims to provide a binocular stereo matching method, apparatus, storage medium, device, and computer program product based on monocular disparity distribution prior and gradient feature enhancement, which at least solves the problem of unstable disparity estimation.

[0006] In a first aspect, embodiments of this application disclose a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, including: Determine at least two first image features of a first image and at least one second image feature of a second image; the first image features include a target first image feature for prior inference of the pixel disparity probability distribution of the first image; Based on all first image features other than the target first image features, and all second image features, geometric enhancement is performed on the target first image features to obtain the enhanced features of the first image; The disparity map of the first image to the second image is determined based on the enhanced features.

[0007] Secondly, embodiments of this application also disclose a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, including: Based on the joint loss constructed from multiple disparity supervision data used to supervise the disparity estimation process, the target data layer in the disparity iterative model is updated with parameters; the target data layer in the disparity iterative model is used to implement the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement as described in the first aspect.

[0008] Thirdly, embodiments of this application also disclose a binocular stereo matching device based on monocular disparity distribution prior and gradient feature enhancement, comprising: The feature extraction module is used to determine at least two first image features of a first image and at least one second image feature of a second image; the first image features include a target first image feature for prior inference of the pixel disparity probability distribution of the first image; The feature enhancement module is used to geometrically enhance the features of the target first image based on all the first image features other than the target first image features, and all the second image features, so as to obtain the enhanced features of the first image; A disparity calculation module is used to determine the disparity map of the first image to the second image based on the enhancement features.

[0009] Fourthly, embodiments of this application also disclose a binocular stereo matching device based on monocular disparity distribution prior and gradient feature enhancement, characterized in that it includes: The model training module is used to update the parameters of the target data layer in the disparity iterative model based on a joint loss constructed from multiple disparity supervision data used to supervise the disparity estimation process; the target data layer in the disparity iterative model is used to implement the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement as described in the first aspect.

[0010] Fifthly, embodiments of this application also disclose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the first or second aspect.

[0011] In a sixth aspect, embodiments of this application also disclose an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the steps as described in the first or second aspect.

[0012] In a seventh aspect, embodiments of this application also disclose a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps described in the first or second aspect.

[0013] In summary, in this embodiment, by setting target first image features in at least two first image features of the first image for prior inference of the pixel disparity probability distribution of the first image, the disparity estimation process can obtain a monocular disparity distribution prior before binocular matching. This provides initial disparity distribution constraints for regions with weak textures or lacking obvious matching cues, reducing disparity instability that easily occurs when relying solely on binocular matching costs. Subsequently, the target first image features are geometrically enhanced based on the remaining image features, allowing the monocular disparity distribution prior to incorporate enhancement information including local structural changes while fusing the binocular matching costs. In this way, the binocular matching relationship can correct the monocular prior, reducing the possibility of bias in complex scenes; simultaneously, the enhancement information can strengthen the response of object edges and structurally changing regions, improving the problems of disparity boundary blurring and detail loss. Finally, the disparity map of the first image to the second image is determined based on the enhanced features, so that the final disparity estimation result benefits from multiple constraints provided by the monocular disparity distribution prior, the binocular matching costs, and gradient feature enhancement. Therefore, the method based on the embodiments of this application can make full use of the structural information provided by monocular prior in weak texture regions and make full use of local boundary information in structural edge regions, thereby improving the stability and detail preservation ability of disparity maps in complex scenes and reducing the probability of phenomena such as unstable disparity estimation, disparity blurring and unclear boundaries. Attached Figure Description

[0014] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a comparison diagram between the embodiments of this application and the prior art; Figure 2 This is a flowchart illustrating the steps of a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement provided in an embodiment of this application. Figure 3 This is a flowchart of another binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement provided in the embodiments of this application; Figure 4 This is a flowchart of another binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the steps of another binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of a binocular stereo matching device based on monocular disparity distribution prior and gradient feature enhancement provided in an embodiment of this application; Figure 7 This is a schematic diagram of another binocular stereo matching device based on monocular disparity distribution prior and gradient feature enhancement provided in the embodiments of this application; Figure 8 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0016] Figure 2 This application provides a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, which specifically includes the following steps: Step 101: Determine at least two first image features of the first image and at least one second image feature of the second image.

[0017] The first image feature includes a target first image feature used to make prior inferences about the pixel disparity probability distribution of the first image.

[0018] In some embodiments of this application, in order to provide usable input information for subsequent geometric enhancement processing and disparity map generation, it is necessary to first obtain basic features that characterize the image content and structure from the first image and the second image. Specifically, this can be achieved by determining at least two first image features of the first image and at least one second image feature of the second image. Here, the target first image feature is a feature used to make prior inferences about the pixel disparity probability distribution of the first image. The target first image feature here refers to the monocular disparity distribution prior tensor obtained after encoding the first image through a monocular disparity inference network, which characterizes the probability distribution of each pixel in the first image under different disparity assumptions. This allows the monocular disparity distribution prior to be obtained before binocular matching begins, enabling subsequent geometric enhancement processing to be carried out under prior constraints, thereby providing initial disparity distribution information for weakly textured regions or regions lacking obvious matching cues.

[0019] In a scenario requiring depth inference from binocular images, it's necessary to extract fundamental features from the left and right views for subsequent disparity estimation. This can be achieved by extracting embedding features and image gradient features from the left view, and then generating a monocular disparity distribution prior tensor for the left view using a monocular disparity inference network. These three types of features are then used as the first image feature of the left view. Simultaneously, embedding features are extracted from the right view as the second image feature. After obtaining these features, both the left and right views possess input information for subsequent geometric augmentation processing. The target's first image feature in the left view provides an initial disparity distribution constraint, enabling subsequent steps to perform disparity estimation with this prior information.

[0020] Step 102: Based on all the first image features other than the target first image features, and all the second image features, perform geometric enhancement on the target first image features to obtain the enhanced features of the first image.

[0021] In some embodiments of this application, to enable the monocular disparity distribution prior to more closely approximate the binocular matching relationship and possess stronger structural expressive power in subsequent disparity estimation, it is necessary to use the available features from the first and second images together to enhance the features of the target first image. Specifically, geometric enhancement can be performed on the features of the target first image based on all the features of the first image other than the features of the target first image, as well as all the features of the second image, to obtain the enhanced features of the first image. Here, geometric enhancement refers to modulating the monocular disparity distribution prior using binocular matching relationships and local structural information, so that it can more accurately reflect the spatial correspondence of the first image under different disparity assumptions. This will enable the monocular disparity distribution prior to form more stable enhanced features after fusing binocular matching costs and local structural change information, providing a more reliable input basis for subsequent disparity regression and iterative optimization.

[0022] In a scenario where maintaining disparity estimation stability in weakly textured regions is crucial, the matching relationship between binocular images needs to supplement and correct the monocular disparity distribution prior. In this case, the embedding features and image gradient features of the left view can be input together with the embedding features of the right view into the geometry enhancement module to enhance the monocular disparity distribution prior of the left view. After this processing, the left view obtains enhanced features containing binocular matching relationships and local structural change information, enabling subsequent steps to perform disparity estimation based on more accurate feature representation, thereby improving disparity stability in weakly textured and structural edge regions.

[0023] Step 103: Determine the disparity map of the first image to the second image based on the enhanced features.

[0024] In some embodiments of this application, in order to generate the final disparity estimation result based on the enhanced features that have fused the binocular matching relationship and local structural information, it is necessary to infer the pixel displacement relationship between the first image and the second image using these enhanced features. Specifically, the disparity map of the first image and the second image can be determined based on the enhanced features. Here, the disparity map refers to the pixel-level disparity distribution obtained by performing disparity regression or iterative optimization on the enhanced features, which reflects the positional offset of each pixel in the first image in the second image. This allows the final disparity estimation result to simultaneously benefit from the monocular disparity distribution prior, the binocular matching cost, and the gradient enhancement information, enabling the generated disparity map to have a more stable depth representation capability in weak texture regions and structural edge regions.

[0025] In an application scenario requiring scene depth recovery from binocular images, a disparity map of the left view relative to the right view needs to be generated based on the augmented features obtained in the previous step. The augmented features can then be input into a disparity regression module or an iterative optimization network to infer the offset of each pixel in the left view relative to its corresponding position in the right view. After this processing, a disparity map of the left view relative to the right view is obtained, containing pixel-level disparity information inferred from the augmented features. This allows subsequent depth reconstruction or 3D understanding tasks to be performed on a more reliable disparity basis.

[0026] In summary, in this embodiment, by setting target first image features in at least two first image features of the first image for prior inference of the pixel disparity probability distribution of the first image, the disparity estimation process can obtain a monocular disparity distribution prior before binocular matching. This provides initial disparity distribution constraints for regions with weak textures or lacking obvious matching cues, reducing disparity instability that easily occurs when relying solely on binocular matching costs. Subsequently, the target first image features are geometrically enhanced based on the remaining image features, allowing the monocular disparity distribution prior to incorporate enhancement information including local structural changes while fusing the binocular matching costs. In this way, the binocular matching relationship can correct the monocular prior, reducing the possibility of bias in complex scenes; simultaneously, the enhancement information can strengthen the response of object edges and structurally changing regions, improving the problems of disparity boundary blurring and detail loss. Finally, the disparity map of the first image to the second image is determined based on the enhanced features, so that the final disparity estimation result benefits from multiple constraints provided by the monocular disparity distribution prior, the binocular matching costs, and gradient feature enhancement. Therefore, the method based on the embodiments of this application can make full use of the structural information provided by monocular prior in weak texture regions and make full use of local boundary information in structural edge regions, thereby improving the stability and detail preservation ability of disparity maps in complex scenes and reducing the probability of phenomena such as unstable disparity estimation, disparity blurring and unclear boundaries.

[0027] Figure 3 This application provides another binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, which specifically includes the following steps: Step 201: Determine the embedding data of the first image and the second image respectively, and determine the image gradient data of the first image. Then, determine the embedding data and image gradient data of the first image as a first image feature other than a target first image feature.

[0028] In some embodiments of this application, in order to provide a usable input basis for subsequent geometric enhancement processing, it is necessary to extract multiple types of feature information that can characterize the image content and structure from the first image and the second image. Specifically, the embedding data of the first image and the second image can be determined separately, and the image gradient data of the first image can be determined. The embedding data and the image gradient data of the first image are respectively determined as first image features other than the target first image features. Here, the embedding data refers to the high-dimensional representation obtained after encoding the image through a feature extraction network, which is used to characterize the semantic and texture information of the image; the image gradient data refers to the local structural change information obtained by calculating the gradient of image pixels, which is used to reflect the intensity changes of detailed regions such as edges and contours. In this way, the first image can have multiple types of auxiliary features other than the target first image features, so that the subsequent geometric enhancement processing can be carried out on the basis of richer feature expression, thereby improving the structural sensitivity and matching stability of the enhancement result.

[0029] In a scenario requiring depth estimation of binocular images, basic features from both images are needed for subsequent augmentation processing. Embedded data and image gradient data can be extracted from the left view, serving as the first image feature of the left view; simultaneously, embedded data can be extracted from the right view as the second image feature. After obtaining these features, both the left and right views possess input information for subsequent geometric augmentation processing. The embedded data and image gradient data from the left view are auxiliary features used to supplement the first image feature, enabling subsequent steps to perform disparity estimation based on a more comprehensive feature representation.

[0030] In some embodiments of this application, in order to enable the embedded data to represent the semantic and texture information of an image at different spatial scales, embedding features at multiple resolutions can be generated by a feature encoder. This multi-scale embedding feature... It can be formalized as , Among them The number of channels for embedding features. and These correspond to the spatial dimensions of the embedded data at different scales. This allows the embedded data to represent the texture, structure, and semantic information of the image in a multi-scale space, providing a richer feature foundation for the subsequent construction of the matching cost 3D tensor.

[0031] Optionally, to determine the image gradient data of the first image, step 201 includes the following sub-steps: Sub-step 2011: Perform gradient calculation on the first image to obtain a gradient map that characterizes the numerical changes of each pixel in the first image with respect to pixels in adjacent rows and columns.

[0032] In some embodiments of this application, in order to extract basic information reflecting local structural changes from the first image, it is necessary to calculate the variation trends of image pixels in the row and column directions. Specifically, gradient calculation can be performed on the first image to obtain a gradient map representing the numerical changes of each pixel in the first image with respect to pixels in adjacent rows and columns. Here, the gradient map refers to two-dimensional data formed by calculating the intensity differences of pixels in the horizontal and vertical directions, used to reflect edges, contours, and local structural changes in the image. This provides a basic input for subsequent gradient feature extraction, enabling the image gradient data to more accurately represent structurally sensitive regions.

[0033] In a scenario where edge and detail regions need to be identified in an image, data reflecting pixel change trends is required. Gradient calculation can be performed on the first image to generate a gradient map containing information on horizontal and vertical variations. This gradient map can then be used as input for subsequent gradient encoding processes, enabling later steps to extract more discriminative gradient features based on information about structural changes.

[0034] In some embodiments of this application, to more accurately characterize the structural changes of the first image in local regions, a gradient map can be calculated based on the intensity difference of pixels in the horizontal and vertical directions. Gradient map At pixel position The gradient value calculation method can be formalized as follows: , This refers to the pixel position of the first or second image. The original pixel intensity; in the expression and These correspond to pixel differences in the horizontal and vertical directions, respectively, and are used to reflect the local variation trends of the image in these two directions; (symbol) This refers to performing a L2 norm operation on the horizontal and vertical differences to synthesize the gradient magnitude at the pixel location. This allows the gradient map to represent the edges, contours, and local structural changes of the image at the pixel level, providing a more reliable input basis for subsequent gradient encoding processing.

[0035] Sub-step 2012 involves inputting the gradient map into the gradient encoding network to use the gradient features of the obtained first image as image gradient data.

[0036] In some embodiments of this application, considering that the original gradient map only contains pixel difference information, which is insufficient for direct use in geometric enhancement processing, it is necessary to further extract higher-level gradient representations. Specifically, the gradient map can be input into a gradient encoding network to use the gradient features of the obtained first image as image gradient data. Here, the gradient encoding network refers to a network structure used for convolutional encoding or multi-scale feature extraction of the gradient map, which transforms the original gradient information into more expressive gradient features. This enables the image gradient data to have stronger structural sensitivity, providing more reliable local change information for subsequent geometric enhancement.

[0037] In a scenario where local structural changes need to be utilized to improve the stability of disparity estimation, the gradient map needs to be transformed into more expressive features. In this case, the gradient map obtained in the previous step can be input into a gradient encoding network to generate gradient features of the first image, which are then used as image gradient data in subsequent processing. After obtaining these gradient features, the model can more accurately identify edge regions and areas of structural change in the subsequent geometric enhancement stage, enabling disparity estimation to maintain more stable performance in complex scenes.

[0038] In some embodiments of this application, in order to enable image gradient data to express local structural changes at different scales, multi-scale gradient features can be generated using a gradient coding network. These gradient features can be formalized as follows: , Here and These correspond to the gradient features of the first and second images at different scales, respectively. The number of channels corresponding to the gradient feature, and This corresponds to the spatial dimensions of gradient features at different scales. This allows gradient features to represent edge, contour, and local structural changes in multi-scale space, providing more reliable structural information for subsequent gradient sensitivity enhancement.

[0039] Step 202: Input the first image into the prior inference network to obtain the monocular disparity distribution prior tensor of the first image as the feature of the target first image.

[0040] In some embodiments of this application, considering the need to provide prior information characterizing the initial disparity distribution of the first image for subsequent geometric enhancement processing, a monocular inference method is typically used to generate the disparity probability distribution of the first image. Specifically, the first image can be input into a prior inference network to obtain a monocular disparity distribution prior tensor of the first image, and this tensor can be used as a feature of the target first image. Here, the monocular disparity distribution prior tensor refers to the pixel-level disparity probability distribution obtained after encoding the first image through a monocular disparity inference network, which is used to express the possible structure of the first image under different disparity assumptions. This allows the first image to have initial disparity distribution information before entering the geometric enhancement stage, thereby providing basic constraints for weak texture regions or regions lacking obvious matching cues.

[0041] In a scenario where an initial disparity reference is needed for the binocular stereo matching process, prior information reflecting pixel depth trends needs to be obtained from the first image. In this case, the first image can be input into a monocular disparity inference network to generate a prior tensor of the monocular disparity distribution of the first image. This tensor is then used as a feature of the target first image in subsequent processing. After obtaining this prior tensor, the first image possesses an initial disparity distribution for subsequent geometric enhancement, enabling subsequent steps to perform disparity estimation based on a more structured prior.

[0042] In some embodiments of this application, disparity regression can be performed using the softargmin operation to obtain monocular disparity usable for intermediate supervision from a priori monocular disparity distribution. This process can be formalized as follows: , Here It refers to monocular disparity inferred based on monocular texture information, used for intermediate supervision; This refers to the parallax dimension. Perform normalization; This represents the maximum parallax range. This will enable the monocular prior to obtain a more stable supervisory signal during training.

[0043] Step 203: Based on the similarity relationship between the embedded data of the first image and the embedded data of the second image, determine the three-dimensional tensor of the matching cost of the second image to the first image.

[0044] In some embodiments of this application, considering the need to utilize the correspondence between binocular images to construct a basic data structure that reflects the difficulty of pixel matching, a matching cost volume is typically generated based on the feature similarity between two images. Specifically, the three-dimensional tensor of the matching cost of the second image to the first image can be determined based on the similarity relationship between the embedded data of the first image and the embedded data of the second image. Here, the three-dimensional tensor of the matching cost refers to a three-dimensional data structure formed by calculating the similarity between the features of two images under multiple disparity assumptions, used to characterize the matching cost of each pixel in the first image at different disparity positions. In this way, basic information reflecting the binocular matching relationship can be provided for subsequent geometric enhancement processing, enabling the enhancement process to be carried out under more accurate matching constraints.

[0045] In a scenario where pixel correspondence needs to be inferred from binocular images, a dataset reflecting the matching difficulty under different disparity assumptions needs to be constructed. In this case, the embedding data of the left and right views can be input into a similarity calculation module to calculate the similarity between them at multiple disparity locations, thereby generating a 3D tensor of the matching cost from the right view to the left view. After obtaining this 3D tensor, this data structure can be used as input for subsequent geometric augmentation processing, enabling disparity inference under binocular matching constraints.

[0046] Optionally, step 203 includes the following sub-steps: Sub-step 2031: Based on the similarity relationship between the embedded data of the first image and the embedded data of the second image under multiple different disparity assumptions, construct the first matching correlation tensor of the second image to the first image.

[0047] In some embodiments of this application, in order to construct a data structure that reflects the correspondence of binocular image features under different disparity assumptions, it is necessary to calculate multi-disparity correlation based on the embedding features of two images. Specifically, a first matching correlation tensor of the second image to the first image can be constructed based on the similarity relationship between the embedding data of the first image and the embedding data of the second image under multiple different disparity assumptions. Here, the first matching correlation tensor refers to a three-dimensional data structure formed by calculating the correlation of the embedding features of the two images at multiple disparity positions, used to characterize the matching relationship under different disparity assumptions. This will provide basic multi-disparity correlation information for subsequent cost construction.

[0048] In a scenario where a basic cost volume needs to be constructed based on the multi-disparity assumption, it is necessary to calculate the feature similarity between the left and right views at different disparity positions. In this case, the correlation between the embedded data of the left view and the embedded data of the right view at multiple disparity offsets can be calculated to generate the first matching correlation tensor of the right view to the left view. This first matching correlation tensor expresses the feature correspondence under multi-disparity conditions and is an important component in constructing the three-dimensional matching cost tensor, enabling subsequent steps to further develop a complete cost structure based on it.

[0049] In some embodiments of this application, in order to express the feature correspondence between a first image and a second image under multiple disparity assumptions, a three-dimensional correlation cost volume can be constructed based on the embedded data of the two images. This cost volume can be formalized as follows: , Here This refers to the parallax hypothesis. Lower pixel position The relevant value; and These correspond to the embedded data of the first and second images at 1 / 4 resolution, respectively; Symbols This is used to perform an inner product operation on two feature vectors, measuring the feature similarity between the two images at that disparity location. This allows the 3D correlation cost volume to express the binocular matching relationship in the disparity dimension, providing fundamental information for subsequent cost fusion and geometric enhancement.

[0050] Sub-step 2032: Group the embedded data of the first image and the embedded data of the second image in the channel dimension respectively, and construct the second matching correlation tensor of the second image to the first image based on the similarity relationship within each channel group.

[0051] In some embodiments of this application, considering that differences in feature distribution between different channel groups may affect matching expressive ability, it is necessary to perform group correlation calculation in the channel dimension. Specifically, the embedded data of the first image and the embedded data of the second image can be grouped in the channel dimension, and a second matching correlation tensor of the second image to the first image can be constructed based on the similarity relationship within each channel group. Here, the second matching correlation tensor refers to a three-dimensional data structure formed by performing intra-group correlation calculation on the grouped features, which is used to supplement the shortcomings of the first matching correlation tensor in fine-grained channel expression. This will enable the matching relationship to have higher discriminative ability in the channel dimension.

[0052] In a scenario where the cost volume needs enhanced channel representation capabilities, the aim is to leverage grouped features to improve the fine-grained representation of matching relationships. This can be achieved by dividing the embedded data of the left and right views into multiple groups along the channel dimension, and calculating feature similarity within each group to generate a second matching correlation tensor for the right view versus the left view. This second matching correlation tensor expresses the fine-grained matching relationships under channel grouping conditions and is another component in constructing the three-dimensional matching cost tensor, enabling the final cost volume to simultaneously possess multi-disparity correlation and fine-grained channel correlation.

[0053] In some embodiments of this application, in order to preserve finer-grained feature representations along the channel dimension, the embedded data can be grouped by channel, and correlations can be calculated within each group to construct a four-dimensional group correlation cost body. This cost body can be formalized as follows: , Here This refers to the group number. Parallax hypothesis With pixel position The related value of the group below; and This refers to the embedded data of the first and second images after channel grouping at 1 / 4 resolution; It is the total number of channels for the embedded features. It is the number of channel groups; the denominator is the number of channels. This is used to normalize the number of channels within a group. This allows the group-related cost volume to express more detailed matching relationships in the channel dimension, providing a richer feature base for subsequent detail enhancement.

[0054] In some embodiments of this application, to enhance the geometric perception capability of the cost volume in detailed regions, a gradient cost volume can be constructed based on gradient features. This cost volume can be formalized as follows: , Here This refers to a group-related cost volume constructed based on gradient features. This allows the cost volume to have stronger structural sensitivity in detailed regions.

[0055] Sub-step 2033 involves fusing the first matching correlation tensor and the second matching correlation tensor to generate a three-dimensional tensor representing the matching cost of the second image to the first image.

[0056] In some embodiments of this application, in order to obtain a cost volume that comprehensively reflects the binocular matching relationship, it is necessary to fuse matching-related information from different sources. Specifically, the first matching-related tensor and the second matching-related tensor can be fused to generate a three-dimensional matching cost tensor of the second image to the first image. Here, the three-dimensional matching cost tensor refers to the three-dimensional cost volume formed by fusing multi-disparity correlation and channel grouping correlation, which is used as the basic input for subsequent geometric enhancement processing. This enables the cost volume to have a more complete matching expression capability in both the disparity dimension and the channel dimension.

[0057] In a scenario where a complete cost volume needs to be constructed to support subsequent disparity estimation, multi-disparity correlation and channel grouping correlation need to be used together to express the binocular matching relationship. In this case, the first and second matching correlation tensors obtained in the previous step can be fused to generate a three-dimensional matching cost tensor for the right view versus the left view. This three-dimensional matching cost tensor is a comprehensive cost volume composed of two types of correlation tensors, enabling subsequent geometric augmentation steps to perform disparity inference under more comprehensive matching constraints.

[0058] Step 204: Enhance the monocular disparity distribution prior tensor by matching the cost 3D tensor and the image gradient data of the first image to obtain the enhanced features of the first image under the second image.

[0059] In some embodiments of this application, considering that the monocular disparity distribution prior relies solely on the first image itself and may exhibit biases in areas with weak texture or complex structures, it is necessary to further adjust the prior by combining binocular information and local structural information. Specifically, the monocular disparity distribution prior tensor can be enhanced using the matching cost 3D tensor and the image gradient data of the first image to obtain enhanced features of the first image in the second image. The enhancement process here refers to modulating the monocular disparity distribution prior using the binocular matching relationship provided by the matching cost 3D tensor and the local structural changes reflected by the image gradient data, making its response under different disparity assumptions closer to the real binocular geometric relationship. This allows the enhanced features to simultaneously possess the structural trends of the monocular prior, the corresponding constraints of binocular matching, and the boundary sensitivity of gradient information, providing a more stable input basis for subsequent disparity inference.

[0060] In a scenario where smooth disparity transitions are required in structural edge regions, it's desirable to supplement the monocular disparity prior using binocular matching relationships and local gradient changes. In this case, the 3D tensor of the matching cost generated between the left and right views can be input into the enhancement module along with the image gradient data of the left view to enhance the monocular disparity distribution prior tensor of the left view. After obtaining the enhanced features, the disparity representation of the left view will more closely resemble the real binocular geometry, allowing subsequent steps to perform disparity estimation based on more reliable features, especially maintaining a more stable disparity change trend in edge and weakly textured regions.

[0061] Optionally, step 204 includes the following sub-steps: Sub-step 2041 involves performing feature modulation processing on the monocular disparity distribution prior tensor based on the matching cost 3D tensor to obtain the first enhanced feature used to characterize the binocular matching relationship.

[0062] In some embodiments of this application, considering that the monocular disparity distribution prior relies solely on the first image itself and cannot fully reflect the binocular matching relationship, it is necessary to modulate this prior using matching cost information. Specifically, feature modulation processing can be performed on the monocular disparity distribution prior tensor based on the three-dimensional tensor of matching cost to obtain a first enhanced feature for characterizing the binocular matching relationship. Here, feature modulation processing refers to adjusting the response of the monocular disparity distribution prior using the multi-disparity matching constraints provided by the three-dimensional tensor of matching cost, making it closer to the binocular geometric relationship under different disparity assumptions. This will enable the first enhanced feature to have a more accurate expression of the matching trend.

[0063] In a scenario where binocular matching relationships need to be used to correct monocular disparity trends, the multi-disparity matching cost needs to be incorporated into the monocular prior adjustment process. In this case, the monocular disparity distribution prior tensor of the left view and the three-dimensional tensor of the matching cost of the right view to the left view can be jointly input into the modulation module to generate the first enhancement feature representing the binocular matching relationship. Through this process, the first enhancement feature can more accurately reflect the pixel correspondence under multi-disparity constraints, providing a foundation for subsequent enhancement steps.

[0064] In some embodiments of this application, to transform the monocular disparity prior volume and the 3D correlation cost volume into volume features usable for cross-modal fusion, linear transformation and depthwise separable convolution can be used. This process can be formalized as follows: , , Here and These correspond to the volumetric features of the monocular disparity prior and the three-dimensional correlation cost volume, respectively. It refers to a linear transformation; It is the prior tensor of monocular parallax distribution; This refers to depthwise separable convolution; This refers to using the SiLU activation function. This allows the two types of volumetric features to have a consistent representation in both spatial and channel dimensions, providing an input basis for cross-modal attention fusion.

[0065] Sub-step 2042: Perform gradient sensitivity enhancement processing on the first enhancement feature based on the image gradient data to obtain a second enhancement feature for characterizing local structural changes.

[0066] In some embodiments of this application, considering that edge regions and structural change regions in an image have a significant impact on disparity estimation, it is necessary to utilize image gradient information to enhance the structural sensitivity of the first enhancement feature. Specifically, gradient sensitivity enhancement processing can be performed on the first enhancement feature based on image gradient data to obtain a second enhancement feature for characterizing local structural changes. Here, gradient sensitivity enhancement processing refers to using the local intensity changes reflected by the image gradient data to enhance the structural details of the first enhancement feature, giving it higher responsiveness in edge regions. This will enable the second enhancement feature to perform more stably in structural change regions.

[0067] In a scenario where the stability of disparity estimation in structural edge regions needs to be improved, it is desirable to leverage image gradient information to enhance the local structural representation of the augmented features. In this case, the first augmented feature and the image gradient data from the left eye view can be input into the gradient enhancement module to generate a second augmented feature representing local structural changes. Through this process, the second augmented feature can maintain a clearer structural representation in the edge regions, providing more reliable detail information for the final augmented feature.

[0068] In some embodiments of this application, to enhance the response of detailed regions in the cost volume, spatial attention weights can be multiplied element-wise with the cost volume. This process can be formalized as follows: , Here It refers to the cost volume after spatial excitation at scale i; This refers to the cost volume at the corresponding scale; This indicates element-wise multiplication. This will enable the cost volume to have greater responsiveness in detailed regions.

[0069] Sub-step 2043: Fuse the first enhancement feature and the second enhancement feature to obtain the enhancement feature.

[0070] In some embodiments of this application, to obtain comprehensive features that simultaneously reflect binocular matching relationships and local structural changes, it is necessary to fuse enhancement features from different sources. Specifically, the first enhancement feature and the second enhancement feature can be fused to obtain the enhanced features. This fusion process refers to jointly expressing binocular matching enhancement information and gradient structure enhancement information, so that the final enhanced features have higher reliability in both disparity trends and structural details. This will provide a more comprehensive input basis for subsequent disparity inference.

[0071] In a scenario where both matching relationships and structural details need to be considered simultaneously, the enhanced features obtained in the first two steps need to be integrated. In this case, the first enhanced feature, representing the binocular matching relationship, and the second enhanced feature, representing local structural changes, can be input into the fusion module to generate the final enhanced features. Through this fusion process, the enhanced features can form a more stable representation under the combined effect of multiple pieces of information, resulting in higher accuracy in the subsequent disparity estimation process.

[0072] In some embodiments of this application, to extract the global pixel correspondence between monocular and binocular features, volume features can be transformed into query features, key features, and value features, and then fused using linear attention. This process can be formalized as follows: , , Here , , These correspond to the query features of binocular volume features, the key features of monocular volume features, and the value features of binocular volume features, respectively. Represents tensor multiplication; This refers to the mean aggregation of key features; This is the attention factor used for normalization. This will enable the fusion of features. It simultaneously expresses the relationship between monocular priors and binocular matching, providing cross-modal geometric information for subsequent enhancement.

[0073] In some embodiments of this application, a forgetting gating mechanism can be introduced to suppress erroneous prior information and retain valid matching information in the binocular cost volume during the fusion process. This mechanism can be formalized as follows: , , Here It refers to the forgetting gating weights generated based on the three-dimensional correlation cost volume, which are used to filter low-confidence regions; This refers to the fused cost body. This allows the fused cost body to retain valid matching information while suppressing interference from erroneous priors.

[0074] In some embodiments of this application, considering that the 3D tensor of the matching cost and image gradient data can provide effective geometric constraints in weakly textured and structurally complex regions, but may still suffer from insufficient local geometric representation in detailed regions, a 3D Detail-Aware Aggregation (DAA) module can be further introduced. This module, combined with a gradient spatial attention mechanism, enhances the details of edge and fine-structure regions in the cost volume, making the enhanced features more focused on local geometric changes in the spatial dimension. Here, the detail-aware aggregation module refers to an aggregation structure that uses multi-scale gradient features and spatial attention weights to compensate for details in the cost volume, thereby improving the geometric sensitivity of the enhanced features in detailed regions. This enables the enhanced features to obtain further geometric compensation at the detail level based on the fusion of binocular matching relationships and gradient change information, allowing subsequent disparity inference to maintain a more stable disparity change trend in edge regions, sparsely textured regions, and fine-structure regions.

[0075] In some embodiments of this application, spatial attention weights can be generated based on multi-scale gradient features to guide the cost volume to focus on detailed regions during the aggregation process. These weights... It can be formalized as , Here This refers to global pooling of gradient features along the channel dimension; This refers to a 7×7 convolution operation; This refers to the Sigmoid activation function, which restricts attention weights to the range of 0–1. This allows spatial attention weights to have a higher response in detailed regions, providing a basis for detailed excitation of the cost volume.

[0076] In some embodiments of this application, to enhance the geometric representation of edges and fine-structure regions in the cost volume, the group-related cost volume and the gradient cost volume can be concatenated and then input into the detail-aware aggregation module. This process can be formalized as follows: , Here It refers to the geometric detail-aware cost volume after being processed by the detail-aware aggregation module; This refers to splicing along the channel dimension. This will enable the cost volume to have a stronger geometric representation in the detailed areas.

[0077] Step 205: Determine the disparity map of the first image to the second image based on the enhanced features.

[0078] The method shown in this step has been explained in step 103 and will not be repeated here.

[0079] Optionally, step 205 includes the following sub-steps: Sub-step 2051: Determine the disparity features used to characterize disparity information by considering the matching relationship of the enhancement features under different disparity assumptions.

[0080] In some embodiments of this application, in order to extract information that directly reflects pixel disparity trends from enhanced features, it is necessary to analyze the matching responses of enhanced features under multiple disparity assumptions. Specifically, the matching relationships of enhanced features under different disparity assumptions can be analyzed to determine the disparity features used to characterize disparity information. Here, disparity features refer to feature expressions formed by aggregating or statistically analyzing the responses of enhanced features in the disparity dimension, used to reflect the probability distribution of each pixel in the first image at different disparity positions. This will provide more explicit disparity trend information for subsequent disparity regression.

[0081] In a scenario where disparity trends need to be extracted from augmented features, the disparity probability of a pixel needs to be determined based on the feature responses under multiple disparity assumptions. In this case, the augmented features can be input into a disparity feature extraction module to analyze their matching relationships at multiple disparity locations and generate disparity features to characterize disparity information. Through this process, disparity features can more clearly express the response trends of pixels under different disparity assumptions, providing a foundation for subsequent disparity regression.

[0082] Sub-step 2052: Perform disparity regression processing on the disparity values ​​of each pixel in the first image based on disparity features to obtain an initial disparity map of the first image relative to the second image.

[0083] In some embodiments of this application, considering that disparity features already contain the response trends of pixels under different disparity assumptions, these features can be used to regress pixel disparity. Specifically, disparity regression processing can be performed on the disparity values ​​of each pixel in the first image based on the disparity features to obtain an initial disparity map of the first image relative to the second image. Here, disparity regression processing refers to using the distribution of disparity features along the disparity dimension to determine the initial disparity value of each pixel through weighted averaging or other regression methods. This provides a reasonable starting point for subsequent iterative optimization.

[0084] In a scenario where preliminary disparity estimation results are needed, it's desirable to determine the initial disparity values ​​of pixels based on disparity features. In this case, the disparity features can be input into a disparity regression module to perform regression calculations on the disparity of each pixel, generating an initial disparity map of the left view relative to the right view. This initial disparity map provides a reliable starting disparity distribution for subsequent iterative optimization.

[0085] In some embodiments of this application, to provide more comprehensive contextual information in the disparity iterative network, a context extractor can be used to generate contextual features at multiple scales. These contextual features... It can be formalized as , Among them The number of channels corresponding to the context features, and The spatial dimensions of the corresponding contextual features at different scales. This enables the disparity iterative network to obtain contextual information of the left eye image in multi-scale space, thereby improving its ability to process structural and weakly textured regions during iterative optimization.

[0086] In some embodiments of this application, a dual-core gated recurrent unit (GRU) can be used as the iterative optimization structure to jointly model local cost volume features, context features, and the current disparity during the iteration process. This structure processes the input features round by round through a recurrent state update mechanism, generating disparity residuals to correct the current disparity, allowing the iterative disparity to gradually approach the true geometric correspondence over multiple updates. Here, the gated recurrent unit refers to a recurrent network unit that controls state propagation through update and reset gates. It can effectively integrate local matching information and context information during the iteration process, thereby improving the disparity stability in weakly textured regions, occluded regions, and structural edge regions. This enables the final disparity map to achieve higher reliability in terms of detail representation and overall consistency, providing a more robust disparity information foundation for subsequent depth reconstruction or 3D understanding tasks.

[0087] In some embodiments of this application, in order to obtain an initial disparity from the geometric detail-aware volume that can serve as the starting point for iterative optimization, the softargmin operation can be used to regress the cost volume in the disparity dimension. This process can be formalized as follows: , Here It refers to the initial disparity obtained based on the geometric detail perceptron, corresponding to the initial disparity of iterative optimization; This refers to the parallax level. The cost of geometric detail perception is the cost volume output by the DAA module. This refers to a probabilistic mapping that normalizes the disparity dimension. This allows the initial disparity to have stronger geometric sensitivity based on the fusion of gradient information and detail enhancement information, providing a reliable prior for subsequent GRU iterations.

[0088] Sub-step 2053 involves optimizing the disparity map using the initial disparity map as the starting point to obtain the disparity map in the disparity iterative network.

[0089] In some embodiments of this application, to further improve the accuracy of disparity estimation, multiple rounds of iterative optimization are required based on the initial disparity. Specifically, the initial disparity map can be used as the starting point for optimization, and optimization processing can be performed in a disparity iterative network to obtain a disparity map. Here, the disparity iterative network refers to a network structure that gradually corrects the disparity through multiple rounds of residual updates or a cyclic structure, used to improve the detailed representation and overall consistency of disparity estimation. This will enable the final disparity map to have higher accuracy in complex scenes.

[0090] In a scenario where improved disparity estimation accuracy is required, it's desirable to further optimize the disparity results based on the initial disparity. In this case, the initial disparity map can be input into an iterative disparity network to progressively correct the disparity values ​​over multiple updates, ultimately generating a disparity map of the left view relative to the right view. Through this iterative optimization process, the disparity map maintains more stable performance in areas with weak textures, occluded regions, and structural edges.

[0091] In some embodiments of this application, in order to progressively correct the disparity estimate over multiple iterations, a disparity residual can be generated based on a dual-core GRU, and the current disparity can be updated. This update process can be formalized as follows: , Here This represents the disparity map for the k-th iteration; This refers to the disparity residual output by the dual-core GRU. This allows the iterative optimization process to adaptively adjust in each round based on local cost volume features, contextual features, and the current disparity, thereby improving the overall accuracy and detail of disparity estimation.

[0092] In some embodiments of this application, in order to sample the cost volume based on the current disparity estimation result during the iteration process, it is necessary to map pixels in the first image to corresponding positions in the second image. This mapping relationship can be formalized as follows: , Here This refers to the pixel coordinates in the first image; This refers to the disparity value of the pixel in the k-th iteration; This refers to the coordinates of the matching points mapped onto the second image. This allows the iterative process to adaptively sample the cost volume based on the current disparity in each round, thereby improving the accuracy of local matching.

[0093] In some embodiments of this application, in order to obtain a more stable matching cost within a local neighborhood, a set of sampling points can be constructed in the second image based on the current disparity. This set of sampling points can be formalized as follows: , Here Refers to pixels Centered on, with a sampling radius of The local sampling point set; This refers to the offset within the sampling radius. This will make the cost volume sampling process more robust in local areas, especially in areas with weak textures or occlusion.

[0094] In some embodiments of this application, in order to extract local matching information from the cost body based on the sampling point set, bilinear sampling can be performed on the cost body to obtain local cost body features. This will enable the local cost volume features to provide more detailed matching information during the iteration process.

[0095] It is important to emphasize that, based on common configurations in engineering practice, the first and second images typically originate from the views of two imaging devices used in conjunction with each other, such as the left and right eye cameras commonly found in stereo vision systems. However, from the perspective of the technical solution of this application, the acquisition method of the two images is not limited to a fixed hardware combination or a specific installation structure. As long as the two images possess the necessary conditions for establishing pixel correspondence, such as a certain range of field-of-view overlap, texture information for matching, or satisfaction of preset geometric constraints, they can be used as the first and second images for subsequent processing. Therefore, this application does not require that the images be acquired from a specific type of camera, nor does it limit the relative positions of the two images to a horizontal arrangement; any image pair that meets the above engineering requirements is applicable to the technical solution of this application.

[0096] In summary, in this embodiment, by setting target first image features in at least two first image features of the first image for prior inference of the pixel disparity probability distribution of the first image, the disparity estimation process can obtain a monocular disparity distribution prior before binocular matching. This provides initial disparity distribution constraints for regions with weak textures or lacking obvious matching cues, reducing disparity instability that easily occurs when relying solely on binocular matching costs. Subsequently, the target first image features are geometrically enhanced based on the remaining image features, allowing the monocular disparity distribution prior to incorporate enhancement information including local structural changes while fusing the binocular matching costs. In this way, the binocular matching relationship can correct the monocular prior, reducing the possibility of bias in complex scenes; simultaneously, the enhancement information can strengthen the response of object edges and structurally changing regions, improving the problems of disparity boundary blurring and detail loss. Finally, the disparity map of the first image to the second image is determined based on the enhanced features, so that the final disparity estimation result benefits from multiple constraints provided by the monocular disparity distribution prior, the binocular matching costs, and gradient feature enhancement. Therefore, the method based on the embodiments of this application can make full use of the structural information provided by monocular prior in weak texture regions and make full use of local boundary information in structural edge regions, thereby improving the stability and detail preservation ability of disparity maps in complex scenes and reducing the probability of phenomena such as unstable disparity estimation, disparity blurring and unclear boundaries.

[0097] Figure 4 This application provides a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, comprising the following steps: Step 301: Update the parameters of the target data layer in the disparity iterative model based on the joint loss constructed from multiple disparity supervision data used to supervise the disparity estimation process.

[0098] In this context, the target data layer is used in the disparity iteration model to implement the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement disclosed in the above embodiments.

[0099] In some embodiments of this application, considering that the disparity iterative model needs to continuously adjust its internal parameters during training to more accurately execute the aforementioned binocular stereo matching method based on monocular disparity distribution priors and gradient feature enhancement, multiple supervisory information is typically used to jointly construct the training objective. Specifically, the parameters of the target data layer in the disparity iterative model can be updated based on a joint loss constructed from multiple disparity supervision data used to supervise the disparity estimation process. Here, the target data layer refers to the trainable layer in the disparity iterative model that undertakes functions such as disparity update, feature fusion, or error backpropagation, and is used to implement the various processing steps in the aforementioned embodiments (e.g., steps 101 to 103, and steps 201 to 205 and their sub-steps). This allows the target data layer to gradually adjust its parameters under the constraint of the joint loss, enabling the model to have a more stable disparity estimation capability in different scenarios.

[0100] In a training scenario where multi-source supervision is needed to improve disparity estimation accuracy, the disparity iterative model needs to continuously optimize its internal parameters through multiple training rounds. In this case, supervision data such as ground truth disparity values, monocular disparity bias, and disparity gradient errors can be used together to construct a joint loss, and this joint loss can be used to update the parameters of the target data layer in the model. After completing this training process, the target data layer can form a more reasonable parameter distribution under various supervision constraints, enabling the model to more stably perform the disparity inference and feature enhancement processing described in the previous steps when performing subsequent binocular stereo matching tasks.

[0101] In summary, in this embodiment, by setting target first image features in at least two first image features of the first image for prior inference of the pixel disparity probability distribution of the first image, the disparity estimation process can obtain a monocular disparity distribution prior before binocular matching. This provides initial disparity distribution constraints for regions with weak textures or lacking obvious matching cues, reducing disparity instability that easily occurs when relying solely on binocular matching costs. Subsequently, the target first image features are geometrically enhanced based on the remaining image features, allowing the monocular disparity distribution prior to incorporate enhancement information including local structural changes while fusing the binocular matching costs. In this way, the binocular matching relationship can correct the monocular prior, reducing the possibility of bias in complex scenes; simultaneously, the enhancement information can strengthen the response of object edges and structurally changing regions, improving the problems of disparity boundary blurring and detail loss. Finally, the disparity map of the first image to the second image is determined based on the enhanced features, so that the final disparity estimation result benefits from multiple constraints provided by the monocular disparity distribution prior, the binocular matching costs, and gradient feature enhancement. Therefore, the method based on the embodiments of this application can make full use of the structural information provided by monocular prior in weak texture regions and make full use of local boundary information in structural edge regions, thereby improving the stability and detail preservation ability of disparity maps in complex scenes and reducing the probability of phenomena such as unstable disparity estimation, disparity blurring and unclear boundaries.

[0102] Figure 5 This application provides another binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, comprising the following steps: Step 401: Establish a joint loss for the target data layer based on the first constraint, the second constraint, and the third constraint.

[0103] In the disparity iteration model, the target data layer is used to implement the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement disclosed in the above embodiments; the first constraint condition is used to characterize the disparity error formed by the disparity map obtained during the iteration process relative to the disparity ground truth; the second constraint condition is used to characterize the monocular disparity deviation of the target first image feature relative to the disparity ground truth; and the third constraint condition is used to characterize the gradient error formed by the gradient of the disparity map obtained during the iteration process relative to the gradient of the disparity ground truth.

[0104] In some embodiments of this application, considering that the disparity iterative model needs to simultaneously meet multiple supervision requirements during training in order to more accurately execute the aforementioned binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, supervision information from different sources is usually integrated into a unified training objective. Specifically, a joint loss for the target data layer can be established based on a first constraint, a second constraint, and a third constraint. Here, the target data layer refers to the trainable layer that undertakes functions such as disparity update, error backpropagation, or feature fusion in the disparity iterative model; the first constraint is used to characterize the disparity error of the disparity map generated during the iteration process relative to the ground truth disparity; the second constraint is used to characterize the monocular disparity bias of the target first image features relative to the ground truth disparity; and the third constraint is used to characterize the gradient error of the gradient of the disparity map generated during the iteration process relative to the gradient of the ground truth disparity. This allows the joint loss to guide model learning under multi-dimensional constraints, enabling the target data layer to gradually form a more stable parameter distribution during training.

[0105] In a training scenario where multiple types of supervision are needed to improve disparity estimation accuracy, the disparity iterative model needs to learn its parameters under different types of error signal constraints. In this case, disparity error, monocular disparity bias, and disparity gradient error can be used as the first, second, and third constraints, respectively, and these three can be used together to construct a joint loss. After obtaining this joint loss, the training process can guide model optimization under the combined effect of multi-source supervision, enabling the target data layer to have a more reasonable parameter update direction in subsequent training.

[0106] In some embodiments of this application, to quantify the above three types of constraints, corresponding loss terms can be constructed based on different supervision objectives. Specifically, the first constraint, used to characterize the disparity error of the disparity map generated during the iteration process relative to the true disparity value, can be implemented by applying a Smooth L1 loss and a weighted exponentially growing L1 loss to the initial disparity and the disparity of each iteration, respectively. This constraint can be formalized as follows: , Here This refers to the true value of parallax; It is the number of disparity iterations; This is an exponential weighting factor (based on an example from this application, it can be 0.9), used to gradually increase the supervision intensity of disparity in later iterations. This allows the iterative network to simultaneously focus on the overall trend of the initial disparity and the gradual convergence of the iterative disparity during the training phase.

[0107] The second constraint, used to characterize the monocular disparity deviation of the first image feature of the target relative to the true disparity value, can be implemented by applying a Smooth L1 loss to the monocular disparity. This constraint can be formalized as follows: , Here This refers to monocular disparity obtained based on the monocular disparity distribution prior, used to express the probability trend of the first image under different disparity assumptions. This allows the monocular disparity distribution prior to obtain more stable structural constraints during training, providing a more reliable geometric prior for weakly textured regions and unmatched regions.

[0108] The third constraint, used to characterize the gradient error of the disparity map generated during the iteration process relative to the ground truth disparity gradient, can be achieved by applying a Smooth L1 loss and an exponentially weighted L1 loss to the initial and iterative disparity gradients. This constraint can be formalized as follows: , in Here and These correspond to the gradients of the initial disparity and the disparity in the i-th iteration, respectively. The disparity map is in The gradient is calculated in the direction. This allows the training process to pay more attention to the disparity consistency between edge regions and regions of structural change, thereby improving the detail representation of the final disparity map.

[0109] By using the three types of loss mentioned above to construct a joint loss, the training process can guide the model learning under multiple constraints of disparity error, monocular disparity bias, and disparity gradient error, enabling the target data layer to gradually form a more stable parameter distribution during the training phase.

[0110] Step 402: Update the joint parameters of all target data layers based on the joint loss.

[0111] In some embodiments of this application, in order to continuously optimize the internal structure of the disparity iterative model under the constraint of joint loss, the joint loss needs to be used to drive the parameter updates of relevant layers in the model. Specifically, joint parameter updates can be performed on all target data layers based on the joint loss. Here, joint parameter updates refer to using the joint loss to simultaneously perform gradient backpropagation and parameter adjustment on multiple target data layers, enabling the model to gradually improve its disparity estimation capability under the combined influence of different supervision signals. This will enable the model to form a more stable optimization path during training, thereby achieving more reliable disparity inference performance when performing binocular stereo matching tasks in the future.

[0112] In a scenario where the model needs to continuously improve its disparity estimation performance over multiple training rounds, it's necessary to optimize the model as a whole based on the joint loss constructed in the previous step. In this case, the joint loss can be input into the training framework to perform joint parameter updates on all target data layers. After this update process, multiple key layers of the model can obtain more reasonable parameter adjustments under joint supervision, enabling the subsequent disparity inference process to maintain more stable performance in different scenarios.

[0113] like Figure 1 As shown, under the same input conditions of left image (column (a)) and right image (column (b)), comparing the disparity estimation results of the IGEV-Stereo method (column (c)) and the method of this application (column (d)) in several typical street scene scenarios, it can be seen that the method of this application exhibits higher stability and geometric consistency in ill-conditioned regions and structural detail regions. Specifically, in areas such as reflective vehicle bodies, distant building edges, vehicle outlines, and weakly textured road surfaces, the disparity map of IGEV-Stereo exhibits obvious local blurring, boundary diffusion, or structural breakage, while the method of this application can maintain clearer edge contours, more continuous structural expression, and smoother depth transitions in the same areas. The magnified local areas marked in the figure further demonstrate that the method of this application can effectively suppress the error accumulation caused by cost volume instability in areas of bright reflection, occlusion boundaries, and abrupt detail changes, so that the disparity estimation results still maintain high reliability under complex lighting and geometric change conditions.

[0114] Furthermore, in some embodiments of this application, to comprehensively verify the stereo matching performance of the method under different scene types, regional characteristics, and various ill-conditioned conditions, systematic quantitative experiments were conducted based on multiple public datasets. The experiments covered typical test environments such as large-scale synthetic scenes, real-world acquisition scenes, road driving scenes, specular regions (SR), and edge regions (ER), and were compared and evaluated with various advanced stereo matching methods. For ease of explanation, the experimental results are summarized in Tables 1 to 5, and the experimental settings, comparison methods, and performance of various scenarios are explained to demonstrate the comprehensive advantages of this method in terms of overall accuracy, detail preservation ability, and robustness to ill-conditioned regions.

[0115] In some embodiments of this application, the Scene Flow dataset was selected for evaluation to verify the stereo matching performance of the method in large-scale synthetic scenes. This dataset contains rich motion scenes and complex geometric structures, and is a commonly used benchmark for evaluating the end-to-end accuracy of stereo matching algorithms. The experiments used mean endpoint error (EPE) and three-pixel error (Bad 3.0, Bad3.0) as the main evaluation metrics. The experimental results on the Scene Flow dataset are shown in Table 1, which lists the performance of various advanced stereo matching methods under the above metrics: Table 1 Experimental results of the SceneFlow dataset

[0116] As can be seen from Table 1, the method of this application achieves the lowest error in both EPE and Bad 3.0 indicators, and has higher matching accuracy and more stable geometric consistency compared with existing methods.

[0117] In some embodiments of this application, the ETH3D dataset was selected for evaluation to verify the robustness of the method in real-world acquisition scenarios. This dataset features high-precision depth annotation and uses Bad 0.5, Bad 1.0, and End Point Error (EPE) as the main evaluation metrics, covering non-occluded (Noc) and all regions, respectively. The experimental results on the ETH3D dataset are shown in Table 2, listing the performance of various advanced methods under different regions and error thresholds. Table 2 Experimental results of the ETH3D dataset

[0118] As can be seen from Table 2, the method of this application achieves lower errors across all metrics, especially in non-occluded areas, indicating that the method has stronger detail preservation capabilities and higher matching stability in real-world scenarios.

[0119] In some embodiments of this application, the KITTI2015 dataset was selected for evaluation to verify the engineering application capability of the method in real-world driving scenarios. This dataset uses background region error (D1 background, D1 bg) and total region error (D1 all, D1 all) as the main evaluation metrics, which can reflect the stability of the algorithm under complex lighting, occlusion, and dynamic object conditions. The experimental results of KITTI2015 are shown in Table 3, listing the performance of various advanced methods under the above metrics: Table 3 Experimental Results of the KITTI2015 Dataset

[0120] As can be seen from Table 3, the method of this application achieves lower errors in all indicators, indicating that it has stronger robustness in real road scenarios and can effectively cope with practical challenges such as complex lighting, dynamic objects and occlusion.

[0121] In some embodiments of this application, to verify the matching ability of the method in typical ill-conditioned regions such as specular regions (SR), the reflective regions in the KITTI2012 dataset were selected for specific evaluation. These regions contain a large amount of specular reflection and bright materials, making them one of the scenarios where stereo matching algorithms are prone to error accumulation. Bad 3.0, Bad 4.0, and Bad 5.0 were used as the main evaluation metrics in the experiments. The experimental results for the reflective regions are shown in Table 4, which lists the performance of various advanced methods under different error thresholds: Table 4 Experimental results of reflective areas in the KITTI2012 dataset

[0122] As can be seen from Table 4, the method of this application achieves the best results in all reflective region indices, indicating that it has stronger geometric consistency and matching stability when dealing with specular reflection and high-brightness regions, and significantly improves the error accumulation problem of traditional methods in such regions.

[0123] In some embodiments of this application, to verify the detail recovery capability of the method in edge regions (ER) and non-edge regions (NER), the Scene Flow test set is divided into edge regions and non-edge regions based on the Canny EdgeDetector (CED), and evaluated separately. The experiments use the mean end-point error (EPE) and Bad 1.0 as the main evaluation metrics. The experimental results for edge and non-edge regions are shown in Table 5, listing the performance of various advanced methods in these two types of regions: Table 5 Experimental results for different regions of the Scene Flow dataset.

[0124] As can be seen from Table 5, the method of this application achieves the best results in both edge and non-edge regions. In particular, the error is significantly reduced in the edge region, indicating that the method has a stronger ability to preserve details in regions of structural abrupt changes and can effectively alleviate edge blurring and parallax jump problems.

[0125] The experimental results above demonstrate that, under various typical test environments, including large-scale synthetic scenes, realistic acquisition scenes, road driving scenes, reflective areas, and edge and non-edge regions, the proposed method outperforms existing state-of-the-art methods in terms of overall error, detail preservation, and robustness to ill-conditioned regions. Based on the comprehensive evaluation results, this method maintains stable disparity estimation quality under diverse practical challenges such as complex lighting, structural abrupt changes, weak textures, and reflection interference, exhibiting high engineering applicability and promotional value.

[0126] In summary, in this embodiment, by setting target first image features in at least two first image features of the first image for prior inference of the pixel disparity probability distribution of the first image, the disparity estimation process can obtain a monocular disparity distribution prior before binocular matching. This provides initial disparity distribution constraints for regions with weak textures or lacking obvious matching cues, reducing disparity instability that easily occurs when relying solely on binocular matching costs. Subsequently, the target first image features are geometrically enhanced based on the remaining image features, allowing the monocular disparity distribution prior to incorporate enhancement information including local structural changes while fusing the binocular matching costs. In this way, the binocular matching relationship can correct the monocular prior, reducing the possibility of bias in complex scenes; simultaneously, the enhancement information can strengthen the response of object edges and structurally changing regions, improving the problems of disparity boundary blurring and detail loss. Finally, the disparity map of the first image to the second image is determined based on the enhanced features, so that the final disparity estimation result benefits from multiple constraints provided by the monocular disparity distribution prior, the binocular matching costs, and gradient feature enhancement. Therefore, the method based on the embodiments of this application can make full use of the structural information provided by monocular prior in weak texture regions and make full use of local boundary information in structural edge regions, thereby improving the stability and detail preservation ability of disparity maps in complex scenes and reducing the probability of phenomena such as unstable disparity estimation, disparity blurring and unclear boundaries.

[0127] like Figure 6 As shown in the illustration, this application also discloses a binocular stereo matching device 50 based on monocular disparity distribution prior and gradient feature enhancement, comprising: The feature extraction module 501 is used to determine at least two first image features of the first image and at least one second image feature of the second image; the first image features include a target first image feature for prior inference of the pixel disparity probability distribution of the first image. The feature enhancement module 502 is used to geometrically enhance the features of the target first image based on all the first image features other than the features of the target first image, as well as all the second image features, to obtain the enhanced features of the first image; The disparity calculation module 503 is used to determine the disparity map of the first image to the second image based on the enhanced features.

[0128] Optionally, the feature extraction module 501 includes: The first feature submodule is used to determine the embedding data of the first image and the second image respectively, and to determine the image gradient data of the first image, and to determine the embedding data and image gradient data of the first image as a first image feature other than a target first image feature; The second feature submodule is used to input the first image into the prior inference network to obtain the monocular disparity distribution prior tensor of the first image as the feature of the target first image.

[0129] Optionally, the first feature submodule includes: A gradient coding unit is used to perform gradient calculation on the first image to obtain a gradient map that characterizes the numerical changes of each pixel in the first image with respect to pixels in adjacent rows and columns. The gradient feature unit is used to input the gradient map into the gradient encoding network to use the gradient features of the obtained first image as image gradient data.

[0130] Optionally, the feature enhancement module 502 includes: The matching cost submodule is used to determine the three-dimensional tensor of the matching cost of the second image to the first image based on the similarity relationship between the embedded data of the first image and the embedded data of the second image. The feature enhancement submodule is used to enhance the monocular disparity distribution prior tensor by matching the cost 3D tensor and the image gradient data of the first image, so as to obtain the enhanced features of the first image under the second image.

[0131] Optionally, the matching cost submodule includes: The first matching unit is used to construct a first matching correlation tensor of the second image to the first image based on the similarity relationship between the embedded data of the first image and the embedded data of the second image under multiple different disparity assumptions; The second matching unit is used to group the embedded data of the first image and the embedded data of the second image in the channel dimension, respectively, and construct the second matching correlation tensor of the second image to the first image based on the similarity relationship within each channel group. The matching cost unit is used to fuse the first matching correlation tensor and the second matching correlation tensor to generate a three-dimensional tensor of the matching cost of the second image to the first image.

[0132] Optionally, the feature enhancement submodule includes: The first enhancement unit is used to perform feature modulation processing on the monocular disparity distribution prior tensor based on the matching cost three-dimensional tensor to obtain the first enhancement feature for characterizing the binocular matching relationship. The second enhancement unit is used to perform gradient-sensitive enhancement processing on the first enhancement feature based on image gradient data to obtain a second enhancement feature for characterizing local structural changes. The fusion enhancement unit is used to fuse the first enhancement feature and the second enhancement feature to obtain the enhancement feature.

[0133] Optionally, the parallax calculation module 503 includes: The disparity feature submodule is used to determine the disparity features used to characterize disparity information by matching the enhancement features under different disparity assumptions. The initial submodule is used to perform disparity regression processing on the disparity values ​​of each pixel in the first image based on disparity features to obtain an initial disparity map of the first image to the second image. The iterative submodule is used to perform optimization processing in the disparity iterative network with the initial disparity map as the starting point to obtain the disparity map.

[0134] like Figure 7 As shown in the figure, this application also discloses a binocular stereo matching device 60 based on monocular disparity distribution prior and gradient feature enhancement, comprising: The model training module 601 is used to update the parameters of the target data layer in the disparity iterative model based on the joint loss constructed from multiple disparity supervision data used to supervise the disparity estimation process; the target data layer is used in the disparity iterative model to implement the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement disclosed in the above embodiments.

[0135] Optionally, the model training module 601 includes: The loss submodule is used to establish a joint loss for the target data layer based on the first constraint, the second constraint, and the third constraint. The first constraint is used to characterize the disparity error formed by the disparity map obtained during the iteration process relative to the true disparity value. The second constraint is used to characterize the monocular disparity bias of the target first image features relative to the true disparity value. The third constraint is used to characterize the gradient error formed by the gradient of the disparity map obtained during the iteration process relative to the gradient of the true disparity value. The update submodule is used to update the joint parameters of all target data layers based on the joint loss.

[0136] In summary, in this embodiment, by setting target first image features in at least two first image features of the first image for prior inference of the pixel disparity probability distribution of the first image, the disparity estimation process can obtain a monocular disparity distribution prior before binocular matching. This provides initial disparity distribution constraints for regions with weak textures or lacking obvious matching cues, reducing disparity instability that easily occurs when relying solely on binocular matching costs. Subsequently, the target first image features are geometrically enhanced based on the remaining image features, allowing the monocular disparity distribution prior to incorporate enhancement information including local structural changes while fusing the binocular matching costs. In this way, the binocular matching relationship can correct the monocular prior, reducing the possibility of bias in complex scenes; simultaneously, the enhancement information can strengthen the response of object edges and structurally changing regions, improving the problems of disparity boundary blurring and detail loss. Finally, the disparity map of the first image to the second image is determined based on the enhanced features, so that the final disparity estimation result benefits from multiple constraints provided by the monocular disparity distribution prior, the binocular matching costs, and gradient feature enhancement. Therefore, the method based on the embodiments of this application can make full use of the structural information provided by monocular prior in weak texture regions and make full use of local boundary information in structural edge regions, thereby improving the stability and detail preservation ability of disparity maps in complex scenes and reducing the probability of phenomena such as unstable disparity estimation, disparity blurring and unclear boundaries.

[0137] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described embodiments of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0138] Figure 8 This is a block diagram of an electronic device 700 provided in an embodiment of this application. For example, the electronic device 700 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0139] Reference Figure 8 The electronic device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output interface 712, sensor component 714, and communication component 716.

[0140] The processing component 702 controls the overall operation of the electronic device 700, including functions such as display, communication, data communication, camera, and recording. The processing component 702 may include one or more processors 720 for executing instructions to complete all or part of the steps of the aforementioned binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement. The processing component 702 may also include several modules for interacting with other components, such as a multimedia module for interacting with the multimedia component 708.

[0141] The memory 704 is used to store data and instructions that support the operation of the electronic device 700, including applications, contacts, messages, pictures, and multimedia content. The memory 704 can be volatile or non-volatile memory, or a combination thereof, such as SRAM, EEPROM, EPROM, PROM, ROM, flash memory, magnetic disk, or optical disk.

[0142] The power supply assembly 706 is used to supply power to the various components of the electronic device 700, and may include a power management system and one or more power supplies.

[0143] The multimedia component 708 may include a display screen for outputting information to a user. In some embodiments, the display screen includes an LCD and a touch panel that can receive user input as a touchscreen. The touch panel can detect touches, swipes, and related gestures. In some embodiments, the multimedia component 708 also includes a front-facing and / or rear-facing camera for capturing or receiving multimedia data.

[0144] Audio component 710 is used for inputting and outputting audio signals and may include a microphone and a speaker. The microphone can receive audio signals and store them or transmit them via communication component 716 in call, recording, or voice recognition modes.

[0145] I / O interface 712 is used to connect the processing component 702 to peripheral interface modules such as keyboard, buttons, home button, volume buttons, power button and lock button.

[0146] Sensor assembly 714 may include multiple sensors for detecting the state of electronic device 700, such as switch state, position change, contact detection, orientation, acceleration, or temperature. Sensor assembly 714 may include proximity sensors, optical sensors (such as CMOS or CCD), accelerometers, gyroscopes, magnetic sensors, pressure sensors, or temperature sensors.

[0147] Communication component 716 enables wired or wireless communication between electronic device 700 and external devices. Electronic device 700 may access WiFi, carrier networks (such as 2G, 3G, 4G, or 7G), etc. In some embodiments, communication component 716 may receive broadcast signals or include an NFC module to support short-range communication, such as based on RFID, IrDA, UWB, or Bluetooth technologies.

[0148] In some embodiments, the electronic device 700 may be implemented by electronic components such as ASIC, DSP, PLD, FPGA, controller, microcontroller, and microprocessor to execute the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement according to the embodiments of this application.

[0149] In some embodiments, a non-transitory computer-readable storage medium, such as memory 704, is also provided, which includes instructions executable by processor 720 to perform the methods described above. The storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical storage device, etc.

[0150] In some embodiments, the electronic device 700 can be used as a server, including a processing component 702 (including one or more processors 720) and a memory 704 for storing and executing applications. The processing component 702 can execute instructions to implement the binocular stereo matching method of the embodiments of this application.

[0151] Other embodiments of this application will readily conceive of by those skilled in the art upon reading this specification. This application is intended to cover various modifications, uses, or adaptations within the scope of its basic principles, including techniques known in the art. The specification and embodiments are for illustrative purposes only, and the scope of protection is defined by the claims.

[0152] It should be understood that this application is not limited to the structure shown in the accompanying drawings, and various modifications can be made without departing from the scope of this application. The embodiments in this specification are described in a progressive manner, and the same content can be referred to in turn. Any combination of the various embodiments is also a implementation method of this application.

[0153] The method of this application does not depend on a specific computer or device and can be implemented on a variety of general-purpose systems. The system architecture required to implement the solution of this application is obvious to those skilled in the art. This application is also not limited to a specific programming language; various languages ​​can be used to implement the content of this application.

[0154] For ease of understanding, the specification contains several specific details, but the implementation of this application is not necessarily subject to these details. Some well-known methods or structures are not described in detail to avoid redundancy.

[0155] The features of this application may be described in a concentrated manner in the exemplary embodiments, but should not be construed as requiring the claims to include all of these features. Each of the claims can be considered as an independent embodiment of this application.

[0156] Those skilled in the art can adjust the modules in the embodiments as needed, and can merge or split multiple modules. Unless there is an explicit exclusion relationship between the features, the features disclosed in this specification can be arbitrarily combined and can be replaced by alternative features with the same function.

[0157] The components of this application can be implemented by hardware, software, or a combination thereof. Microprocessors or digital signal processors can be used to implement all or part of the functions of the methods in this application. This application can also be implemented by a device program, which can be stored in a computer-readable medium or provided in the form of signals.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement.

[0159] The above embodiments can be implemented by software, hardware, firmware, or a combination thereof. When implemented in software form, it can be provided as a computer program product, which, after being loaded and executed on a computer, can realize all or part of the functions of this application. Computer instructions can be stored in a computer-readable medium or transmitted between different media via wired or wireless means.

[0160] The above embodiments are illustrative of this application and not intended to limit it. Those skilled in the art can make various substitutions or improvements to this application without departing from the scope of the claims. The word "comprising" does not exclude the presence of elements or steps not listed, and "a" or "an" does not exclude the presence of multiple similar elements. Reference numerals should not be considered as limitations on the claims.

Claims

1. A binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, characterized in that, include: Determine at least two first image features of a first image and at least one second image feature of a second image; the first image features include a target first image feature for prior inference of the pixel disparity probability distribution of the first image; Based on all first image features other than the target first image features, and all second image features, geometric enhancement is performed on the target first image features to obtain the enhanced features of the first image; Based on the enhanced features, a disparity map of the first image to the second image is determined; The step of determining at least two first image features of the first image and at least one second image feature of the second image includes: determining the embedding data of the first image and the second image respectively; performing gradient calculation on the first image to obtain a gradient map characterizing the numerical changes of each pixel of the first image with respect to pixels in adjacent rows and columns; inputting the gradient map into a gradient encoding network to use the obtained gradient features of the first image as image gradient data; and determining the embedding data and image gradient data of the first image as first image features other than the target first image feature respectively; and inputting the first image into a prior inference network to use the obtained monocular disparity distribution prior tensor of the first image as the target first image feature. The step of geometrically enhancing the target first image features based on all first image features other than the target first image features, and all second image features, to obtain enhanced features of the first image includes: determining a three-dimensional tensor of the matching cost of the second image to the first image based on the similarity relationship between the embedded data of the first image and the embedded data of the second image; enhancing the monocular disparity distribution prior tensor with the matching cost three-dimensional tensor and the image gradient data of the first image to obtain enhanced features of the first image under the second image includes: performing feature modulation processing on the monocular disparity distribution prior tensor based on the matching cost three-dimensional tensor to obtain a first enhanced feature for characterizing the binocular matching relationship; performing gradient sensitivity enhancement processing on the first enhanced feature based on the image gradient data to obtain a second enhanced feature for characterizing local structural changes; and fusing the first enhanced feature and the second enhanced feature to obtain the enhanced feature.

2. The binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement as described in claim 1, characterized in that, Determining the disparity map of the first image to the second image based on the enhanced features includes: Based on the matching relationship of the enhanced features under different disparity assumptions, determine the disparity features used to characterize disparity information; Based on the disparity features, disparity regression processing is performed on the disparity values ​​of each pixel in the first image to obtain an initial disparity map of the first image relative to the second image. The initial disparity map is used as the starting point for optimization in the disparity iterative network to obtain the disparity map.

3. A binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement, characterized in that, include: The parameters of the target data layer in the disparity iterative model are updated based on the joint loss constructed from multiple disparity supervision data used to supervise the disparity estimation process. The target data layer in the disparity iteration model is used to implement the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement as described in any one of claims 1 to 2.

4. A binocular stereo matching device based on monocular disparity distribution prior and gradient feature enhancement, characterized in that, include: The feature extraction module is used to determine at least two first image features of a first image and at least one second image feature of a second image; the first image features include a target first image feature for prior inference of the pixel disparity probability distribution of the first image; The feature enhancement module is used to geometrically enhance the features of the target first image based on all the first image features other than the target first image features, and all the second image features, so as to obtain the enhanced features of the first image; A disparity calculation module is used to determine the disparity map of the first image to the second image based on the enhancement features; The feature extraction module is further configured to: determine the embedding data of the first image and the second image respectively; perform gradient calculation on the first image to obtain a gradient map representing the numerical changes of each pixel of the first image with respect to pixels in adjacent rows and columns; input the gradient map into a gradient encoding network to use the obtained gradient features of the first image as image gradient data; and determine the embedding data and image gradient data of the first image as a first image feature other than the target first image feature; input the first image into a prior inference network to use the obtained monocular disparity distribution prior tensor of the first image as the target first image feature; The feature enhancement module is further configured to: determine the matching cost three-dimensional tensor of the second image to the first image based on the similarity relationship between the embedded data of the first image and the embedded data of the second image; enhance the monocular disparity distribution prior tensor by using the matching cost three-dimensional tensor and the image gradient data of the first image to obtain enhanced features of the first image under the second image, including: performing feature modulation processing on the monocular disparity distribution prior tensor based on the matching cost three-dimensional tensor to obtain a first enhanced feature for characterizing the binocular matching relationship; performing gradient sensitivity enhancement processing on the first enhanced feature based on the image gradient data to obtain a second enhanced feature for characterizing local structural changes; and fusing the first enhanced feature and the second enhanced feature to obtain the enhanced feature.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement as described in any one of claims 1 to 3.

6. An electronic device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the binocular stereo matching method based on monocular disparity distribution prior and gradient feature enhancement as described in any one of claims 1 to 3.

Images