An efficient stereo matching method and device for large disparity scenes

Abstract

The application provides a high-efficiency stereo matching method and device for a large-parallax scene, feature extraction is performed on a first image and a second image, pixels in the first image and pixels in the second image are matched and optimized according to a small-parallax range, a medium-parallax range and a large-parallax range respectively, a matching relationship group under each parallax range is obtained, a selective weight is obtained according to the first image and the matching relationship group, the matching relationship group under each parallax range is screened according to the selective weight, and an effective matching relationship under each parallax range is obtained as a fusion feature; the initial disparity map is iteratively optimized through the fusion feature, and a final disparity map is obtained; the matching relationship group under multiple parallax ranges is constructed through the matching of the pixels in the image under different parallax ranges, the calculation and prediction of the small-parallax range, the medium-parallax range and the large-parallax range are considered at the same time, and accurate prediction of the disparity map is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a high-efficiency stereo matching method and device for large disparity scene. BACKGROUND

[0002] Stereo matching is of great significance because it can infer the geometric structure of a three-dimensional scene from multiple captured images, and its application range covers fields such as three-dimensional reconstruction, robotics and autonomous driving. The key to stereo matching is to find the corresponding pixel positions in the left and right images taken by the two cameras at the same longitudinal coordinate height. The difference between these corresponding pixel positions is called disparity, which can be used to infer depth and reconstruct a three-dimensional scene.

[0003] In common stereo matching, small disparity scenes, medium disparity scenes and large disparity scenes are usually involved, where the small disparity scene refers to the pixel matching scene of the same long shot position in the left and right images, the medium disparity scene refers to the pixel matching scene of the same medium shot position in the left and right images, and the large disparity scene refers to the pixel matching scene of the same close shot position in the left and right images; the small disparity scene is because the objects are all long shots, and the single geometric structure in the long shot usually occupies fewer pixels in the image, and the objects in the medium disparity scene are all medium shots, and the single geometric structure in the medium shot also usually occupies not many pixels in the image, so the pixel range of the required disparity of the small disparity scene and the medium disparity scene is usually not large; while the objects in the large disparity scene are all close shots, and the single geometric structure in the close shot occupies more pixels in the image, and the pixel range of the required disparity is more, so the disparity range defined by the existing stereo matching method is difficult to consider the large disparity scene, and therefore it is difficult to accurately perform stereo matching between pixels and accurately predict the disparity map in the case of facing the large disparity scene.

[0004] In view of this, overcoming the defects of the prior art is a problem to be solved in the technical field. SUMMARY

[0005] The technical problem to be solved by the present application is how to accurately predict the disparity map while considering small disparity scenes, medium disparity scenes and large disparity scenes.

[0006] The present application adopts the following technical solutions:

[0007] In a first aspect, a high-efficiency stereo matching method for large disparity scenes is provided, comprising:

[0008] The pixels in the first image and the pixels in the second image are matched and optimized according to the small disparity range, the medium disparity range and the large disparity range respectively, and a matching relationship group of each pixel in the first image under the small disparity range, the medium disparity range and the large disparity range is obtained;

[0009] obtaining a selective weight based on the image features in the first image and the matching relationship group, screening the matching relationship group under each disparity range according to the selective weight, obtaining a valid matching relationship under each disparity range, and using the valid matching relationships under all the disparity ranges as fusion features;

[0010] An initial disparity map is obtained according to the matching relationship group in a small disparity range, and the initial disparity map is iteratively optimized according to the fusion feature to obtain a final disparity map.

[0011] Preferably, matching and optimizing the relationship between pixels in the first image and pixels in the second image according to the small parallax range, the medium parallax range, and the large parallax range, respectively, specifically includes:

[0012] Dividing pixels in the same vertical coordinate direction in the second image into a plurality of first preset pixel groups according to a small parallax range, dividing pixels in the same vertical coordinate direction in the second image into a plurality of second preset pixel groups according to a medium parallax range, and dividing pixels in the same vertical coordinate direction in the second image into a plurality of third preset pixel groups according to a large parallax range;

[0013] Matching each pixel in the first image with each pixel in each first preset pixel group according to corresponding matching weights to obtain a first initial relationship group; matching each pixel in the first image with each pixel in each second preset pixel group according to corresponding matching weights to obtain a second initial relationship group; matching each pixel in the first image with each pixel in each third preset pixel group according to corresponding matching weights to obtain a third initial relationship group;

[0014] The first initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a small parallax range; the second initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a medium parallax range; the first initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a large parallax range.

[0015] Preferably, each pixel in the first image is matched with each pixel in each first preset pixel group according to a corresponding matching weight to obtain a first initial relationship group, and a specific construction formula is:

[0016] ;

[0017] in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the small parallax range in the first initial relationship group, is the disparity range of the first initial relationship group, x is the horizontal coordinate of the pixel, and y is the vertical coordinate of the pixel.

[0018] Preferably, each pixel in the first image is matched with each pixel in each second preset pixel group according to a corresponding matching weight to obtain a second initial relationship group, and a specific construction formula is:

[0019] ;

[0020] in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the middle parallax range in the second initial relationship group, is the disparity range of the second initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the disparity range The serial number in is the matching weight of the i-th pixel in the second preset pixel group.

[0021] Preferably, each pixel in the first image is matched with each pixel in each third preset pixel group according to a corresponding matching weight to obtain a third initial relationship group, and a specific construction formula is:

[0022] ;

[0023] in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the large parallax range in the third initial relationship group, is the disparity range of the third initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the disparity range The serial number in is the matching weight of the i-th pixel in the third preset pixel group.

[0024] Preferably, obtaining the selective weight according to the image features in the first image and the matching relationship group specifically includes:

[0025] Obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a small parallax range and using it as the first matching relationship, obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a medium parallax range and using it as the second matching relationship, and obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a large parallax range and using it as the third matching relationship;

[0026] Performing convolution processing on the first matching relationship, the second matching relationship, and the third matching relationship to obtain a disparity feature set;

[0027] The disparity feature set and the image features of the first image are jointly subjected to secondary convolution processing to obtain a small disparity weight under a small disparity range, a medium disparity weight under a medium disparity range, and a large disparity weight under a large disparity range, respectively. The weight corresponding to the disparity range is selected from the small disparity weight, the medium disparity weight, and the large disparity weight as the selective weight.

[0028] Preferably, the matching relationship groups under each parallax range are screened according to the selective weight to obtain valid matching relationships under each parallax range, and the valid matching relationships under all parallax ranges are used as fusion features. The specific formula is:

[0029]

[0030] in, To fusion features, is the small disparity weight, is the matching relationship group under a small parallax range, is the median disparity weight, is the matching relationship group under the medium parallax range, is the large disparity weight, is the matching relationship group under a large parallax range, is the effective matching relationship under a small parallax range, is the effective matching relationship under the medium parallax range, This is an effective matching relationship under a large parallax range.

[0031] Preferably, obtaining an initial disparity map according to the matching relationship group in a small disparity range, and iteratively optimizing the initial disparity map according to the fusion feature to obtain a final disparity map specifically includes:

[0032] A preset number of iterations is set, and the initial disparity map is input as an input disparity map into the iteration. Each iteration includes:

[0033] Sampling the input disparity map and the fused features to a first preset resolution and performing a first convolution process to obtain a first hidden state feature;

[0034] Sampling the first hidden state feature to a second preset resolution and performing a second convolution process to obtain a second hidden state feature at a third preset resolution;

[0035] Determine whether the current number of iterations is greater than the preset number of iterations;

[0036] When the current iteration number is greater than or equal to the preset iteration number, iteration is stopped, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map to obtain the final disparity map.

[0037] When the current iteration number is less than or equal to the preset iteration number, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map and input into the next round of iteration as a new input disparity map.

[0038] In a second aspect, an efficient stereo matching device for a large disparity scene is provided, which includes at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the efficient stereo matching method for a large disparity scene.

[0039] In a third aspect, the present application further provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method of the first aspect.

[0040] In a fourth aspect, a chip is provided, which includes a processor and an interface, and is used to call and run a computer program stored in a memory to execute the method of the first aspect.

[0041] In a fifth aspect, a computer program product including instructions is provided, and when the instructions are executed on a computer or a processor, the computer or the processor executes the method of the first aspect.

[0042] In a sixth aspect, an efficient stereo matching system for a large disparity scene is provided, which includes the efficient stereo matching device for a large disparity scene of the second aspect and uses the efficient stereo matching method for a large disparity scene of the first aspect.

[0043] The application provides a high-efficiency stereo matching method and device for a large-disparity scene, feature extraction is performed on a first image and a second image, pixels in the first image and pixels in the second image are matched and optimized according to a small-disparity range, a medium-disparity range and a large-disparity range respectively, a matching relationship group under each disparity range is obtained, a selective weight is obtained according to the first image and the matching relationship group, the matching relationship group under each disparity range is screened according to the selective weight, and an effective matching relationship under each disparity range is obtained as a fusion feature; and the initial disparity map is iteratively optimized through the fusion feature, and a final disparity map is obtained. The matching relationship group under each disparity range is constructed by matching the pixels in the image under different disparity ranges, and the calculation and prediction of the small-disparity range, the medium-disparity range and the large-disparity range are considered, so that the accurate prediction of the disparity map is realized. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments of the application. Obviously, the drawings described below are only some of the embodiments of the application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0045] Figure 1 It is a method flowchart of a high-efficiency stereo matching method for a large-disparity scene provided by the embodiments of the application;

[0046] Figure 2 It is a method flowchart of obtaining the matching relationship group under each disparity range in the high-efficiency stereo matching method for a large-disparity scene provided by the embodiments of the application;

[0047] Figure 3 It is a method flowchart of obtaining the selective weight in the high-efficiency stereo matching method for a large-disparity scene provided by the embodiments of the application;

[0048] Figure 4 It is a method flowchart of iteratively optimizing the initial disparity map in the high-efficiency stereo matching method for a large-disparity scene provided by the embodiments of the application;

[0049] Figure 5 It is a system schematic diagram of a high-efficiency stereo matching system for a large-disparity scene provided by the embodiments of the application;

[0050] Figure 6 It is another system schematic diagram of a high-efficiency stereo matching system for a large-disparity scene provided by the embodiments of the application;

[0051] Figure 73 is a schematic diagram of an efficient stereo matching device for large parallax scenes provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] In the description of the present invention, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure.

[0054] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "multiple" means two or more. In addition, for example, the description may also use the method of adding "A" and "B" at the end to describe the same type of nouns as two independent individuals. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the description purposes of the same type of individuals, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0055] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and errors associated with measurement of the particular quantity (i.e., limitations of the measurement system).

[0056] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that the specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0057] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0058] Embodiment 1:

[0059] This embodiment provides an efficient stereo matching method for large parallax scenes, such as Figure 1 As shown, the method flow includes:

[0060] In step 101, the pixels in the first image and the pixels in the second image are matched and optimized according to the small parallax range, the medium parallax range and the large parallax range, respectively, to obtain a matching relationship group for each pixel in the first image under the small parallax range, the medium parallax range and the large parallax range.

[0061] In this embodiment, the first image and the second image can be obtained by capturing the same target using two cameras located at the same ordinate height. Due to the difference in the positions of the two cameras on the ordinate, the first image and the second image have different shooting angles, resulting in parallax. In this embodiment, in order to infer the actual three-dimensional scene structure based on the first image and the second image, it is necessary to match the pixels of the same object in the first image and the second image to obtain an accurate three-dimensional scene structure. In this embodiment, the first image and the second image can first be input into a feature extraction network, which extracts corresponding features from the first image and the second image for subsequent calculation of the matching relationship between the pixels of the two images.

[0062] The small disparity in the images refers to the disparity of distant objects captured by the two cameras. Since the distance of the object is inversely proportional to the disparity, the disparity of distant objects in the two images is often smaller. In this embodiment, matching pixels in the first image with pixels in the second image within a small disparity range means matching all pixels in the first image with pixels in the second image. This matching can be performed by matching a single pixel in the first image with every pixel at the same vertical coordinate in the second image. The set of all matching relationships within the small disparity range is the matching relationship group within the small disparity range.

[0063] The medium disparity in an image refers to the disparity of the mid-ground portion captured by the two cameras. Since the distance of the object is inversely proportional to the disparity, the disparity of objects in the mid-ground between the two images is greater than that of objects in the distant portion. Therefore, when matching pixels in the first image with pixels in the second image within the medium disparity range, a certain number of adjacent pixels on the same vertical coordinate in the second image can be grouped (for example, every two adjacent pixels on the same vertical coordinate are grouped). During actual matching, each pixel in the first image has a corresponding matching weight for each pixel in the second image. These matching weights are adaptively learned during the training process of the matching model. For each pixel in the first image, the pixel with the largest matching weight is selected from each group of adjacent pixels in the second image for matching. The set of all matching relationships within the medium disparity range is the matching relationship group within the medium disparity range.

[0064] The large parallax in the image refers to the parallax of the close-up portion captured by the two cameras. Since the distance of the subject is inversely proportional to the parallax, the parallax of the close-up object in the two images is larger than that of the distant and mid-range objects. When matching pixels in the first image with pixels in the second image in a large parallax range, it is necessary to group every certain number of adjacent pixels on the same vertical coordinate in the second image (for example, every four adjacent pixels on the same vertical coordinate are grouped). Similarly, during actual matching, each pixel in the second image has a corresponding matching weight for each pixel in the second image. The matching weight is adaptively learned by the matching model during training. Each pixel in the first image will select a pixel with the largest matching weight from each group of adjacent pixels in the second image for matching. The set of all matching relationships under a large parallax range is the matching relationship group under a large parallax range.

[0065] In summary, all pixels in the first image will be matched with multiple pixels in the second image in the small parallax range, the medium parallax range, and the large parallax range, that is, all pixels in the first image will undergo three rounds of matching.

[0066] However, in the actual three-dimensional structure that we ultimately need to construct, each pixel in the first image should have a correctly corresponding pixel point in the second image. For example, the same bicycle is photographed at a close point in both the first and second images. The upper end of the front pedal of the bicycle is correctly matched with the pixel points in the first image and the pixel points in the second image, while the upper end of the front pedal of the bicycle is incorrectly matched with the pixel points in the first image and the pixel points in other positions in the second image. Therefore, the matching model in this embodiment has a corresponding score for each matching relationship of a single pixel in the first image, which is used to judge whether the matching relationship is more correct, so that the matching relationship of each pixel is judged and screened based on the score in subsequent steps.

[0067] However, due to the influence of the external environment in the actual shooting scene or the errors caused by blurry shooting, the scores of some matching relationships may be too high or too low. Therefore, it is necessary to optimize the scores of all matching relationships so that the score change trend of all matching relationships tends to be smooth, avoiding the scores of some matching relationships being too prominent and affecting the correctness of the correspondence between pixel points.

[0068] In step 102, a selective weight is obtained based on the image features in the first image and the matching relationship group, and the matching relationship group under each disparity range is screened according to the selective weight to obtain a valid matching relationship under each disparity range, and the valid matching relationships under all disparity ranges are used as fusion features.

[0069] In this embodiment, since the matching of pixel points between the above-mentioned first image and the second image is performed in three rounds under different parallax ranges (i.e., one round of relationship matching is performed on all pixel points in the first image according to the small parallax range, one round of relationship matching is performed on all pixel points in the first image according to the medium parallax range, and one round of relationship matching is performed on all pixel points in the first image according to the range), but the parallax ranges of pixel points at different positions in the image are different, the correct matching between each pixel point needs to be performed in the corresponding parallax range. For example, the relationship matching between pixels in the large parallax part of the first image and the second image in the large parallax range belongs to the said valid matching relationship, while the relationship matching between pixels in the large parallax part of the first image and the second image in the medium parallax range or the small parallax range does not belong to the said valid matching relationship. In this embodiment, it is necessary to screen out all valid matching relationships in the matching relationship group.

[0070] In this embodiment, the selective weight is used to reflect the disparity range of different objects in the image. Therefore, the matching relationship with the correct disparity range can be screened out from the matching relationship group according to the selective weight as the effective matching relationship, and all effective matching relationships are collectively used as the fusion feature.

[0071] In step 103 , an initial disparity map is obtained according to the matching relationship group in a small disparity range, and the initial disparity map is iteratively optimized according to the fusion feature to obtain a final disparity map.

[0072] In this embodiment, since most pixels in the captured image typically fall within a small disparity range when the camera is shooting, in order to quickly obtain the initial disparity map required for iteration, the features obtained based on the matching relationship group within the small disparity range are used as the initial disparity map for input into subsequent iterations. In this embodiment, although the matching relationship between each pixel in the fused feature is selected based on the matching model score, the matching model score still has corresponding errors. Therefore, in order to ensure the accuracy of the final disparity map, it is necessary to convolute the initial disparity map and the fused feature at different scales and perform multiple rounds of iterative operations to converge the disparity map and obtain a relatively accurate final disparity map.

[0073] In this embodiment, the pixels in the first image and the pixels in the second image are matched and optimized according to the small parallax range, the medium parallax range and the large parallax range, respectively. Figure 2 As shown, the corresponding method flow is as follows:

[0074] In step 201, the pixels in the same vertical coordinate direction in the second image are divided into multiple first preset pixel groups according to a small parallax range, the pixels in the same vertical coordinate direction in the second image are divided into multiple second preset pixel groups according to a medium parallax range, and the pixels in the same vertical coordinate direction in the second image are divided into multiple third preset pixel groups according to a large parallax range.

[0075] In this embodiment, since the small parallax range corresponds to the distant portion of the image, the object structure in this portion occupies the least number of pixels in the image. Therefore, every adjacent pixel in the second image can be used as a first preset pixel group. Since the medium parallax range corresponds to the mid-ground portion of the image, the object structure in this portion occupies a relatively greater number of pixels in the image. Therefore, every adjacent two pixels in the second image can be used as a second preset pixel group. Since the large parallax range corresponds to the near-ground portion of the image, the object structure in this portion occupies the largest number of pixels in the image. Therefore, every adjacent four pixels in the second image can be used as a third preset pixel group.

[0076] In step 202, each pixel in the first image is matched with each pixel in each first preset pixel group according to the corresponding matching weight to obtain a first initial relationship group; each pixel in the first image is matched with each pixel in each second preset pixel group according to the corresponding matching weight to obtain a second initial relationship group; each pixel in the first image is matched with each pixel in each third preset pixel group according to the corresponding matching weight to obtain a third initial relationship group.

[0077] It should be noted that, when each pixel in the first image is matched with the first preset pixel group, the second preset pixel group, or the third preset pixel group, it will be matched with the pixel with the largest matching weight in each first preset pixel group, the second preset pixel group, or the third preset pixel group, thereby increasing the accuracy of the match and speeding up subsequent iterations. For example, there are 100 pixels at the position with a vertical coordinate of 3 in the second image, and every 4 adjacent pixels form a third preset pixel group, for a total of 25 third preset pixel groups. Then, for each pixel in the first image at the position with a vertical coordinate of 3, a pixel with the largest matching weight will be selected in each third preset pixel group for matching, that is, each pixel in the first image at the position with a vertical coordinate of 3 will establish a matching relationship with 25 pixels in the second image.

[0078] The construction formula of the first initial relationship group is:

[0079] ;

[0080] in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the small parallax range in the first initial relationship group, is the disparity range of the first initial relationship group, x is the horizontal coordinate of the pixel, and y is the vertical coordinate of the pixel.

[0081] The construction formula of the second initial relationship group is:

[0082] ;

[0083] in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the middle parallax range in the second initial relationship group, is the disparity range of the second initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the disparity range The serial number in is the matching weight of the i-th pixel in the second preset pixel group.

[0084] The construction formula of the third initial relationship group is:

[0085] ;

[0086] in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the large parallax range in the third initial relationship group, is the disparity range of the third initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the disparity range The serial number in is the matching weight of the i-th pixel in the third preset pixel group.

[0087] In step 203, the first initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a small parallax range; the second initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a medium parallax range; the first initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a large parallax range.

[0088] In this embodiment, the score smoothing process can be performed by a three-dimensional cost aggregation unit composed of a lightweight three-dimensional U-shaped network, which includes three down-sampling convolution blocks and three up-sampling convolution blocks; each down-sampling convolution block can include two 3 3 For 3D convolution with 3 kernels, the number of channels of the three downsampling convolution blocks can be 16, 32, and 48 respectively; each upsampling convolution block can include a 4 4 4-core 3D transposed convolution and two 3 3 3D convolution with 3 kernels. The 3D cost aggregation unit can integrate spatial clues of multi-scale matching relationships. The score smoothing process uses convolution to filter and smooth the matching relationships, propagating high-confidence matching relationships to the surrounding blurred areas, and finally obtaining a filtered or aggregated matching relationship group.

[0089] In this embodiment, the selective weight is obtained according to the image features in the first image and the matching relationship group, such as Figure 3 As shown, the corresponding method flow is as follows:

[0090] In step 301, the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a small parallax range is obtained and used as the first matching relationship, the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a medium parallax range is obtained and used as the second matching relationship, and the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a large parallax range is obtained and used as the third matching relationship.

[0091] In step 302, convolution processing is performed on the first matching relationship, the second matching relationship, and the third matching relationship to obtain a disparity feature set.

[0092] In this embodiment, the first matching relationship, the second matching relationship and the third matching relationship are: the matching relationships that are currently most likely to be correct for each pixel in the first image. These matching relationships are naturally most likely to correspond to the correct disparity range and can relatively accurately reflect the correct disparity range corresponding to each pixel in the image.

[0093] In this embodiment, convolution processing is performed on the first matching relationship, the second matching relationship, and the third matching relationship to obtain a disparity feature set. The corresponding formula is:

[0094] ;

[0095] in, is the disparity feature set, conv is the first convolutional layer processing, concat is the second convolutional layer processing, is the first matching relationship, For the second matching relationship, It is the third matching relationship.

[0096] In step 303, the disparity feature set and the image features of the first image are subjected to a secondary convolution process to obtain a small disparity weight under a small disparity range, a medium disparity weight under a medium disparity range, and a large disparity weight under a large disparity range, respectively. The weight corresponding to the disparity range is selected from the small disparity weight, the medium disparity weight, and the large disparity weight as the selective weight.

[0097] In this embodiment, the calculation formula for the selectivity weight is:

[0098] ;

[0099] in, is the feature of the first image, is the small disparity weight, is the median disparity weight, is the large disparity weight.

[0100] After obtaining the small parallax weight, medium parallax weight, and large parallax weight, the matching relationship groups under each parallax range can be screened according to the corresponding selective weights to obtain the effective matching relationships under each parallax range. The effective matching relationships under all parallax ranges are used as fusion features. The corresponding calculation formula is:

[0101] ;

[0102] in, is the feature of the first image, To fusion features, is the small disparity weight, is the matching relationship group under a small parallax range, is the median disparity weight, is the matching relationship group under the medium parallax range, is the large disparity weight, is the matching relationship group under a large parallax range, is the effective matching relationship under a small parallax range, is the effective matching relationship under the medium parallax range, This is an effective matching relationship under a large parallax range.

[0103] After obtaining the fusion features, the initial disparity map can be iterated multiple times using the fusion features. In each iteration, convolution processing is performed at different scales and resolutions to obtain the corresponding disparity update amount. The disparity map in each iteration is updated so that the disparity map gradually converges to be accurate, and a relatively accurate disparity map is obtained. The corresponding method flow is as follows:

[0104] First, a preset number of iterations is set, and the initial disparity map is input as the input disparity map into the iteration, such as Figure 4 As shown, each iteration includes:

[0105] In step 401, the input disparity map and the fusion feature are sampled to a first preset resolution and a first convolution process is performed to obtain a first hidden state feature.

[0106] In this embodiment, the first preset resolution may be a 1 / 16 resolution, and the first convolution processing may be performed by one convolution unit.

[0107] In step 402, the first hidden state feature is sampled to a second preset resolution and a second convolution process is performed to obtain a second hidden state feature at a third preset resolution.

[0108] In this embodiment, the second preset resolution may be 1 / 8 resolution, and the second convolution process may be performed by another convolution unit. The third preset resolution may be 1 / 4 resolution.

[0109] In this embodiment, the receptive range of the convolution can be increased by optimizing multiple resolutions of different sizes, making full use of non-local spatial cues, thereby improving the accuracy and efficiency of the convolution processing and the entire iterative process.

[0110] In step 403, it is determined whether the current number of iterations is greater than the preset number of iterations.

[0111] The preset number of iterations is set by those skilled in the art according to actual conditions. In this embodiment, the preset number of iterations may be 15 or more.

[0112] In step 404 , when the current number of iterations is greater than or equal to the preset number of iterations, the iterations are stopped, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map to obtain the final disparity map.

[0113] In step 405 , when the current iteration number is less than or equal to the preset iteration number, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map and input as a new input disparity map into the next iteration.

[0114] In this embodiment, the disparity update amount of this iteration can be obtained by using the second hidden state feature in each iteration. , the input disparity map of this round of iteration is , so the update expression of this round of iteration is as follows:

[0115] ;

[0116] in, is the disparity map optimized in the kth iteration.

[0117] Example 2:

[0118] Based on Example 1, this embodiment provides an efficient stereo matching system for large parallax scenes, which is used to execute the efficient stereo matching method for large parallax scenes described in Example 1, wherein: Figure 5 As shown, the efficient stereo matching system for large parallax scenes includes: an encoding volume construction module, a feature fusion module and an iterative optimization module, wherein:

[0119] The encoding body construction module, feature fusion module and iterative optimization module are connected in sequence. The first image and the second image are input into the feature extraction network, which extracts features from the first image and the second image and sends the extracted features to the encoding body construction module.

[0120] The encoding body construction module includes a three-way matching path, which is used to match and optimize the relationships between pixels in the first image and pixels in the second image according to a small parallax range, a medium parallax range, and a large parallax range, respectively, to obtain a matching relationship group for each pixel in the first image under the small parallax range, the medium parallax range, and the large parallax range, thereby constructing a matching relationship group for multiple parallax ranges.

[0121] In this embodiment, each matching path includes a matching unit, a three-dimensional cost aggregation unit, and a code volume generation unit.

[0122] The matching unit is used to efficiently and accurately match the relationship between pixels within a corresponding parallax range to obtain a first initial relationship group, a second initial relationship group, and a third initial relationship group.

[0123] The three-dimensional cost aggregation unit is used to perform score smoothing processing on the first initial relationship group, the second initial relationship group and the third initial relationship group, and propagate the matching relationship with high confidence to the surrounding fuzzy area.

[0124] The encoding body generation unit is used to take the filtered or aggregated matching relationships as a set to obtain a plurality of matching relationship groups of disparity ranges.

[0125] The feature fusion module is used to obtain the weights under each disparity range based on the feature content of the first image and the matching relationship group under each disparity range, and together serve as the selective weights. The selective weights are used to reflect the disparity ranges of different objects in the image. Therefore, the correct matching relationship in the disparity range can be screened out from the matching relationship group according to the selective weights as the effective matching relationship, and all effective matching relationships are collectively used as the fusion features.

[0126] The iterative optimization module is used to iterate the initial disparity map based on the fused features. Through an iterative search operation, the geometric features corresponding to the current disparity are indexed from the fused features. In each iteration, the indexed local geometric features are decomposed into three resolution scales and then sequentially input into the convolution unit for iterative optimization. After dozens of iterations, a nearly accurate disparity map is obtained. The initial disparity map is derived from the features of the matching relationship group within a small disparity range. Therefore, the iterative optimization module is also connected to the encoder generation unit on the matching path within the small disparity range to obtain the initial disparity map.

[0127] like Figure 6As shown, this is a schematic diagram of an efficient stereo matching system for large parallax scenes in a practical application scenario, wherein the left image can be regarded as the first image, the right image can be regarded as the second image, the feature network can be regarded as the feature extraction network, the adaptive block matching can be regarded as the matching module on the large parallax range matching path, the 3D CNN can be regarded as the three-dimensional cost aggregation unit, the latter module of the 3D CNN can be regarded as the encoding volume generation unit, and the selective geometric feature fusion can be regarded as the feature fusion module.

[0128] Example 3:

[0129] like Figure 7 , which is a schematic diagram of an efficient stereo matching apparatus for large parallax scenes according to an embodiment of the present invention. The efficient stereo matching apparatus for large parallax scenes according to this embodiment includes one or more processors 41 and a memory 42.

[0130] The processor 41 and the memory 42 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0131] Memory 42, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the efficient stereo matching method for large parallax scenes described in the above embodiment. Processor 41 executes the efficient stereo matching method for large parallax scenes by running the non-volatile software program and instructions stored in memory 42.

[0132] The memory 42 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 42 may optionally include a memory remotely located relative to the processor 41, and such remote memory may be connected to the processor 41 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The program instructions / modules are stored in the memory 42, and when executed by the one or more processors 41, the efficient stereo matching method for large parallax scenes in the above embodiment is executed, for example, the above described Figures 1-4 The steps shown.

[0134] An embodiment of the present invention further provides a computer storage medium having computer program instructions stored thereon; when the computer program instructions are executed by a processor, the efficient stereo matching method for large parallax scenes provided by the embodiment of the present invention is implemented.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An efficient stereo matching method for large parallax scenes, characterized by: include: Matching and optimizing the relationships between pixels in the first image and pixels in the second image according to a small parallax range, a medium parallax range, and a large parallax range, respectively, to obtain a matching relationship group for each pixel in the first image under the small parallax range, the medium parallax range, and the large parallax range; Dividing pixels in the same vertical coordinate direction in the second image into a plurality of first preset pixel groups according to a small parallax range, dividing pixels in the same vertical coordinate direction in the second image into a plurality of second preset pixel groups according to a medium parallax range, and dividing pixels in the same vertical coordinate direction in the second image into a plurality of third preset pixel groups according to a large parallax range; Matching each pixel in the first image with pixels in each first preset pixel group according to corresponding matching weights to obtain a first initial relationship group; Matching each pixel in the first image with pixels in each second preset pixel group according to corresponding matching weights to obtain a second initial relationship group; Matching each pixel in the first image with pixels in each third preset pixel group according to corresponding matching weights to obtain a third initial relationship group; Perform score smoothing processing on the first initial relationship group to obtain a matching relationship group for each pixel in the first image within a small disparity range; Performing score smoothing processing on the second initial relationship group to obtain a matching relationship group for each pixel in the first image in a medium disparity range; performing score smoothing processing on the first initial relationship group to obtain a matching relationship group for each pixel in the first image in a large disparity range; obtaining a selective weight based on the image features in the first image and the matching relationship group, screening the matching relationship group under each disparity range according to the selective weight, obtaining a valid matching relationship under each disparity range, and using the valid matching relationships under all the disparity ranges as fusion features; Obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a small parallax range and using it as the first matching relationship, obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a medium parallax range and using it as the second matching relationship, and obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a large parallax range and using it as the third matching relationship; Performing convolution processing on the first matching relationship, the second matching relationship, and the third matching relationship to obtain a disparity feature set; performing a secondary convolution process on the disparity feature set and the image features of the first image to obtain a small disparity weight under a small disparity range, a medium disparity weight under a medium disparity range, and a large disparity weight under a large disparity range, respectively; and selecting a weight corresponding to the disparity range from the small disparity weight, the medium disparity weight, and the large disparity weight as the selective weight; An initial disparity map is obtained according to the matching relationship group in a small disparity range, and the initial disparity map is iteratively optimized according to the fusion feature to obtain a final disparity map.

2. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: Each pixel in the first image is matched with each pixel in each first preset pixel group according to the corresponding matching weight to obtain a first initial relationship group. The specific construction formula is: ； in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the small parallax range in the first initial relationship group, is the disparity range of the first initial relationship group, x is the horizontal coordinate of the pixel, and y is the vertical coordinate of the pixel.

3. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: Each pixel in the first image is matched with each pixel in each second preset pixel group according to the corresponding matching weight to obtain a second initial relationship group. The specific construction formula is: ； in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the middle parallax range in the second initial relationship group, is the disparity range of the second initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the disparity range The serial number in is the matching weight of the i-th pixel in the second preset pixel group.

4. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: Each pixel in the first image is matched with each pixel in each third preset pixel group according to the corresponding matching weight to obtain a third initial relationship group. The specific construction formula is: ； in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the large parallax range in the third initial relationship group, is the disparity range of the third initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the disparity range The serial number in is the matching weight of the i-th pixel in the third preset pixel group.

5. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: The matching relationship groups under each parallax range are screened according to the selective weight to obtain the valid matching relationship under each parallax range, and the valid matching relationships under all parallax ranges are used as fusion features. The specific formula is: in, To fusion features, is the small disparity weight, is the matching relationship group under a small parallax range, is the median disparity weight, is the matching relationship group under the medium parallax range, is the large disparity weight, is the matching relationship group under a large parallax range, is the effective matching relationship under a small parallax range, is the effective matching relationship under the medium parallax range, This is an effective matching relationship under a large parallax range.

6. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: The step of obtaining an initial disparity map according to the matching relationship group in a small disparity range and iteratively optimizing the initial disparity map according to the fusion feature to obtain a final disparity map specifically includes: A preset number of iterations is set, and the initial disparity map is input as an input disparity map into the iteration. Each iteration includes: Sampling the input disparity map and the fused features to a first preset resolution and performing a first convolution process to obtain a first hidden state feature; Sampling the first hidden state feature to a second preset resolution and performing a second convolution process to obtain a second hidden state feature at a third preset resolution; Determine whether the current number of iterations is greater than the preset number of iterations; When the current iteration number is greater than or equal to the preset iteration number, the iteration is stopped, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map to obtain the final disparity map; When the current iteration number is less than or equal to the preset iteration number, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map and input into the next iteration as a new input disparity map.

7. An efficient stereo matching device for large parallax scenes, characterized by: The invention comprises at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the efficient stereo matching method for large parallax scenes according to any one of claims 1 to 6.

8. A non-volatile computer storage medium, characterized in that The computer storage medium stores computer program instructions, which, when executed by one or more processors, implement the efficient stereo matching method for large parallax scenes according to any one of claims 1 to 6.

Images