High-efficiency stereo matching method and device for large parallax scene

Abstract

The invention provides an efficient stereo matching method and device for a large parallax scene, and the method comprises the steps: carrying out the feature extraction of a first image and a second image, carrying out the relation matching of pixels in the first image and pixels in the second image according to a small parallax range, a middle parallax range and a large parallax range, and optimizing the relation. Obtaining a matching relation group under each parallax range, obtaining a selective weight according to the first image and the matching relation group, screening the matching relation group under each parallax range according to the selective weight, and obtaining an effective matching relation under each parallax range as a fusion feature; performing multi-round iterative optimization on the initial disparity map through the fusion features to obtain a final disparity map; through matching of pixels in the image in different parallax ranges, a matching relation group in a plurality of parallax ranges is constructed, calculation prediction of a small parallax range, a middle parallax range and a large parallax range is considered at the same time, and accurate prediction of the parallax image is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an efficient stereo matching method and device for large parallax scenes. Background Art

[0002] Stereo matching is important because it can infer the geometry of a 3D scene from multiple captured images, with applications ranging from 3D reconstruction to robotics and autonomous driving. The key to stereo matching is to find the corresponding pixel positions in the left and right images taken by two cameras at the same ordinate height. The difference between these corresponding pixel positions is called disparity, which can be used to infer depth and reconstruct the 3D scene.

[0003] Common stereo matching usually involves small parallax scenes, medium parallax scenes and large parallax scenes, among which small parallax scenes refer to pixel matching scenes for the same distant view position in the left and right images, medium parallax scenes refer to pixel matching scenes for the same middle view position in the left and right images, and large parallax scenes refer to pixel matching scenes for the same near view position in the left and right images; in small parallax scenes, since the objects are all distant views, a single geometric structure in the distant view usually occupies fewer pixels in the image, and the objects in medium parallax scenes are all middle views, a single geometric structure in the middle view usually occupies fewer pixels in the image, so the pixels of the parallax range required for small parallax scenes and medium parallax scenes are usually not large; while the objects in large parallax scenes are all near views, a single geometric structure in the near view occupies more pixels in the image, and more pixels of the parallax range are required. The parallax range defined by the existing stereo matching method is difficult to take into account large parallax scenes. Therefore, in the case of large parallax scenes, it is difficult to accurately perform stereo matching between pixels and accurately predict the disparity map.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to achieve accurate prediction of a disparity map while taking into account small disparity scenes, medium disparity scenes and large disparity scenes.

[0006] The present invention adopts the following technical solution: In a first aspect, an efficient stereo matching method for large parallax scenes is provided, comprising: Relationship matching and optimization are performed on pixels in the first image and pixels in the second image according to a small parallax range, a medium parallax range, and a large parallax range, respectively, to obtain a matching relationship group for each pixel in the first image under the small parallax range, the medium parallax range, and the large parallax range; Acquire a selective weight according to the image features in the first image and the matching relationship group, screen the matching relationship group under each parallax range according to the selective weight, obtain a valid matching relationship under each parallax range, and use the valid matching relationships under all parallax ranges as fusion features; An initial disparity map is obtained according to the matching relationship group under a small disparity range, and the initial disparity map is iteratively optimized according to the fusion feature to obtain a final disparity map.

[0007] Preferably, matching and optimizing the relationship between pixels in the first image and pixels in the second image according to the small parallax range, the medium parallax range and the large parallax range respectively specifically includes: The pixels in the same vertical coordinate direction in the second image are divided into a plurality of first preset pixel groups according to a small parallax range, the pixels in the same vertical coordinate direction in the second image are divided into a plurality of second preset pixel groups according to a medium parallax range, and the pixels in the same vertical coordinate direction in the second image are divided into a plurality of third preset pixel groups according to a large parallax range; Matching each pixel in the first image with each pixel in each first preset pixel group according to corresponding matching weights to obtain a first initial relationship group; matching each pixel in the first image with each pixel in each second preset pixel group according to corresponding matching weights to obtain a second initial relationship group; matching each pixel in the first image with each pixel in each third preset pixel group according to corresponding matching weights to obtain a third initial relationship group; The first initial relationship group is subjected to score smoothing processing to obtain a matching relationship group for each pixel in the first image under a small parallax range; the second initial relationship group is subjected to score smoothing processing to obtain a matching relationship group for each pixel in the first image under a medium parallax range; the first initial relationship group is subjected to score smoothing processing to obtain a matching relationship group for each pixel in the first image under a large parallax range.

[0008] Preferably, each pixel in the first image is matched with each pixel in each first preset pixel group according to a corresponding matching weight to obtain a first initial relationship group, and the specific construction formula is: ; in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the small parallax range in the first initial relationship group, is the disparity range of the first initial relationship group, x is the horizontal coordinate of the pixel, and y is the vertical coordinate of the pixel.

[0009] Preferably, each pixel in the first image is matched with each pixel in each second preset pixel group according to a corresponding matching weight to obtain a second initial relationship group, and the specific construction formula is: ; in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the middle parallax range in the second initial relationship group, is the parallax range of the second initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the parallax range The serial number in is the matching weight of the i-th pixel in the second preset pixel group.

[0010] Preferably, each pixel in the first image is matched with each pixel in each third preset pixel group according to a corresponding matching weight to obtain a third initial relationship group, and the specific construction formula is: ; in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the large parallax range in the third initial relationship group, is the parallax range of the third initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the parallax range The serial number in is the matching weight of the i-th pixel in the third preset pixel group.

[0011] Preferably, the obtaining of the selectivity weight according to the image features in the first image and the matching relationship group specifically includes: Obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a small parallax range and using it as the first matching relationship, obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a medium parallax range and using it as the second matching relationship, and obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a large parallax range and using it as the third matching relationship; Performing convolution processing on the first matching relationship, the second matching relationship, and the third matching relationship to obtain a disparity feature set; The disparity feature set and the image features of the first image are subjected to secondary convolution processing to obtain a small disparity weight under a small disparity range, a medium disparity weight under a medium disparity range, and a large disparity weight under a large disparity range, respectively. The weight corresponding to the disparity range is selected from the small disparity weight, the medium disparity weight and the large disparity weight as the selective weight.

[0012] Preferably, the matching relationship groups under each parallax range are screened according to the selective weight to obtain the valid matching relationship under each parallax range, and the valid matching relationships under all parallax ranges are used as fusion features. The specific formula is:

[0013] in, To fusion features, is the small disparity weight, is the matching relationship group under a small parallax range, is the median disparity weight, is the matching relationship group under the medium parallax range, is the large disparity weight, is the matching relationship group under a large parallax range, is the effective matching relationship under a small parallax range, is the effective matching relationship under the medium parallax range, It is an effective matching relationship under a large parallax range.

[0014] Preferably, obtaining an initial disparity map according to the matching relationship group in a small disparity range, and iteratively optimizing the initial disparity map according to the fusion feature to obtain a final disparity map specifically includes: A preset number of iterations is set, and the initial disparity map is input as an input disparity map into the iteration. Each round of iteration includes: Sampling the input disparity map and the fusion feature to a first preset resolution and performing a first convolution process to obtain a first hidden state feature; Sampling the first hidden state feature to a second preset resolution and performing a second convolution process to obtain a second hidden state feature at a third preset resolution; Determine whether the current number of iterations is greater than the preset number of iterations; When the current number of iterations is greater than or equal to the preset number of iterations, the iteration is stopped, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map to obtain the final disparity map; When the current iteration number is less than or equal to the preset iteration number, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map and input into the next iteration as a new input disparity map.

[0015] In a second aspect, an efficient stereo matching device for large parallax scenes is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the efficient stereo matching method for large parallax scenes.

[0016] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0017] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0018] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method of the first aspect.

[0019] In a sixth aspect, an efficient stereo matching system for large parallax scenes is provided, including an efficient stereo matching device for large parallax scenes as in the second aspect, and using an efficient stereo matching method for large parallax scenes as in the first aspect.

[0020] The present invention provides an efficient stereo matching method and device for large parallax scenes, which extracts features from a first image and a second image, matches and optimizes the relationship between pixels in the first image and pixels in the second image according to a small parallax range, a medium parallax range, and a large parallax range, respectively, to obtain a matching relationship group under each parallax range, and then obtains a selective weight according to the first image and the matching relationship group, and screens the matching relationship group under each parallax range according to the selective weight, to obtain an effective matching relationship under each parallax range as a fusion feature; the initial disparity map is optimized for multiple rounds of iterations through the fusion feature to obtain a final disparity map. By matching pixels in the image under different disparity ranges, matching relationship groups under multiple disparity ranges are constructed, while taking into account the calculation prediction of the small disparity range, the medium disparity range, and the large disparity range, to achieve accurate prediction of the disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a method flow chart of an efficient stereo matching method for large parallax scenes provided by an embodiment of the present invention; Figure 2 It is a flow chart of a method for obtaining a matching relationship group in each parallax range in an efficient stereo matching method for a large parallax scene provided by an embodiment of the present invention; Figure 3 It is a flow chart of a method for obtaining selective weights in an efficient stereo matching method for large parallax scenes provided by an embodiment of the present invention; Figure 4 It is a flow chart of a method for iteratively optimizing an initial disparity map in an efficient stereo matching method for a large disparity scene provided by an embodiment of the present invention; Figure 5 is a system schematic diagram of an efficient stereo matching system for large parallax scenes provided by an embodiment of the present invention; Figure 6 is a system schematic diagram of another efficient stereo matching system for large parallax scenes provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of an efficient stereo matching device for large parallax scenes provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] In the description of the present invention, it is necessary to understand that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0025] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0026] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0027] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0028] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] Embodiment 1: This embodiment provides an efficient stereo matching method for large parallax scenes. Figure 1 As shown, the method flow includes: In step 101, the pixels in the first image and the pixels in the second image are matched and optimized according to the small parallax range, the medium parallax range and the large parallax range, respectively, to obtain a matching relationship group for each pixel in the first image under the small parallax range, the medium parallax range and the large parallax range.

[0030] In this embodiment, the first image and the second image can be obtained by shooting the same target by two cameras located at the same ordinate height. Since the positions of the two cameras on the abscissa are different, the first image and the second image have different shooting angles, so there is parallax. In this embodiment, in order to infer the actual three-dimensional scene structure based on the first image and the second image, it is necessary to match the pixels of the same object in the first image and the second image, so as to obtain an accurate three-dimensional scene structure. In this embodiment, the first image and the second image can be first input into a feature extraction network, and the corresponding features in the first image and the second image can be extracted by the feature extraction network for subsequent calculation of the matching relationship between the pixels of the two images.

[0031] The small parallax in the image refers to the parallax of the distant view captured by the two cameras. Since the distance of the object is inversely proportional to the parallax, the parallax of the distant view object in the two images is often small. In this embodiment, matching the pixels in the first image with the pixels in the second image in the small parallax range means matching all pixels in the first image with pixels in the second image; the matching method can be: matching a single pixel in the first image with each pixel of the same vertical coordinate in the second image; the set of all matching relationships in the small parallax range is the matching relationship group in the small parallax range.

[0032] The medium disparity in the image refers to the disparity of the middle view captured by the two cameras. Since the distance of the object is inversely proportional to the disparity, the disparity of the object in the middle view in the two images is larger than that of the distant view. Therefore, when matching the pixels in the first image with the pixels in the second image in the medium disparity range, each certain number of adjacent pixels on the same ordinate in the second image can be regarded as a group (for example, each two adjacent pixels on the same ordinate are a group). When actually matching, each pixel in the first image has a corresponding matching weight for each pixel in the second image. The matching weight is obtained by adaptive learning of the matching model during the training process. Each pixel in the first image will select a pixel with the largest matching weight from each group of adjacent pixels in the second image for matching. The set of all matching relationships under the medium disparity range is the matching relationship group under the medium disparity range.

[0033] The large parallax in the image refers to the parallax of the close-up portion captured by the two cameras. Since the distance of the object being photographed is inversely proportional to the parallax, the parallax of the close-up object in the two images is larger than that of the distant and mid-view objects. When matching the pixels in the first image with the pixels in the second image in a large parallax range, it is necessary to group every certain number of adjacent pixels on the same ordinate in the second image (for example, every four adjacent pixels on the same ordinate are grouped together). Similarly, when actually matching, each pixel in the second image has a corresponding matching weight for each pixel in the second image. The matching weight is adaptively learned by the matching model during the training process. Each pixel in the first image will select a pixel with the largest matching weight from each group of adjacent pixels in the second image for matching. The set of all matching relationships under a large parallax range is the matching relationship group under a large parallax range.

[0034] In summary, all pixels in the first image will be matched with multiple pixels in the second image in a small parallax range, a medium parallax range, and a large parallax range, that is, all pixels in the first image will undergo three rounds of matching.

[0035] However, in the actual three-dimensional structure that we need to construct in the end, each pixel in the first image should have a correctly corresponding pixel point in the second image. For example, the same bicycle is photographed at a close point in both the first image and the second image. The upper end of the front pedal of the bicycle is correctly matched with the pixel points in the first image and the pixel points in the second image, while the upper end of the front pedal of the bicycle is incorrectly matched with the pixel points in the first image and the pixel points in other positions in the second image. Therefore, the matching model in this embodiment has a corresponding score for each matching relationship of a single pixel in the first image, which is used to judge whether the matching relationship is more correct, so that the matching relationship of each pixel is judged and screened according to the score in the subsequent steps.

[0036] However, due to the influence of the external environment in the actual shooting scene or the errors caused by blurry and unclear shooting, the scores of some matching relationships may be too high or too low. Therefore, it is necessary to optimize the scores of all matching relationships so that the score change trend of all matching relationships tends to be smooth, avoiding the scores of some matching relationships being too prominent and affecting the correctness of the correspondence between pixel points.

[0037] In step 102, a selective weight is obtained according to the image features in the first image and the matching relationship group, and the matching relationship group under each disparity range is screened according to the selective weight to obtain a valid matching relationship under each disparity range, and the valid matching relationships under all disparity ranges are used as fusion features.

[0038] In the present embodiment, since the matching of the pixels between the first image and the second image is performed in three rounds under different parallax ranges (i.e., one round of relationship matching is performed on all the pixels in the first image according to the small parallax range, one round of relationship matching is performed on all the pixels in the first image according to the medium parallax range, and one round of relationship matching is performed on all the pixels in the first image according to the range), but the parallax ranges of the pixels at different positions in the image are different, the correct matching between the pixels needs to be performed in the corresponding parallax range. For example, the relationship matching between the pixels in the large parallax part of the first image and the second image in the large parallax range belongs to the valid matching relationship, while the relationship matching between the pixels in the large parallax part of the first image and the second image in the medium parallax range or the small parallax range does not belong to the valid matching relationship. In the present embodiment, all the valid matching relationships need to be screened out in the matching relationship group.

[0039] In this embodiment, the selective weight is used to reflect the disparity range of different objects in the image. Therefore, the matching relationship with the correct disparity range can be screened out from the matching relationship group according to the selective weight as the effective matching relationship, and all the effective matching relationships are collectively used as the fusion feature.

[0040] In step 103, an initial disparity map is obtained according to the matching relationship group in a small disparity range, and the initial disparity map is iteratively optimized according to the fusion feature to obtain a final disparity map.

[0041] In this embodiment, since most of the pixels in the captured image usually belong to a small disparity range when the camera is shooting, in order to quickly obtain the most initial disparity map required for iteration, the features obtained according to the matching relationship group under the small disparity range are used as the initial disparity map for input into subsequent iterations. In this embodiment, although the matching relationship between each pixel in the fusion feature is selected according to the score of the matching model, there is still a corresponding error in the score of the matching model. Therefore, in order to ensure the accuracy of the disparity map finally obtained, it is necessary to perform convolution processing on the initial disparity map and the fusion feature at different scales, and perform multiple rounds of iterative operations to make the disparity map converge and obtain a relatively accurate final disparity map.

[0042] In this embodiment, the pixels in the first image and the pixels in the second image are matched and optimized according to the small parallax range, the medium parallax range and the large parallax range, respectively. Figure 2 As shown, the corresponding method flow is as follows: In step 201, the pixels in the same vertical coordinate direction in the second image are divided into multiple first preset pixel groups according to a small parallax range, the pixels in the same vertical coordinate direction in the second image are divided into multiple second preset pixel groups according to a medium parallax range, and the pixels in the same vertical coordinate direction in the second image are divided into multiple third preset pixel groups according to a large parallax range.

[0043] In this embodiment, since the small parallax range corresponds to the distant view portion of the image, the object structure of this portion occupies the least number of pixels in the image, so each adjacent pixel in the second image can be used as a first preset pixel group. Since the medium parallax range corresponds to the mid-view portion of the image, the object structure of this portion occupies a relatively larger number of pixels in the image, so each adjacent two pixels in the second image can be used as a second preset pixel group. Since the large parallax range corresponds to the near view portion of the image, the object structure of this portion occupies the largest number of pixels in the image, so each adjacent four pixels in the second image can be used as a third preset pixel group.

[0044] In step 202, each pixel in the first image is matched with each pixel in each first preset pixel group according to the corresponding matching weight to obtain a first initial relationship group; each pixel in the first image is matched with each pixel in each second preset pixel group according to the corresponding matching weight to obtain a second initial relationship group; each pixel in the first image is matched with each pixel in each third preset pixel group according to the corresponding matching weight to obtain a third initial relationship group.

[0045] It should be noted that when each pixel in the first image is matched with the first preset pixel group, the second preset pixel group or the third preset pixel group, it will be matched with the pixel with the largest matching weight in each first preset pixel group, the second preset pixel group or the third preset pixel group, thereby increasing the accuracy of the match and speeding up subsequent iterations. For example, there are 100 pixels at the position of the ordinate 3 of the second image, and every 4 adjacent pixels form a third preset pixel group, for a total of 25 third preset pixel groups. Then, for each pixel of the first image at the position of the ordinate 3, a pixel with the largest matching weight will be selected in each third preset pixel group for matching, that is, each pixel of the first image at the position of the ordinate 3 will establish a matching relationship with 25 pixels in the second image.

[0046] The construction formula of the first initial relationship group is: ; in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the small parallax range in the first initial relationship group, is the disparity range of the first initial relationship group, x is the horizontal coordinate of the pixel, and y is the vertical coordinate of the pixel.

[0047] The construction formula of the second initial relationship group is: ; in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the middle parallax range in the second initial relationship group, is the parallax range of the second initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the parallax range The serial number in is the matching weight of the i-th pixel in the second preset pixel group.

[0048] The construction formula of the third initial relationship group is: ; in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the large parallax range in the third initial relationship group, is the parallax range of the third initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the parallax range The serial number in is the matching weight of the i-th pixel in the third preset pixel group.

[0049] In step 203, the first initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a small parallax range; the second initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a medium parallax range; the first initial relationship group is score-smoothed to obtain a matching relationship group for each pixel in the first image under a large parallax range.

[0050] In this embodiment, the score smoothing process can be performed by a three-dimensional cost aggregation unit composed of a lightweight three-dimensional U-shaped network, and the lightweight three-dimensional U-shaped network includes three down-sampling convolution blocks and three up-sampling convolution blocks; each down-sampling convolution block can include two 3 3 For a 3D convolution with 3 kernels, the number of channels of the three downsampling convolution blocks can be 16, 32, and 48 respectively; each upsampling convolution block can include a 4 4 4-core 3D transposed convolution and two 3 3 3D convolution with 3 kernels. The 3D cost aggregation unit can integrate spatial clues of multi-scale matching relationships. The score smoothing process uses convolution to filter and smooth the matching relationships, and propagate high-confidence matching relationships to the surrounding fuzzy areas, and finally obtain a filtered or aggregated matching relationship group.

[0051] In this embodiment, the selective weight is obtained according to the image features in the first image and the matching relationship group, such as Figure 3 As shown, the corresponding method flow is as follows: In step 301, the matching relationship with the highest score in the matching relationship group under a small parallax range for each pixel in the first image is obtained and used as the first matching relationship, the matching relationship with the highest score in the matching relationship group under a medium parallax range for each pixel in the first image is obtained and used as the second matching relationship, and the matching relationship with the highest score in the matching relationship group under a large parallax range for each pixel in the first image is obtained and used as the third matching relationship.

[0052] In step 302, the first matching relationship, the second matching relationship and the third matching relationship are convolved to obtain a disparity feature set.

[0053] In this embodiment, the first matching relationship, the second matching relationship and the third matching relationship are: the most likely correct matching relationships for each pixel in the first image. These matching relationships are naturally most likely to correspond to the correct disparity range, and can relatively correctly reflect the correct disparity range corresponding to each pixel in the image.

[0054] In this embodiment, the first matching relationship, the second matching relationship and the third matching relationship are convoluted to obtain a disparity feature set, and the corresponding formula is: ; in, is the collection of disparity features, conv is the first convolutional layer processing, and concat is the second convolutional layer processing. is the first matching relationship, is the second matching relationship, It is the third matching relationship.

[0055] In step 303, the disparity feature set and the image features of the first image are subjected to secondary convolution processing to obtain small disparity weights under a small disparity range, medium disparity weights under a medium disparity range, and large disparity weights under a large disparity range, respectively. The weight corresponding to the disparity range is selected from the small disparity weight, the medium disparity weight, and the large disparity weight as the selective weight.

[0056] In this embodiment, the calculation formula of the selectivity weight is: ; in, is the feature of the first image, is the small disparity weight, is the median disparity weight, is the large disparity weight.

[0057] After obtaining the small parallax weight, medium parallax weight, and large parallax weight, the matching relationship group under each parallax range can be screened according to the corresponding selective weight to obtain the effective matching relationship under each parallax range. The effective matching relationship under all parallax ranges is used as the fusion feature. The corresponding calculation formula is: ; in, is the feature of the first image, To fusion features, is the small disparity weight, is the matching relationship group under a small parallax range, is the median disparity weight, is the matching relationship group under the medium parallax range, is the large disparity weight, is the matching relationship group under a large parallax range, is the effective matching relationship under a small parallax range, is the effective matching relationship under the medium parallax range, It is an effective matching relationship under a large parallax range.

[0058] After obtaining the fusion features, the initial disparity map can be iterated multiple times through the fusion features. In each iteration, convolution processing is performed at different size resolutions to obtain the corresponding disparity update amount, and the disparity map in each iteration is updated so that the disparity map gradually converges to be accurate, and a relatively accurate disparity map is obtained. The corresponding method flow is as follows: First, a preset number of iterations is set, and the initial disparity map is input as an input disparity map into the iteration, such as Figure 4 As shown, each round of iteration includes: In step 401, the input disparity map and the fusion feature are sampled to a first preset resolution and a first convolution process is performed to obtain a first hidden state feature.

[0059] In this embodiment, the first preset resolution may be a 1 / 16 resolution, and the first convolution processing may be performed by one convolution unit.

[0060] In step 402, the first hidden state feature is sampled to a second preset resolution and a second convolution process is performed to obtain a second hidden state feature at a third preset resolution.

[0061] In this embodiment, the second preset resolution may be 1 / 8 resolution, and the second convolution process may be performed by another convolution unit. The third preset resolution may be 1 / 4 resolution.

[0062] In this embodiment, the receptive range of the convolution can be increased through optimization methods of multiple different size resolutions, and non-local spatial cues can be fully utilized, thereby improving the accuracy and efficiency of the convolution processing and the entire iterative process.

[0063] In step 403, it is determined whether the current number of iterations is greater than the preset number of iterations.

[0064] The preset number of iterations is set by those skilled in the art according to actual conditions. In this embodiment, the preset number of iterations may be more than 15 times.

[0065] In step 404, when the current number of iterations is greater than or equal to the preset number of iterations, the iteration is stopped, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map to obtain the final disparity map.

[0066] In step 405, when the current iteration number is less than or equal to the preset iteration number, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map and input into the next iteration as a new input disparity map.

[0067] In this embodiment, in each round of iteration, the disparity update amount of the current round of iteration can be obtained through the second hidden state feature. , the input disparity map of this round of iteration is , so the update expression for this round of iteration is as follows: ; in, is the optimized disparity map in the kth iteration.

[0068] Embodiment 2: Based on Example 1, this embodiment provides an efficient stereo matching system for large parallax scenes, which is used to execute the efficient stereo matching method for large parallax scenes described in Example 1, wherein: Figure 5 As shown, the efficient stereo matching system for large parallax scenes includes: a coding volume construction module, a feature fusion module and an iterative optimization module, wherein: The encoding body construction module, feature fusion module and iterative optimization module are connected in sequence. The first image and the second image are input into a feature extraction network, which extracts features from the first image and the second image and sends the extracted features to the encoding body construction module.

[0069] The encoding body construction module includes three matching paths, which are used to match and optimize the pixels in the first image and the pixels in the second image according to the small parallax range, the medium parallax range and the large parallax range, respectively, to obtain the matching relationship group of each pixel in the first image under the small parallax range, the medium parallax range and the large parallax range, thereby constructing a matching relationship group of multiple parallax ranges.

[0070] In this embodiment, each matching path includes a matching unit, a three-dimensional cost aggregation unit and a code body generation unit.

[0071] The matching unit is used to efficiently and accurately match the relationship between pixels within a corresponding parallax range, and obtain a first initial relationship group, a second initial relationship group, and a third initial relationship group.

[0072] The three-dimensional cost aggregation unit is used to perform score smoothing processing on the first initial relationship group, the second initial relationship group and the third initial relationship group, and propagate the matching relationship with high confidence to the surrounding fuzzy area.

[0073] The encoding body generation unit is used to take the filtered or aggregated matching relationships as a set to obtain a plurality of matching relationship groups of parallax ranges.

[0074] The feature fusion module is used to obtain weights under each disparity range based on the feature content of the first image and the matching relationship group under each disparity range, and together use them as the selective weights. The selective weights are used to reflect the disparity ranges of different objects in the image. Therefore, the correct matching relationship of the disparity range can be screened out from the matching relationship group according to the selective weights as the effective matching relationship, and all the effective matching relationships are collectively used as the fusion features.

[0075] The iterative optimization module is used to iterate the initial disparity map according to the fusion features, and iteratively indexes the geometric features corresponding to the current disparity from the fusion features through the search operation. The local geometric features indexed in each iteration are decomposed into three resolution scales, and then sequentially input into the convolution unit for iterative optimization. After dozens of iterations, a disparity map that converges to an accurate level is obtained. The initial disparity map is obtained by the features of the matching relationship group under a small disparity range, so the iterative optimization module is also connected to the encoding body generation unit on the matching path of the small disparity range to obtain the initial disparity map.

[0076] like Figure 6 As shown, it is a schematic diagram of an efficient stereo matching system for large parallax scenes in actual application scenarios, wherein the left image can be regarded as the first image, the right image can be regarded as the second image, the feature network can be regarded as the feature extraction network, the adaptive block matching can be regarded as the matching module on the large parallax range matching path, the 3D CNN can be regarded as the three-dimensional cost aggregation unit, the latter module of the 3D CNN can be regarded as the encoding volume generation unit, and the selective geometric feature fusion can be regarded as the feature fusion module.

[0077] Embodiment 3: like Figure 7 , which is a schematic diagram of an efficient stereo matching device for large parallax scenes according to an embodiment of the present invention. The efficient stereo matching device for large parallax scenes according to this embodiment includes one or more processors 41 and a memory 42 .

[0078] The processor 41 and the memory 42 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0079] The memory 42 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the efficient stereo matching method for large parallax scenes in the above embodiment. The processor 41 executes the efficient stereo matching method for large parallax scenes by running the non-volatile software programs and instructions stored in the memory 42.

[0080] The memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 42 may optionally include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the processor 41 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The program instructions / modules are stored in the memory 42, and when executed by the one or more processors 41, the efficient stereo matching method for large parallax scenes in the above embodiment is executed, for example, the above described Figure 1-Figure 4 The steps shown.

[0082] The embodiment of the present invention further provides a computer storage medium on which computer program instructions are stored; when the computer program instructions are executed by a processor, the efficient stereo matching method for large parallax scenes provided by the embodiment of the present invention is implemented.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient stereo matching method for large parallax scenes, characterized in that: include: Relationship matching and optimization are performed on pixels in the first image and pixels in the second image according to a small parallax range, a medium parallax range, and a large parallax range, respectively, to obtain a matching relationship group for each pixel in the first image under the small parallax range, the medium parallax range, and the large parallax range; Acquire a selective weight according to the image features in the first image and the matching relationship group, screen the matching relationship group under each parallax range according to the selective weight, obtain a valid matching relationship under each parallax range, and use the valid matching relationships under all parallax ranges as fusion features; An initial disparity map is obtained according to the matching relationship group under a small disparity range, and the initial disparity map is iteratively optimized according to the fusion feature to obtain a final disparity map.

2. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: The matching and optimizing the relationship between pixels in the first image and pixels in the second image according to the small parallax range, the medium parallax range and the large parallax range respectively specifically includes: The pixels in the same vertical coordinate direction in the second image are divided into a plurality of first preset pixel groups according to a small parallax range, the pixels in the same vertical coordinate direction in the second image are divided into a plurality of second preset pixel groups according to a medium parallax range, and the pixels in the same vertical coordinate direction in the second image are divided into a plurality of third preset pixel groups according to a large parallax range; Matching each pixel in the first image with each pixel in each first preset pixel group according to corresponding matching weights to obtain a first initial relationship group; matching each pixel in the first image with each pixel in each second preset pixel group according to corresponding matching weights to obtain a second initial relationship group; matching each pixel in the first image with each pixel in each third preset pixel group according to corresponding matching weights to obtain a third initial relationship group; The first initial relationship group is subjected to score smoothing processing to obtain a matching relationship group for each pixel in the first image under a small parallax range; the second initial relationship group is subjected to score smoothing processing to obtain a matching relationship group for each pixel in the first image under a medium parallax range; the first initial relationship group is subjected to score smoothing processing to obtain a matching relationship group for each pixel in the first image under a large parallax range.

3. The efficient stereo matching method for large parallax scenes according to claim 2, characterized in that: Each pixel in the first image is matched with each pixel in each first preset pixel group according to the corresponding matching weight to obtain a first initial relationship group. The specific construction formula is: ； in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the small parallax range in the first initial relationship group, is the disparity range of the first initial relationship group, x is the horizontal coordinate of the pixel, and y is the vertical coordinate of the pixel.

4. The efficient stereo matching method for large parallax scenes according to claim 2, characterized in that: Each pixel in the first image is matched with each pixel in each second preset pixel group according to the corresponding matching weight to obtain a second initial relationship group. The specific construction formula is: ； in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the middle parallax range in the second initial relationship group, is the parallax range of the second initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the parallax range The serial number in is the matching weight of the i-th pixel in the second preset pixel group.

5. The efficient stereo matching method for large parallax scenes according to claim 2, characterized in that: Each pixel in the first image is matched with each pixel in each third preset pixel group according to the corresponding matching weight to obtain a third initial relationship group. The specific construction formula is: ； in, is the image feature of the first image, is the image feature of the second image, is the matching relationship of the large parallax range in the third initial relationship group, is the parallax range of the third initial relationship group, x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and i is the parallax range The serial number in is the matching weight of the i-th pixel in the third preset pixel group.

6. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: The obtaining of the selective weight according to the image features in the first image and the matching relationship group specifically includes: Obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a small parallax range and using it as the first matching relationship, obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a medium parallax range and using it as the second matching relationship, and obtaining the matching relationship with the highest score in the matching relationship group for each pixel in the first image under a large parallax range and using it as the third matching relationship; Performing convolution processing on the first matching relationship, the second matching relationship, and the third matching relationship to obtain a disparity feature set; The disparity feature set and the image features of the first image are subjected to secondary convolution processing to obtain a small disparity weight under a small disparity range, a medium disparity weight under a medium disparity range, and a large disparity weight under a large disparity range, respectively. The weight corresponding to the disparity range is selected from the small disparity weight, the medium disparity weight and the large disparity weight as the selective weight.

7. The efficient stereo matching method for large parallax scenes according to claim 6, characterized in that: The matching relationship groups under each parallax range are screened according to the selective weight to obtain the effective matching relationship under each parallax range, and the effective matching relationships under all parallax ranges are used as fusion features. The specific formula is: in, To fusion features, is the small disparity weight, is the matching relationship group under a small parallax range, is the median disparity weight, is the matching relationship group under the medium parallax range, is the large disparity weight, is the matching relationship group under a large parallax range, is the effective matching relationship under a small parallax range, is the effective matching relationship under the medium parallax range, It is an effective matching relationship under a large parallax range.

8. The efficient stereo matching method for large parallax scenes according to claim 1, characterized in that: The step of obtaining an initial disparity map according to the matching relationship group in a small disparity range, and iteratively optimizing the initial disparity map according to the fusion feature to obtain a final disparity map specifically includes: A preset number of iterations is set, and the initial disparity map is input as an input disparity map into the iteration. Each round of iteration includes: Sampling the input disparity map and the fusion feature to a first preset resolution and performing a first convolution process to obtain a first hidden state feature; Sampling the first hidden state feature to a second preset resolution and performing a second convolution process to obtain a second hidden state feature at a third preset resolution; Determine whether the current number of iterations is greater than the preset number of iterations; When the current number of iterations is greater than or equal to the preset number of iterations, the iteration is stopped, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map to obtain the final disparity map; When the current iteration number is less than or equal to the preset iteration number, a disparity update amount is obtained according to the second hidden state feature, and the disparity update amount is added to the input disparity map and input into the next iteration as a new input disparity map.

9. An efficient stereo matching device for large parallax scenes, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the efficient stereo matching method for large parallax scenes according to any one of claims 1 to 8.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer program instructions, which, when executed by one or more processors, implement the efficient stereo matching method for large parallax scenes as described in any one of claims 1 to 8.

Images