Binocular stereo matching methods, devices, and media

Abstract

The application discloses a binocular stereo matching method, device and medium. The method comprises the following steps: dividing a first image into a plurality of first image sub-regions; each first image sub-region contains contour information of a photographed object; determining a second image sub-region matched with each first image sub-region from a second image; the first image is an image of the photographed object captured by a left camera, and the second image is an image of the photographed object captured by a right camera; for each first pixel point in each first image sub-region, determining a second pixel point matched with the first pixel point from the matched second image sub-region; and performing binocular stereo matching based on each first pixel point and the matched second pixel point. By using the method, the accuracy of binocular stereo matching can be improved.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer vision technology, specifically to the field of image processing technology, and in particular to a binocular stereo matching method, device, and medium. Background Technology

[0002] Currently, binocular stereo vision matching is widely used in technologies such as virtual reality, line inspection drone operations, intelligent vehicles, and non-contact ranging.

[0003] In the prior art, stereo matching can be performed using local stereo matching algorithms, global stereo matching algorithms, or semi-global matching algorithms. That is, matching pixels in the binocular image (e.g., left and right images captured by a binocular camera) are determined, and then binocular stereo matching is performed based on the matching pixels (e.g., generating a disparity map).

[0004] However, due to the presence of repetitive and weak texture features in binocular images, the existence of these features can interfere with the determination of matching pixels. For example, a pixel in the left image may be matched as multiple pixels in the right image, thus affecting the accuracy of binocular stereo matching. Summary of the Invention

[0005] Therefore, it is necessary to provide a binocular stereo matching method, device, and medium to address the aforementioned technical problems. The method of this application can improve the accuracy of binocular stereo matching.

[0006] Firstly, a binocular stereo matching method is provided, the method comprising:

[0007] The first image is divided into multiple first image sub-regions; each first image sub-region contains the outline information of the captured object;

[0008] Determine the second image sub-regions that match each of the first image sub-regions from the second image; the first image is an image of the subject captured by the left camera, and the second image is an image of the subject captured by the right camera;

[0009] For each first pixel in each first image sub-region, determine the second pixel that matches the first pixel from the matching second image sub-region;

[0010] Stereo matching is performed based on each first pixel and the matching second pixel.

[0011] In a second aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements the steps of the method described in the first aspect and any possible implementation of the first aspect.

[0012] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect and any possible implementation thereof.

[0013] Fourthly, a computer program product is provided, the computer program product including instructions, which, when executed, implement the steps of the method described in the first aspect and any possible implementation of the first aspect.

[0014] In existing technologies for stereo vision matching, the presence of repetitive and weak texture features in the stereo images can interfere with the determination of matching pixels. For example, a single pixel in the left image may be matched as multiple pixels in the right image, thus affecting the accuracy of stereo matching.

[0015] When performing stereo matching using the method of this application, firstly, the first image can be divided into multiple first image sub-regions, each containing the contour information of the object being photographed. Further, when determining the second image sub-region in the second image that matches each of the first image sub-regions, since each first image sub-region contains the contour information of the object being photographed, this contour information can improve the accuracy of stereo matching. Additionally, when determining the second pixel in the second image that matches the first pixel in the first image, region-level matching is performed first, that is, determining the second image region in the second image that matches the first image sub-region in the first image. Then, pixel-level matching is performed within the second image sub-region, that is, determining the second pixel within the second image sub-region that matches the first pixel in the first image sub-region. This narrows the search range for the second pixel, reduces the computational load, and shortens the computation time.

[0016] In summary, the method of this application not only improves the accuracy of binocular stereo matching, but also narrows the search range for determining the second pixel that matches the first pixel by using region-level and pixel-level matching, thereby shortening the computation time of the binocular stereo matching process. Therefore, the method of this application can improve the efficiency of binocular stereo matching. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0018] Figure 1 A flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0019] Figure 2Another flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0020] Figure 3 Another flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0021] Figure 4 Another flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0022] Figure 5 Another flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0023] Figure 6 Another flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0024] Figure 7 Another flowchart of the binocular stereo matching method provided in the embodiments of this application;

[0025] Figure 8 A structural diagram of feature pyramid sampling provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of the structure of the binocular stereo matching device provided in the embodiments of this application;

[0027] Figure 10 A structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The present application will now be described in further detail with reference to the embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0029] It should be noted that, unless otherwise specified, the embodiments and features of the embodiments in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security and prevention, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial recognition payment, facial recognition unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beauty filters, cosmetics, medical aesthetics, and intelligent temperature measurement.

[0031] In the prior art, when performing stereo vision matching, the presence of repetitive and weak texture features in the stereo images can interfere with the determination of matching pixels. For example, a pixel in the left image may be matched as multiple pixels in the right image, thus affecting the accuracy of stereo matching.

[0032] The binocular stereo matching method provided in this application can be executed by electronic devices such as vehicle terminals, cameras, and mobile phones, or by a server.

[0033] Optionally, in another embodiment of this application, Figure 1 The flowchart of the binocular stereo matching method provided in the embodiments of this application can be found in the following document. Figure 1 The method includes the following steps:

[0034] Step 101: Divide the first image into multiple first image sub-regions; each first image sub-region contains the outline information of the captured object;

[0035] Step 102: Determine the second image sub-regions that match each of the first image sub-regions from the second image; the first image is the image of the subject captured by the left camera, and the second image is the image of the subject captured by the right camera;

[0036] Step 103: For each first pixel in each first image sub-region, determine the second pixel that matches the first pixel from the matching second image sub-region;

[0037] Step 104: Perform stereo matching based on each first pixel and the matching second pixel.

[0038] In step 101, the first image is an image of the subject captured by the left camera. For example, the image of the subject captured by the left camera can be the original image of the subject directly captured by the left camera, or it can be an image obtained after resizing the original image. Specifically, the subject can be the same object, the same scene, or the same frame. That is, in this embodiment, when matching pixels in the image of the subject captured by the left camera and the image of the subject captured by the right camera, the image of the subject captured by the right camera (e.g., the original image of the subject captured by the right camera) can be used as a reference image for stereo matching, i.e., determining the pixels in the image of the subject captured by the right camera that match the pixels in the image of the subject captured by the left camera.

[0039] The first image sub-region contains the outline information of the subject. This outline information distinguishes the subject from other objects or the background.

[0040] In one possible implementation, firstly, the first image can be labeled, with regions containing the outline information of the subject marked as 1 and regions not containing the outline information of the subject marked as 0. Secondly, the first image can be divided based on 0 and 1. For example, the first image can be divided into 3*3 image sub-regions. If one of the nine image sub-regions is a region marked only as 0, i.e., a region not containing the outline information of the subject, then a merging method can be adopted. For example, the region marked only as 0 can be merged into the region marked as 0 and 1, or merged into the region marked as 1. Alternatively, the region marked only as 0 can be ignored and not included in the calculation during the stereo matching process, or the region marked only as 0 can be removed, so that during the stereo matching process, regions marked as 0 and 1 and regions marked as 1 participate in the calculation, while regions marked only as 0 do not participate in the calculation. By adopting the above approach, the influence of regions that do not contain the contour information of the subject on the determination of matching pixels can be reduced from the source, i.e., from the image itself, during the process of binocular stereo matching, thereby improving the accuracy of binocular stereo matching based on matching pixels (e.g., generating disparity maps).

[0041] In step 102, the second image is the image of the subject captured by the right-side camera. It should be noted that the image of the subject captured by the right-side camera is the original image of the subject directly captured by the right-side camera.

[0042] The second image sub-region is the region that matches the first image sub-region in the first image.

[0043] In one possible implementation, region similarity matching can be performed on the first image sub-region and the second image based on similarity to obtain the similarity between the first image sub-region and each region in the second image. The region in the second image with the highest similarity to the first image sub-region is taken as the second image sub-region that matches the first image sub-region.

[0044] In specific implementation, a second image sub-region matching the first image sub-region can be searched within the second image based on the search range. Taking a first image sub-region as an example, for a first image sub-region, multiple possible regions can be searched within the aforementioned search range. The similarity between the first image sub-region and these multiple possible regions can be calculated, and the region with the highest similarity can be taken as the second image sub-region matching the first image sub-region. Similarly, for each first image sub-region, the region with the highest similarity to each first image sub-region can be taken as the second image sub-region matching each first image sub-region. In the embodiments of this application, when performing binocular stereo matching, the similar regions of the first and second images are first determined, narrowing the search range of binocular stereo matching from the entire image to a specific region within the image. Then, binocular stereo matching is performed based on the region within the image, which can greatly reduce the search space of binocular stereo matching, reduce the computational load, and thus improve the efficiency of binocular stereo matching.

[0045] In step 103, the first pixel is a pixel in the first image sub-region, and the second pixel is a pixel in the second image sub-region that matches the first pixel in the first image sub-region. A first pixel and a second pixel form a matching pair of pixels.

[0046] In one possible implementation, pixels in a first image sub-region and a second image sub-region can be matched based on pixel similarity to obtain second pixels that match each first pixel in the first image sub-region. Taking a first pixel A in the first image sub-region as an example, a pixel A* at the same pixel position is determined from the second image sub-region based on the pixel position of first pixel A. A neighborhood search can be performed on pixel A* in the second image sub-region to obtain the pixel values ​​of each pixel in the neighborhood. The similarity between each pixel in the neighborhood and the first pixel A in the first image sub-region can be calculated based on the pixel values. The pixel with the highest similarity to pixel A can be selected as the second pixel that matches first pixel A.

[0047] It is understandable that each first pixel corresponds to a matching second pixel. A first pixel and a second pixel form a matching pixel pair. Based on multiple pairs of matching pixels, stereo matching can be performed to generate a disparity map of the first and second images. This disparity map has the same size as either the first or second image. It is also understood that the first image can be the original image of the subject captured by either the left or right camera, hence the disparity map has the same size as the original image. Furthermore, the disparity map can be used to determine the distance between the subject and the stereo camera, and this distance information can be applied to fields such as autonomous driving or assisted driving.

[0048] In one possible implementation, after obtaining the pixel difference between the first pixel in each first image sub-region and the second pixel in the corresponding matching second image sub-region, stereo matching can be performed. For example, the disparity value between the first pixel and the matching second pixel can be obtained, and the disparity map of the first image can be generated from the disparity values ​​of each first pixel in the first image, thus completing stereo matching.

[0049] When performing stereo matching using the method provided in this application, the first image can firstly be divided into multiple first image sub-regions, each containing the contour information of the object being photographed. Further, when determining the second image sub-region in the second image that matches each of the first image sub-regions, since each first image sub-region contains the contour information of the object being photographed, this contour information can improve the accuracy of stereo matching. Additionally, when determining the second pixel in the second image that matches the first pixel in the first image, region-level matching is first performed, i.e., determining the second image region in the second image that matches the first image sub-region in the first image. Then, pixel-level matching is performed within the second image sub-region, i.e., determining the second pixel within the second image sub-region that matches the first pixel in the first image sub-region. This narrows the search range for the second pixel, reduces the computational load, and shortens the computation time.

[0050] In summary, the method of this application not only improves the accuracy of binocular stereo matching, but also narrows the search range for determining the second pixel that matches the first pixel by using region-level and pixel-level matching, thereby shortening the computation time of the binocular stereo matching process. Therefore, the method of this application can improve the efficiency of binocular stereo matching.

[0051] The embodiments described above illustrate how to perform binocular stereo matching. Another embodiment of this application describes how to specifically divide the image into multiple first image sub-regions. For example, the specific implementation of "dividing the first image into multiple first image sub-regions" involved in the preceding embodiments includes... Figure 2 Steps:

[0052] Step 201: Determine the gradient image corresponding to the first image, and determine the gradient region and noise region in the first image;

[0053] The gradient image is the image obtained by calculating the gradient of the first image. It can be understood that the first image can be the original image of the subject captured by the left or right camera, or it can be an image obtained by resizing the original image. For example, the gradient image corresponding to the first image can be I... T .

[0054] In one possible implementation, gradient regions in the gradient image corresponding to the first image can be marked as 1, noise regions as 0, and regions containing both gradient and noise regions can be marked as both 0 and 1. It should be noted that the regions marked as 0 and 1 are also gradient regions. For example, the gradient image corresponding to the first image with noise marking could be I... m It is understood that the use of 0 and 1 to label the gradient image here is merely illustrative, and other values ​​or methods can also be used for labeling. In other words, this application does not limit the specific numerical values ​​of the labels.

[0055] Step 202: Divide the first image after noise labeling into regions to obtain multiple initial image sub-regions; each initial image sub-region contains a gradient region and / or a noise region;

[0056] The initial image sub-region can be a region containing gradient regions or a region containing noise regions. Specifically, the region containing gradient regions can be a region containing only gradient regions or a region containing both gradient regions and noise regions.

[0057] In one possible implementation, the first image after noise labeling can be divided into rows at equal intervals and columns at unequal intervals, or the first image after noise labeling can be divided into rows at unequal intervals and columns at equal intervals, thereby obtaining multiple initial image sub-regions of different sizes. For example, the first image after noise labeling can be I m The aforementioned initial image sub-region can be I. m1 ~I m16 .

[0058] It should be noted that the first image can also be divided into regions first to obtain initial image sub-regions, and then noise regions can be marked to obtain noise-marked initial image sub-regions.

[0059] Step 203: Merge the image sub-regions containing only noise regions and the image sub-regions containing gradients in the multiple initial image sub-regions to obtain multiple first image sub-regions; or, filter the image sub-regions containing only noise regions to obtain multiple first image sub-regions.

[0060] Here, the image sub-region is the region obtained after processing the initial image sub-region. For example, the processed region can be a region containing only gradient regions, or a region containing both noise regions and gradient regions.

[0061] In one possible implementation, if the initial image sub-region is I m1 ~I m16, among which, I m2 For regions containing only noise, I can be... m2 Merging with image sub-regions containing gradient regions, for example, can be done with I m1 Or I m3 After merging, the first image sub-region I is obtained, which contains both gradient regions and noise regions. m1’ Or I m3’ The remaining first image sub-regions I are also obtained. m3 ~I m16 Or I m1 I m4 ~I m16 .

[0062] In one possible implementation, if the initial image sub-region is I m1 ~I m16 , among which, I m2 For regions containing only noise, the first image I after noise labeling can be processed. m Image processing can be performed, for example, removing regions from an image that contain only noise features, to obtain I. m’ Alternatively, the first image I after noise labeling can be omitted. m In image processing, for example, during subsequent processing, regions I containing only noise are ignored or disregarded. m2 .

[0063] The preceding implementation described how to divide the image into multiple first image sub-regions. In another embodiment of this application, when the first image is a resized image, how to determine the gradient image corresponding to the first image is described. For example, the specific implementation of "determining the gradient image corresponding to the first image" involved in the preceding steps includes... Figure 3 Steps:

[0064] Step 301: Perform the target number of first size transformations on the original image to obtain the target number of candidate images of different sizes; the size of the candidate image is less than or equal to that of the original image; the first image is the smallest candidate image among the target number of candidate images of different sizes.

[0065] The original image is the image that has not undergone any resizing. For example, it could be an image of the subject captured by the camera on the left.

[0066] Here, candidate images are a target number of images of different sizes obtained by performing a target number of first-size transformations on the original image. It can be understood that the size of the candidate image is smaller than or equal to that of the original image. After performing the target number of size transformations, a first image is obtained, and the first image is the smallest image among the target number of candidate images. For example, the specific number of target number transformations can be determined based on the size of the desired smallest candidate image (i.e., the first image). For instance, it could be 3 transformations.

[0067] In one possible implementation, the original image can be downsampled based on a feature pyramid, resulting in multiple images of different sizes arranged in a pyramidal pattern. For example, the image size decreases progressively from the base to the apex of the pyramid, which could be I... L1 ~I L3 .

[0068] Step 302: Perform gradient processing on the first image to obtain a gradient image.

[0069] The first image is the smallest candidate image among multiple candidate images of different sizes. For example, the smallest image could be the image of the pyramid's apex. L3 .

[0070] In one possible implementation, the image I at the top of the pyramid can be used. L3 Gradient processing is performed using the Sobel operator to obtain the gradient image I corresponding to the first image. T1 .

[0071] It should be noted that in the scheme of downsampling the original image and determining the gradient image corresponding to the first image based on the smallest candidate image (i.e., the first image), when performing region similarity matching on the first image and the second image, the gradient image is determined by (0, d). max The search scope is determined in the second image by identifying the second image sub-regions that match each of the multiple first image sub-regions in the first image.

[0072] Where, d max It is the maximum parallax, which is determined by the parameters of the binocular image acquisition device, namely the device focal length f, the base distance b, and the distance x between the object (equivalent to the shooting object in the previous embodiment of this application) and the device.

[0073] The embodiments described above illustrate how to determine the gradient image corresponding to the first image when the first image is an image that has undergone size transformation. In another embodiment of this application, when the first image is an image that has undergone size transformation, a disparity map of the original image is generated, i.e., binocular stereo matching. For example, the specific implementation of the step described above, "performing binocular stereo matching based on each first pixel and the matching second pixel," includes... Figure 4 Steps:

[0074] Step 401: For the first image, generate a disparity map of the first image based on the pixel difference between each first pixel and the second pixel that matches each first pixel.

[0075] The disparity map of the first image is an image generated based on the pixel difference between each first pixel in the first image and the second pixel in the matching second image.

[0076] In one possible implementation, after performing three first-size transformations on the original image, the first image I is obtained. L3 In the middle (i.e., the candidate image with the smallest size), calculate the first image I. L3 The pixel difference between each first pixel in the first image and the second pixel in the matching second image is used to obtain multiple pixel differences. The image composed of multiple pixel differences is the disparity map of the first image.

[0077] Step 402: Generate a disparity map of the original image based on the disparity map of the first image.

[0078] The parallax map of the original image is an image of the same size as the left or right camera.

[0079] In one possible implementation, the disparity map of the first image can be upsampled to obtain a result similar to the candidate image I. L2 Images of the same size are compared with candidate image I. L2 Perform pixel matching to obtain candidate image I L2 The corresponding multiple pixel differences are repeatedly processed until the candidate image is I. L1 (i.e., candidate images with the same size as the original image), resulting in candidate image I. L1 The corresponding multiple pixel differences, candidate image I L1 The image composed of the corresponding differences between multiple pixels is the disparity map of the original image.

[0080] The embodiments described above introduced how to generate a disparity map of the original image when the first image is an image that has undergone size transformation, i.e., binocular stereo matching. Another embodiment of this application describes how to obtain a disparity map of the original image based on the disparity map of the first image when the first image is an image that has undergone size transformation. For example, the specific implementation of the step described above, "generating a disparity map of the original image based on the disparity map of the first image," includes... Figure 5 Steps:

[0081] Step 501: Perform a second size transformation on the disparity map of the first image to obtain a reference disparity map;

[0082] The reference disparity map is an image obtained by performing a second size transformation on the disparity map of the first image. For example, it can be an image with an enlarged size. For example, it can be an image compared to the candidate image I. L2 Images of the same size.

[0083] In one possible implementation, the disparity map of the first image can be upsampled once based on the feature pyramid, thus completing the second size transformation process.

[0084] Specifically, the disparity map of the first image can be upsampled based on the feature pyramid, that is, the size of the disparity map of the first image can be enlarged to obtain a similar image to the candidate image I. L2 Images of the same size are considered candidate images (I). L2 Reference parallax map.

[0085] Step 502: Based on the reference disparity map and the candidate image corresponding to the reference disparity map, perform pixel similarity matching, and determine the disparity map of the candidate image corresponding to the reference disparity map based on the matching results; the reference disparity map and the candidate image corresponding to the reference disparity map have the same size;

[0086] The reference disparity map is an image obtained by performing a second size transformation on the disparity map of the first image. For example, the reference disparity map is an image enlarged from the disparity map of the first image; for instance, it could be an image similar to the candidate image I. L2 Images of the same size.

[0087] The disparity map of the candidate image corresponding to the reference disparity map is an image generated by taking the pixel difference between each pixel of the reference disparity map and each first pixel of the candidate image corresponding to the reference disparity map. For example, the disparity map of the candidate image corresponding to the reference disparity map can be the same as I. L2 Images of the same size.

[0088] In one possible implementation, the candidate image I corresponding to each pixel in the reference disparity map can be calculated. L2 The similarity between corresponding pixels in the image is used to determine the first pixel, which is the pixel with the highest similarity to each pixel in each reference disparity map. Based on the pixel difference between each pixel and the matching first pixel, a candidate image I corresponding to the reference disparity map can be generated. L2 The disparity map.

[0089] Step 503: If the disparity map of the candidate image corresponding to the reference disparity map has the same size as the original image, then the disparity map of the candidate image corresponding to the reference disparity map is used as the disparity map of the original image; if the size of the disparity map of the candidate image corresponding to the reference disparity map is smaller than the size of the original image, then the disparity map of the candidate image corresponding to the reference disparity map is used as the initial disparity map of the next candidate image, and pixel similarity matching is performed until the disparity map of the candidate image corresponding to the reference disparity map has the same size as the original image.

[0090] In one possible implementation, if the size of the disparity map of the candidate image corresponding to the reference disparity map is the same as that of the candidate image I... L3 The dimensions are the same, and the dimensions of the first image are the same as those of the candidate image I. L1 The dimensions are the same. Because the candidate image I... L3 The size is smaller than I L1 The size will be the same as I. L3 The disparity map of the corresponding image is used as I. L2 The initial disparity map is used to perform comparisons with candidate images I. L2 The pixel similarity matching yields a disparity map of candidate images corresponding to the reference disparity map, with a size of I. L1 , with the original image size I L1 If they are the same, then they will be with I. L1 The disparity map of the candidate image corresponding to the reference disparity map of the same size is used as the disparity map of the original image.

[0091] Understandably, when using the currently determined disparity map as the reference disparity map for the next candidate image and performing pixel similarity matching with the second image, the value range of the search range corresponding to the next candidate image is twice the current search range. For example, the value range of the search range corresponding to the first candidate image is {0, d...} max / 8}, the search range corresponding to the second candidate image is {0, d}. max / 4}, the search range corresponding to the third candidate image is {0, d}. max / 2}, the search range corresponding to the fourth candidate image is {0, d max}

[0092] The embodiments described above illustrate how to obtain the disparity map of the original image based on the disparity map of the first image when the first image is an image after size transformation. In another embodiment of this application, the specific implementation of pixel similarity matching is described, such that each first pixel in a sub-region of the first image has a matching second pixel in the corresponding sub-region of the second image. For example, the specific implementation of the step described above, "for each first pixel in each sub-region of the first image, determining the matching second pixel from the matching sub-region of the second image," includes... Figure 6 Steps:

[0093] Step 601: For each pixel in the sub-region of the first image, determine the reference pixel that is in the same position in the second image;

[0094] The reference pixel is the pixel in the second image that has the same position as the pixel in the sub-region of the first image. For example, if the coordinates of a pixel in the sub-region of the first image are (x1, y1), then the position of the reference pixel in the second image that has the same position is also (x1, y1).

[0095] In one possible implementation, when the coordinates of a pixel in the first image sub-region are (x1, y1), the position of a reference pixel in the second image that is at the same position can be (x1, y1).

[0096] Step 602: Determine multiple candidate pixels based on the search range and reference pixels;

[0097] The search range is used to characterize the area where the second pixel is located. For example, it can be a circle with a fixed center and radius.

[0098] Candidate pixels are used to represent the set of pixels to which the target pixel belongs. For example, a pixel in the set of pixels may be the second pixel.

[0099] In one possible implementation, when a second image sub-region matching the first image sub-region is determined, the deviation range value between the reference pixel and the second pixel can be obtained simultaneously. This deviation range value can be used as the search range, with the reference pixel as the center of the search range. Since the first pixel and the second pixel are a pair of matching pixels, their pixel values ​​are similar. With the reference pixel as the center and the deviation range value as the search range, several points can be selected around the center at equal or unequal distances, and these points can be determined as multiple candidate reference points.

[0100] Step 603: Based on the similarity between multiple candidate pixels, determine the second pixel from the multiple candidate pixels; wherein the search range is determined based on the offset between the first image sub-region and the second image sub-region.

[0101] Similarity is used to characterize the degree of similarity between candidate pixels. For example, it could mean that the similarity between candidate pixels is very high, or that the difference between candidate pixels is very low.

[0102] The second pixel is used to represent a pixel that forms a matching pair with a pixel in the first image sub-region. For example, it could be two pixels with a positional offset.

[0103] In one possible implementation, the similarity between each candidate pixel can be calculated, and the candidate pixel with the highest similarity value can be taken as the second pixel, which forms a pair of matching points with the aforementioned pixel.

[0104] The embodiments described above illustrate how pixel similarity matching is performed so that each first pixel in the first image sub-region has a matching second pixel in the corresponding first image sub-region. In another embodiment of this application, the specific range of the search area is described. The specific range of the search area can be:

[0105] The search range is (xa, x+a), where x is the average offset between the first and second image sub-regions, and a is a constant.

[0106] The average offset is used to characterize the average pixel offset between the first image sub-region and the corresponding second image sub-region, or the quantized value of the offset between the first and second image sub-regions. For example, if the average offset is 3 pixels, it means that the positional offset between each first pixel in the first image sub-region and its corresponding second pixel in the second image sub-region does not exceed 3 pixels.

[0107] In one possible implementation, if the average offset x between the first image sub-region and the matching second image sub-region is 3, then when searching for a target pixel, the search range in the second image sub-region is (3-a, 3+a). Here, a is a constant, and this constant does not exceed the size of the second image sub-region.

[0108] The specific range of the search area was described in the embodiments described above. In another embodiment of this application, a flowchart for generating a disparity map is described. Figure 7 This is another flowchart of generating a disparity map provided in an embodiment of this application. (Refer to...) Figure 7 The specific execution steps of this flowchart are as follows:

[0109] P1, Calibration parameters.

[0110] It should be noted that the parameters are those of the binocular image acquisition device, such as the device focal length f, base distance b, and distance x between the object (equivalent to the shooting object in the previous embodiment of this application) and the device.

[0111] P2. Calculate the maximum disparity.

[0112] It should be noted that the maximum disparity dmax = b * f / x. Preferably, d max It can be a multiple of 8.

[0113] P3, stereoscopically corrected dual-camera image.

[0114] It should be noted that the stereoscopic corrected dual-camera image here is equivalent to the original image in the previous embodiments of this application.

[0115] P4, pyramid sampling.

[0116] It should be noted that the pyramid sampling here is equivalent to the first size transformation in the previous embodiments of this application. For example, sampling can end when the image size is less than 200 pixels, and the number of sampling layers can start from I. L1 ~I L4 . Figure 8 A structural diagram of a feature pyramid sampling method provided in an embodiment of this application. (Reference) Figure 8 , Figure 8 The left half of the image shows four candidate images of different sizes obtained after downsampling the original image four times (i.e., the first size transformation). From top to bottom, they are the first candidate image I. L1 Second candidate image I L2 Third candidate image I L3 Fourth candidate image I L4 .

[0117] P5, Gradient Calculation.

[0118] It should be noted that the gradient calculation here is equivalent to determining the gradient image corresponding to the first image in the previous embodiments of this application. It can be either the gradient image of the first image or the image obtained after performing a first size transformation on the first image. For example, a downsampled image.

[0119] P6, Weight Calculation.

[0120] It should be noted that the weight calculation here is equivalent to the noise labeling of the gradient image in the previous embodiments of this application, where the noise region is labeled as 0 and the gradient region is labeled as 1.

[0121] P7, Global Matching.

[0122] It should be noted that the global matching here is equivalent to the region similarity matching in the previous embodiments of this application.

[0123] P8, Two-way matching.

[0124] It should be noted that the bidirectional matching here is equivalent to the pixel similarity matching in the previous embodiments of this application. Furthermore, the bidirectional matching can employ the LR check method.

[0125] P9, Pyramid Matching.

[0126] It should be noted that the pyramid matching here is equivalent to determining the disparity map of the first image based on the reference disparity map in the previous embodiments of this application. Figure 8 A structural diagram of a feature pyramid sampling method provided in an embodiment of this application. (Reference) Figure 8 , Figure 8 The right half of the image contains disparity maps of multiple candidate images of different sizes. From top to bottom, they are candidate image disparity map D1, candidate image disparity map D2, candidate image disparity map D3, and candidate image disparity map D4.

[0127] In one possible implementation, the disparity map of the original image can be generated based on the disparity map of the first image. (See reference) Figure 10 First, a disparity map D1 of the first image can be generated based on the pixels in each sub-region of the first image and the second pixels in the sub-region of the second image in the second image that match each sub-region of the first image.

[0128] Secondly, D1 can be upsampled (i.e., the second size transformation) to obtain an image with the same size as the candidate image IL2. This image is used as a reference disparity map of the candidate image IL2. The pixels of the reference disparity map and the pixels of the candidate image IL2 are matched for pixel similarity to obtain the disparity map D2 of the candidate image IL2.

[0129] Then, D2 can be upsampled (i.e., the second size transformation) to obtain an image with the same size as the candidate image IL3. This image is used as a reference disparity map of the candidate image IL3. The pixels of the reference disparity map and the pixels of the candidate image IL3 are matched for pixel similarity to obtain the disparity map D3 of the candidate image IL3.

[0130] Next, D3 can be upsampled (i.e., the second size transformation) to obtain an image with the same size as the candidate image IL4. This image is used as the reference disparity map of the candidate image IL4. Pixel similarity matching is performed between the pixels of the reference disparity map and the pixels of the candidate image IL4 to obtain the disparity map D4 of the candidate image IL4. If the disparity map of the candidate image IL4 has the same size as the original image, then the disparity map of the candidate image IL4 is the disparity map of the original image. It can be understood that if the size of the disparity map of the candidate image IL4 is smaller than the size of the original image, the above upsampling and subsequent steps can continue until the size of the disparity map of the candidate image ILn is equal to the size of the original image, at which point the disparity map of the candidate image ILn is used as the candidate disparity map of the original image.

[0131] The embodiments described above present a complete flowchart for generating a disparity map. Figure 9 This is a schematic diagram of the structure of a binocular stereo matching device provided in an embodiment of this application. The device includes: a segmentation unit 901, a region determination unit 902, a pixel determination unit 903, and a matching unit 904. Wherein:

[0132] The segmentation unit 901 is used to divide the first image into multiple first image sub-regions; each first image sub-region contains the outline information of the captured object;

[0133] The region determination unit 902 is used to determine a second image sub-region that matches each of the first image sub-regions from the second image; the first image is an image of the subject captured by the left camera, and the second image is an image of the subject captured by the right camera;

[0134] The pixel determination unit 903 is configured to determine, for each first pixel in each first image sub-region, a second pixel that matches the first pixel from a matching second image sub-region;

[0135] Matching unit 904 is used for stereo matching based on each first pixel and the matched second pixel.

[0136] In one embodiment, the segmentation unit 901 is specifically used to determine the gradient image corresponding to the first image, and to determine the gradient region and noise region in the first image;

[0137] The first image after noise labeling is divided into regions to obtain multiple initial image sub-regions; each initial image sub-region contains a gradient region and / or a noise region.

[0138] Multiple initial image sub-regions are merged by combining image sub-regions containing only noise regions with image sub-regions containing gradients to obtain multiple first image sub-regions; or, image sub-regions containing only noise regions are filtered to obtain multiple first image sub-regions.

[0139] In one embodiment, the segmentation unit 901 is specifically used to determine the gradient image corresponding to the first image, including:

[0140] The original image is subjected to the target number of first size transformations to obtain the target number of candidate images of different sizes; the size of the candidate image is less than or equal to that of the original image; the first image is the smallest candidate image among the target number of candidate images of different sizes.

[0141] Gradient processing is performed on the first image to obtain a gradient image.

[0142] In one embodiment, the matching unit 904 is configured to generate a disparity map of the first image based on the pixel difference between each first pixel in the first image and a second pixel that matches each first pixel.

[0143] Generate a disparity map of the original image based on the disparity map of the first image.

[0144] In one embodiment, the matching unit 904 specifically performs a second size transformation on the disparity map of the first image to obtain a reference disparity map;

[0145] Pixel similarity matching is performed based on the reference disparity map and the corresponding candidate image. The disparity map of the candidate image corresponding to the reference disparity map is determined based on the matching results. The reference disparity map and the corresponding candidate image have the same size.

[0146] If the disparity map of the candidate image corresponding to the reference disparity map is the same size as the original image, then the disparity map of the candidate image corresponding to the reference disparity map is used as the disparity map of the original image; if the size of the disparity map of the candidate image corresponding to the reference disparity map is smaller than the size of the original image, then the disparity map of the candidate image corresponding to the reference disparity map is used as the initial disparity map of the next candidate image, and pixel similarity matching is performed until the disparity map of the candidate image corresponding to the reference disparity map is the same size as the original image.

[0147] In one embodiment, the pixel determination unit 903 is specifically used to determine, for each pixel in the first image sub-region, a reference pixel that is in the same position in the second image;

[0148] Multiple candidate pixels are determined based on the search range and reference pixels;

[0149] The second pixel is determined from multiple candidate pixels based on the similarity between multiple candidate pixels.

[0150] The search range is determined based on the offset between the first image sub-region and the second image sub-region; the search range is (xa, x+a), where x is the average offset between the first image sub-region and the second image sub-region, and a is a constant.

[0151] Specific limitations regarding the binocular stereo matching device can be found in the limitations of the binocular stereo matching method above, and will not be repeated here. Each module of the aforementioned binocular stereo matching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the computer device's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0152] In one embodiment, an electronic device is provided. Figure 10 This is a structural block diagram of the electronic device provided in the embodiments of this application, with reference to... Figure 10 The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned binocular stereo matching method embodiment.

[0153] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned binocular stereo matching method embodiment.

[0154] This application provides a computer program product including instructions that, when executed, cause the method described in this application to be performed. For example, it can execute... Figure 1 The steps of the binocular stereo matching method are shown.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A binocular stereo matching method, characterized in that, The method includes: The first image is divided into multiple first image sub-regions; each first image sub-region contains the outline information of the captured object; A second image sub-region is determined from the second image that matches each of the first image sub-regions; the first image is an image of the subject captured by the left camera, and the second image is an image of the subject captured by the right camera; For each first pixel in each first image sub-region, a second pixel matching the first pixel is determined from the matching second image sub-region; Perform stereo matching based on each first pixel and the matching second pixel; The process of dividing the first image into multiple first image sub-regions includes: Determine the gradient image corresponding to the first image, and determine the gradient region and noise region in the first image; The first image after noise labeling is divided into regions to obtain multiple initial image sub-regions; each initial image sub-region contains a gradient region and / or a noise region. The image sub-regions containing only noise regions and the image sub-regions containing gradients in the plurality of initial image sub-regions are merged to obtain the plurality of first image sub-regions; or, the image sub-regions containing only noise regions are filtered to obtain the plurality of first image sub-regions.

2. The method according to claim 1, characterized in that, The first image is the original image of the subject captured by the left-side camera, which has been resized. Determining the gradient image corresponding to the first image includes: The original image is subjected to a target number of first size transformations to obtain a target number of candidate images of different sizes; the size of the candidate images is less than or equal to that of the original image; the first image is the smallest candidate image among the target number of candidate images of different sizes. Gradient processing is performed on the first image to obtain the gradient image.

3. The method according to claim 1 or 2, characterized in that, The first image is the original image of the subject captured by the left-side camera, which has been resized. The stereo matching based on each first pixel and the matching second pixel includes: For the first image, a disparity map of the first image is generated based on the pixel difference between each first pixel in the first image and the second pixel that matches each first pixel. The disparity map of the original image is generated based on the disparity map of the first image.

4. The method according to claim 3, characterized in that, The step of generating the disparity map of the original image based on the disparity map of the first image includes: A second size transformation is performed on the disparity map of the first image to obtain a reference disparity map; Based on the reference disparity map and the candidate image corresponding to the reference disparity map, pixel similarity matching is performed, and the disparity map of the candidate image corresponding to the reference disparity map is determined based on the matching result; the reference disparity map and the candidate image corresponding to the reference disparity map have the same size; If the disparity map of the candidate image corresponding to the reference disparity map has the same size as the original image, then the disparity map of the candidate image corresponding to the reference disparity map is used as the disparity map of the original image; if the size of the disparity map of the candidate image corresponding to the reference disparity map is smaller than the size of the original image, then the disparity map of the candidate image corresponding to the reference disparity map is used as the initial disparity map of the next candidate image, and pixel similarity matching is performed until the disparity map of the candidate image corresponding to the reference disparity map has the same size as the original image.

5. The method according to claim 1 or 2, characterized in that, The step of determining a matching second pixel from a matching second image sub-region for each first pixel in each first image sub-region includes: For each pixel in the first image sub-region, a reference pixel is determined that has the same position in the second image; Multiple candidate pixels are determined based on the search range and the reference pixels; The second pixel is determined from the plurality of candidate pixels based on the similarity between the plurality of candidate pixels and the pixel. The search range is determined based on the offset between the first image sub-region and the second image sub-region.

6. The method according to claim 5, characterized in that, The search range is (xa, x+a), where x is the average offset between the first image sub-region and the second image sub-region, and a is a constant.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the binocular stereo matching method as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the binocular stereo matching method as described in any one of claims 1-6.

9. A computer program product, comprising instructions, characterized in that, When the instruction is executed, it implements the stereo matching method as described in any one of claims 1-6.

Images