An Image Sparse Point Stereo Matching Method

Abstract

The present invention discloses an image sparse point stereo matching method, including: performing stereo rectification on the images collected by each camera of a multi-camera; obtaining the common view area of each camera; grouping the sparse points in the sparse point sets obtained by extracting the images of the common view area collected by each camera according to the positions of the sparse points in the longitudinal or transverse direction, and aligning the groups in each sparse point set accordingly; and making the sparse points in the aligned groups correspond one by one. The method of the present invention can match the images collected by a black and white binocular camera, and finally realize spatial positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image sparse point stereo matching method. Background Art

[0002] A binocular or multi-camera positioning system (hereinafter collectively referred to as a multi-camera system), as Figure 1 shown, is a process of simultaneously capturing images of the same target object in space by different cameras at different angles, extracting the pixel positions of the target object in the images, and then reconstructing the spatial position of the target object. Stereo matching (which can be called stereo matching for both binocular and multi-camera systems) is essential for whether the target object can be reconstructed. The traditional sparse point stereo matching strategy is to extract the feature point information (such as descriptors, etc.) of the target object in each image, and then perform matching according to the similarity of the feature point descriptors. Since the sparse points extracted from black and white images do not have features such as descriptors, they cannot be matched using similarity, so a new sparse point stereo matching method is needed. Summary of the Invention

[0003] Object of the Invention: Aiming at the above deficiencies, the present invention proposes an image sparse point stereo matching method, which can match the images collected by black and white binocular cameras and finally achieve spatial positioning.

[0004] Technical Solution:

[0005] An image sparse point stereo matching method includes:

[0006] Performing stereo calibration on the images collected by each camera of the multi-camera;

[0007] Obtaining the common viewing area of each camera;

[0008] Grouping the sparse points in each sparse point set according to the positions of the sparse points in the longitudinal or transverse direction in the images of the common viewing area collected by each camera, and aligning the groups in each sparse point set accordingly;

[0009] Making the sparse points in the aligned groups correspond one by one.

[0010] Before grouping, a denoising operation is performed on each sparse point set, specifically:

[0011] For each sparse point in the sparse point set of each image, calculate the difference between its y coordinate and the y coordinates of all sparse points in the sparse point sets of other images in their respective corresponding image coordinate systems. If there is a result where the difference is less than a set threshold, it is considered that there is a matching point; otherwise, it is considered a noise point and is removed.

[0012] Grouping the sparse points according to the positions of the sparse points in the longitudinal or transverse direction specifically includes:

[0013] For all sparse points in each sparse point set, select any coordinate in the corresponding image coordinate system and arrange them in ascending or descending order. Calculate the difference between adjacent two sparse points at the selected coordinate respectively. If the difference is less than the set threshold, continue to calculate the difference of the next adjacent sparse point; if the difference is greater than the set threshold, take the average value of the two sparse points at the selected coordinate as a segmentation value and put it into the segmentation value set of the corresponding image until all differences are calculated;

[0014] The segmentation values in each segmentation value set correspond to each other to form corresponding segmentation value groups. Take the largest segmentation value in each corresponding segmentation value group as the actual segmentation value of the group, so as to obtain an actual segmentation value set. Use the segmentation values of the actual segmentation value set to group each sparse point set to obtain the grouping set of each sparse point set.

[0015] The specific group alignment is as follows:

[0016] Calculate the average value of the selected coordinates of all sparse points in each group in each grouping set as the description of the group respectively. Compare the group descriptions of each group in each grouping set with the group descriptions of each group in other grouping sets respectively. If the difference between the two is less than the set threshold, it is considered that the two groups are corresponding groups, and finally the corresponding relationship of the grouping sets under each image is obtained.

[0017] The selected coordinate is determined according to the grouping method of the positions of each sparse point in the vertical or horizontal direction. Among them, the vertical grouping is to select the y coordinate as the selected coordinate, and the horizontal grouping is to select the x coordinate as the selected coordinate.

[0018] For horizontal grouping, the one-to-one correspondence of sparse points in each aligned group is specifically as follows:

[0019] Sort the x coordinates of the sparse points in the corresponding group respectively; if the number of sparse points in the two corresponding groups is the same, it is considered that the two groups of sparse points are matched, and the sorted sparse points are one-to-one corresponding matching points; if the number of sparse points in the two corresponding groups differs by n points, it is necessary to judge whether the image with more sparse points is collected by the left camera or the right camera. If it is collected by the left camera, remove the n sparse points from left to right in the corresponding group in the image collected by this camera. If it is collected by the right camera, remove the n sparse points from right to left in the corresponding group in the image collected by this camera; thus, one-to-one corresponding matching points in the corresponding group are obtained.

[0020] For vertical grouping, the one-to-one correspondence of sparse points in each aligned group is specifically as follows:

[0021] Judge whether the number of groups obtained by grouping the sparse point sets of each image is equal. If they are equal, perform corresponding group sparse matching; if the number of groups of the two differs by m groups, directly remove the m uncorresponding groups; sort the y coordinates of the sparse points in the corresponding group, and the sorted sparse points are one-to-one corresponding matching points.

[0022] The specific method for obtaining the common viewing area of each camera is as follows:

[0023] Through the stereo rectification, the optical centers of the images collected by each camera are aligned. Place a target object, and when the target object appears in the fields of view of all cameras, obtain the boundary of the common viewing area, thereby obtaining the common viewing area of each camera.

[0024] It also includes a verification step:

[0025] According to the internal and external parameters of the corresponding camera and the coordinates of the corresponding sparse points, triangulation is performed to obtain the spatial coordinates of the matching points respectively. If the corresponding spatial coordinates meet the set requirements, they are considered correct matching points; otherwise, they are considered incorrect matching points and are excluded.

[0026] The set requirements are determined according to the actual field of view of the camera and are specifically set as the coordinate offset threshold.

[0027] Beneficial effects: The traditional method of using the similarity of all feature points in the images collected by binocular cameras for matching cannot meet the requirements for black-and-white images. Black-and-white images do not have other features except for position features. The sparse point matching method of the present invention can match the images collected by black-and-white binocular cameras and finally achieve spatial positioning. Brief Description of the Drawings

[0028] Figure 1 It is a schematic diagram of a binocular or multi-camera positioning system;

[0029] Figure 2 It is a flowchart of the sparse point stereo matching method of the present invention;

[0030] Figure 3 It is a schematic diagram of stereo rectification;

[0031] Figure 4 It is a schematic diagram of the obtained common viewing area;

[0032] Figure 5 It is a schematic diagram of the grouping of pixel points of multi-camera images;

[0033] Figure 6 It is a schematic diagram of the stereo matching result of the present invention. Detailed Embodiments

[0034] The present invention will be further illustrated below in conjunction with the drawings and specific embodiments.

[0035] Figure 2This is a flow chart of the sparse point stereo matching method of the present invention. The present invention performs stereo matching on a number of images collected by a multi-camera, especially black and white images. In the matching process, a number of images are matched in pairs. The purpose of stereo matching is to find the corresponding sparse points in other sparse point sets for each sparse point in a sparse point set, thereby calculating the corresponding spatial position, and finally achieving spatial positioning. Figure 2 As shown, the sparse point stereo matching method of the present invention comprises the following steps:

[0036] (1) Perform stereo correction on the images collected by each camera of the multi-eye system;

[0037] Since there are errors in the installation of each camera of the multi-eye system, the images taken by each camera of the multi-eye system are not aligned in the same plane. The present invention performs stereo correction on the images collected by each camera of the multi-eye system. The stereo correction of the image is specifically as follows: the imaging points of the objects in the space on the images collected by each camera are obtained, and the points with the same y coordinates or x coordinates on the corresponding image coordinate systems are aligned, such as Figure 3 As shown, the present invention takes a binocular camera as an example. After stereo calibration, each image has completed coplanar alignment;

[0038] (2) Obtain the common viewing area of ​​each camera in the multi-eye system;

[0039] Because stereo correction can align the optical centers of the images collected by each camera in the multi-eye system, such as Figure 3 As shown in the figure, after stereo correction, the images taken by different cameras will be shifted up and down. After the optical centers are aligned, the common field of view of multiple cameras needs to be calculated, such as Figure 4 The specific calculation is as follows:

[0040] Place a target object, perform anti-distortion on the target object’s pixels, and then find the upper, lower, left, and right boundaries when the target object appears in the field of view of all cameras. Pixels that exceed the common viewing area are removed to prevent interference, thereby obtaining the common field of view of multiple cameras, that is, the common viewing area of ​​each camera in the multi-camera system.

[0041] (3) extracting sparse point coordinates from the images in the common view area captured by each camera (i.e., the effective pixel range of the images captured by each camera) to obtain a sparse point set;

[0042] Extract the sparse points on each image and obtain multiple sparse point sets P0, P1, ..., P i ;

[0043] Considering that there are noise points in the images collected by each camera, which affect the subsequent sparse point matching, it is necessary to remove the noise points before matching. The specific method is to calculate the difference in the y - coordinates of each sparse point in the sparse point set of any image with all sparse points in the sparse point sets of other images in their respective corresponding image coordinate systems. If there is a result where the difference is less than val, it is considered that there is a matching point; otherwise, it is considered a noise point and needs to be removed. Here, val is an empirical threshold, usually within 1 pixel.

[0044] After the above operations, the denoised sparse point sets P0', P1',..., P i ' can be obtained;

[0045] (4) Group the denoised sparse point sets of each image respectively;

[0046] The present invention can perform horizontal segmentation on each sparse point set: Arrange all the sparse points in each sparse point set P0', P1',..., P i ' in ascending or descending order according to their y - coordinates in the corresponding image coordinate system. Calculate the difference d in the y - coordinates of two adjacent sparse points p k and p k+1 respectively. If d ≤ val, continue to calculate the difference d of the next adjacent sparse point; if d > val, take l k =(y k +y k+1 ) / 2 as a segmentation value and record it, and continue to calculate the difference d of the next adjacent sparse point. Here, val is an empirical threshold, usually within 1 pixel. y k and y k+1 respectively represent the y - coordinates of sparse points p k and p k+1 ; until the differences d of all adjacent sparse points in each sparse point set are calculated, and take the multiple segmentation values of each sparse point set recorded as the corresponding segmentation value set of each sparse point set;

[0047] Since after step (3), the sparse points in each image are corresponding sparse points, there is also a corresponding relationship among the segmentation values in the segmentation value sets of each sparse point set. Compare the corresponding segmentation values in each sparse point set one by one, and select the largest segmentation value as the actual segmentation value, so as to obtain an actual segmentation value set; use the segmentation values in the actual segmentation value set to group the sparse points in the sparse point sets of each image, and obtain a grouping set. As Figure 5 shown, at this time, the sparse point set in the image is divided into several groups g1, g2,..., g6 according to the y - coordinate. In this embodiment, the grouping sets of the images collected by the two cameras are group0 and group1 respectively;

[0048] In another specific embodiment, the present invention can also perform vertical segmentation on each sparse point set: all the sparse points in all the image sparse point sets are arranged in ascending or descending order according to their x coordinates in the corresponding image coordinate system, and the difference d between the x coordinates of two adjacent sparse points p k and p k+1 is calculated respectively according to the arrangement order. If d ≤ val, then continue to calculate the difference d of the next adjacent sparse point; if d > val, then take l k =(x k +x k+1 ) / 2 as a segmentation value and record it, and continue to calculate the difference d of the next adjacent sparse point. Here, val is an empirical threshold, usually within 1 pixel. x k and x k+1 respectively represent the x coordinates of the sparse points p k and p k+1 ; until the differences d of all adjacent sparse points in each sparse point set are calculated, and the multiple segmentation values of each sparse point set recorded are used as the corresponding segmentation value set of each sparse point set;

[0049] Since the sparse points in each image are corresponding sparse points, there is also a corresponding relationship between the segmentation values in the segmentation value sets of each sparse point set. The corresponding segmentation values in each sparse point set are compared one by one, and the largest segmentation value is selected as the actual segmentation value, thereby obtaining an actual segmentation value set; the sparse points in each sparse point set are grouped with the segmentation values in the actual segmentation value set to obtain a grouping set, that is, at this time, the sparse point set is divided into several groups according to the x coordinate;

[0050] (5) Perform group alignment on the grouping set obtained in step (4) and perform sparse point matching;

[0051] The group alignment is pairwise alignment, that is, alignment between the groups of two images;

[0052] For the case of horizontal segmentation of each sparse point set, calculate the average value of the y coordinates of all the sparse points in each group in each grouping set as the group description, and compare each group description in one grouping set with each group description in other grouping sets respectively. If the difference < val, then it is considered that the two groups are corresponding groups, and finally the corresponding relationship of the grouping sets under each image is obtained.

[0053] Sort the x - coordinates of the sparse points in each corresponding group from small to large or from large to small. If the points in space have a left - right topological structure, then they also have a left - right topological structure on the image, and their arrangement will not change. And because the co - visible area of the image has been cropped, that is, the image is the image within the co - visible area of each camera. So if the number of sparse points in two corresponding groups is the same, it is considered that the two groups of sparse points are matched, and the sorted sparse points are one - to - one corresponding matching points.

[0054] Since the horizontal field - of - view ranges within the co - visible area of the two cameras may be inconsistent, it may lead to an inconsistent number of sparse points extracted from the images collected by the two cameras in the horizontal direction. If the number of sparse points in two corresponding groups differs by n points, it is necessary to determine whether the image with more sparse points is collected by the left camera or the right camera. If it is collected by the left camera, then remove the n sparse points from left to right in the corresponding group on this image. If it is collected by the right camera, then remove the n sparse points from right to left in the corresponding group on this image.

[0055] After the above operations, one - to - one corresponding matching points within the same group can be obtained, that is, the stereo matching of multiple images collected by the multi - camera is completed. For the case of longitudinally dividing each sparse point set, calculate the average value of the x - coordinates of all sparse points in each group in each grouping set. As a group description, compare each group description in one grouping set with each group description in other grouping sets respectively. If the difference < val, then the two groups are considered corresponding groups, and finally the corresponding relationship of the grouping sets under each image is obtained.

[0056] Since the horizontal field - of - view ranges within the co - visible area of the two cameras may be inconsistent, it may lead to an inconsistent number of sparse points extracted from the images collected by the two cameras in the horizontal direction. So there may be a situation where the number of groups in the two images is different after longitudinal division and grouping.

[0057] If the number of groups obtained by grouping the sparse point sets of the two images differs by m groups, then directly remove the m uncorresponding groups. In the present invention, generally, the extra groups are on one side of a certain image. That is, after determining whether the image with more groups is collected by the left camera or the right camera. If it is collected by the left camera, then remove the m groups from left to right on this image. If it is collected by the right camera, then remove the m groups from right to left in the corresponding group on this image.

[0058] After the above operations, the corresponding groups of the two images can be obtained.

[0059] Since the vertical field - of - view ranges of the two cameras are the same, the number of sparse points extracted from the images collected by the two cameras in the vertical direction must be the same. Then the sparse points in two corresponding groups are matched, and the sorted sparse points are one - to - one corresponding matching points.

[0060] Then, sort the y - coordinates of the sparse points in the corresponding group from small to large or from large to small. The sorted sparse points are the corresponding matching points;

[0061] (6) Verification;

[0062] The obtained matching points can be further verified. That is, the spatial coordinates of the matching points are obtained by triangulation using the internal and external parameters of the corresponding camera and the coordinates of the sparse points in the corresponding image. If the corresponding spatial coordinates meet the set requirements, they are considered correct matching points; otherwise, they are considered incorrect matching points and can be removed. The result is as Figure 6 shown; the specific set requirements are determined according to the actual camera field of view, generally set as the coordinate offset, and the corresponding threshold is set.

[0063] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above - mentioned embodiments. Within the technical concept of the present invention, various equivalent transformations (such as quantity, shape, position, etc.) can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. An image sparse point stereo matching method, characterized in that: Including: Performing stereo rectification on the images collected by each camera of the multi-view camera; Obtaining the common viewing area of each camera; Grouping the sparse points in the sparse point sets obtained by extracting the images of the common viewing area collected by each camera according to the positions of the sparse points in the longitudinal or transverse direction, and aligning the groups in each sparse point set accordingly; Making the sparse points in each group after alignment correspond one by one, specifically as follows: For longitudinal grouping, judge whether the number of groups obtained by grouping the sparse point sets of each image is equal. If they are equal, perform sparse matching on the corresponding groups; if the difference in the number of groups between the two is m groups, directly remove the m non-corresponding groups; Sort the y coordinates of the sparse points in the corresponding group, and the sorted sparse points are the corresponding matching points; For transverse grouping, sort the x coordinates of the sparse points in the corresponding group respectively; if the number of sparse points in the two corresponding groups is the same, it is considered that the two groups of sparse points are matched, and the sorted sparse points are the corresponding matching points; if the difference in the number of sparse points in the two corresponding groups is n points, it is necessary to judge whether the image with more sparse points is collected by the left camera or the right camera. If it is collected by the left camera, remove the n sparse points from left to right in the corresponding group of the image collected by this camera. If it is collected by the right camera, remove the n sparse points from right to left in the corresponding group of the image collected by this camera; Thus, the corresponding matching points in each corresponding group are obtained.

2. The image sparse point stereo matching method according to claim 1, characterized in that: Before grouping, perform denoising operations on each sparse point set, specifically: For each sparse point in the sparse point set of each image, calculate the difference between its y coordinate and the y coordinates of all sparse points in the sparse point sets of other images in their respective corresponding image coordinate systems. If there is a result where the difference is less than the set threshold, it is considered that there are matching points; otherwise, it is considered a noise point and is removed.

3. The sparse point stereo matching method for images according to claim 1 or 2, characterized in that: Grouping according to the positions of the sparse points in the sparse point sets in the longitudinal or transverse direction is specifically as follows: Arrange all the sparse points in each sparse point set in ascending or descending order of any coordinate in the corresponding image coordinate system, and calculate the difference between adjacent two sparse points under the selected coordinate. If the difference is less than the set threshold, continue to calculate the difference between the next adjacent sparse points; if the difference is greater than the set threshold, use the average value of the two sparse points under the selected coordinate as a segmentation value and put it into the segmentation value set of the corresponding image until all differences are calculated; The segmentation values in each segmentation value set correspond to each other to form a corresponding segmentation value group. Take the largest segmentation value in each corresponding segmentation value group as the actual segmentation value of this segmentation value group, so as to obtain an actual segmentation value set. Use the segmentation values of the actual segmentation value set to group each sparse point set to obtain the grouping set of each sparse point set.

4. The method for sparse point stereo matching of images according to claim 3, wherein: The alignment of the groups in each sparse point set is specifically as follows: Calculate the average value of the selected coordinates of all sparse points in each group in each grouping set as the description of the group, and compare the group description of each group in each grouping set with the group descriptions of each group in other grouping sets respectively. If the difference between the two is less than the set threshold, it is considered that the two groups are corresponding groups, and finally obtain the corresponding relationship of the grouping sets under each image.

5. The image sparse point stereo matching method according to claim 1, wherein: Obtaining the common viewing area of each camera is specifically: Align the optical centers of the images collected by each camera through the stereo calibration. Place a target object, and obtain the boundary of the co-visible area when the target object appears in the fields of view of all cameras, so as to obtain the co-visible area of each camera.

6. The image sparse point stereo matching method according to claim 1, wherein: It further includes a verification step: Triangulate according to the internal and external parameters of the corresponding camera and the corresponding sparse point coordinates to obtain the spatial coordinates of the matching points respectively. If the corresponding spatial coordinates of the two meet the set requirements, they are considered correct matching points; otherwise, they are considered incorrect matching points and are eliminated.

7. The image sparse point stereo matching method according to claim 6, wherein: The set requirements are determined according to the actual camera field of view and are specifically set as the coordinate offset threshold.

Images