A Stereo Matching Method for Large Scene Features Based on Two-Dimensional Array Representation

By constructing an extensible combined imaging model based on a two-dimensional array representation and performing optimal rigid transformation, the problem of distortion interference in large-scene stereo matching of fisheye cameras is solved, and the accuracy and quantity of feature matching are improved without losing scene information.

CN117372244BActive Publication Date: 2026-11-10TIANJIN UNIV OF COMMERCE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311455085.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-03
Publication Date
2026-11-10
Estimated Expiration
2043-11-03

AI Technical Summary

Technical Problem

Existing technologies for stereo matching of large scene features using fisheye cameras suffer from the problem that the nonlinear distortion of fisheye images severely interferes with feature extraction and description, and perspective correction methods eliminate the wide field of view advantage of fisheye cameras and lead to the loss of scene information.

Method used

A two-dimensional array-based method is adopted. The stereo fisheye camera is calibrated by a circular marker calibration plate, an extensible combined imaging model is constructed, the azimuth and polar angles are calculated using an equal solid angle projection model, a two-dimensional array model is established and optimal rigidity transformation is performed, simulated transformation primitives are generated, and affine transformation and SIFT algorithm are performed to extract feature points for matching.

Benefits of technology

Without causing image stretching or scene loss, it improves the accuracy and quantity of feature matching, fully utilizes the wide field of view of fisheye cameras, and solves the problem of feature extraction difficulties caused by fisheye image distortion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117372244B_ABST
    Figure CN117372244B_ABST
Patent Text Reader

Abstract

This invention relates to a large-scene feature stereo matching method based on a two-dimensional array representation. Its key features include: parameter calibration of a stereo fisheye camera system to obtain the intrinsic parameters of the stereo fisheye camera; acquisition of fisheye images under different relative camera poses; construction of an extensible combined imaging model based on a spherical imaging model; calculation of the mapping relationship between fisheye image pixels and hemispherical image points; calculation of the mapping relationship between hemispherical image points and geodesic plane points of an icosahedron; PCA dimensionality reduction of the three-dimensional coordinates of each point on the geodesic plane; establishment of a two-dimensional array model and generation of analog transformation primitives using optimal rigidity transformation; performing affine transformation on the analog transformation primitives to obtain an analog image; extraction of feature points; generation of descriptors; and matching of feature points. This invention effectively solves the problem of difficulty in extracting accurate features in the peripheral region due to severe nonlinear distortion, improving the distribution range, quantity, and accuracy of fisheye image feature matching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to feature stereo matching methods, especially a large-scene feature stereo matching method based on two-dimensional array representation. Background Technology

[0002] For a long time, research on stereo matching has primarily focused on perspective images. Pinhole cameras typically have a field of view of only 40° to 60°, and a single shot can only capture very limited local scene information. Pinhole cameras can meet the needs of applications where the target area is limited to a small local area, such as defect detection and facial recognition. However, for applications without a specific observation target, global information is more valuable, such as autonomous driving, forest monitoring, robot autonomous navigation and obstacle avoidance, and pedestrian detection and tracking. Compared to pinhole cameras, fisheye cameras have a wider field of view, generally exceeding 140° or even reaching 270°. The scene information covered by a single fisheye image requires complex stitching of multiple perspective images. Therefore, in large-scale applications, especially in robot navigation and autonomous driving, fisheye cameras have a significant advantage, as their wide field of view allows for the simultaneous visualization of objects in multiple directions.

[0003] In order to take advantage of the wide field of view of fisheye cameras, some feature stereo matching methods based on fisheye images have emerged in recent years. Among them, how to deal with the severe nonlinear distortion of fisheye images is a challenge that each method needs to face. This nonlinear distortion will seriously interfere with the accurate extraction and description of features.

[0004] One of the most direct and commonly used methods is to first correct the fisheye image into a perspective image, and then directly apply the perspective image matching method to the corrected image. The literature “Fiala M, Roth G. Automatic alignment and graph map building of panoramas[C]. IEEE International Workshop on HapticAudio-Visual Environments and their Applications, Ottawa, Canada, 2005: 104-109.” converts large-scene fisheye images into cube surface maps, and then calculates the SIFT features on each cube image face. The literature “Zhang J, Yin X, Luan J, et al. An improved vehicle panoramic imagegeneration algorithm[J]. Multimedia Tools and Applications, 2019, 78(19):27663-27682.” improves the spherical perspective projection algorithm based on the scanline idea and corrects the fisheye image, then extracts SURF features from the corrected image. The paper “Lin Y, Gao F, Qin T, et al. Autonomous aerial navigation using monocular visual‐inertial fusion[J]. Journal of FieldRobotics, 2018, 35(1): 23-51” proposes a fisheye camera UAV navigation system, but only utilizes part of the scene information of the fisheye image: two sub-regions in the horizontal direction are corrected into perspective images for subsequent processing, while most of the field of view information in the vertical direction is discarded.The paper "Miiller MG, Steidle F, Schuster MJ, et al. Robust visual-inertial state estimation with multiple odometries and efficient mapping on an MAV with ultra-wide FOV stereo vision[C]. IEEE International Conference on Intelligent Robots and Systems, Madrid, Spain, 2018: 3701-3708" proposes a visual-inertial odometry and omnidirectional 3D mapping system. To overcome the distortion of fisheye images, the central region of each fisheye image is corrected into two perspective images, while peripheral scene information is discarded. Perspective correction can effectively eliminate nonlinear distortion, but it also fundamentally eliminates the wide field-of-view advantage of fisheye cameras. In addition, objects in the outer region of the fisheye image disk are highly stretched, while objects near the center of the disk are highly compressed.

[0005] Besides perspective correction, some scholars use equal-rectangular images to represent fisheye images. This method can preserve the wide field of view of fisheye cameras and also alleviate the distortion of fisheye images to a certain extent. In order to generate a three-dimensional scene surround view, the literature "Lo I, Shih K, Chen H H. Image stitching for dual fisheye cameras[C]. IEEE International Conference on Image Processing, Athens, Greece, 2018: 3164-3168." corrects the overlapping field of view images of two fisheye cameras into equal-rectangular images, then extracts features and completes the entire view stitching process. The literature "Zhao Q, Feng W, Wan L, et al. SPHORB: Afast and robust binary feature on the sphere[J]. International Journal of Computer Vision, 2015, 113(2): 143-159." improves the traditional ORB algorithm, constructs new binary features for equal-rectangular images in spherical space and proposes a spherical ORB algorithm. The paper "Shan Y, Li S. Descriptor matching for a discrete spherical image with a convolutional neural network[J]. IEEEAccess, 2018, 6: 20748-20755." converts fisheye images into equal-rectangular images, uses an accelerated segmentation feature testing algorithm to extract feature points, and describes the features using a convolutional neural network. The equal-rectangular image is obtained by unfolding the spherical image based on a spherical model. This process introduces secondary distortion, especially around the extreme points of the sphere, causing severe image stretching and deformation, leading to degraded descriptor performance and serious mismatch problems. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a large-scene feature stereo matching method based on two-dimensional array representation. It makes full use of the wide field of view advantage of fisheye cameras and solves the feature stereo matching problem of large field of view fisheye images without producing obvious image stretching or scene loss.

[0007] The present invention solves the existing technical problems by adopting the following technical solution: A method for stereo matching of large-scene features based on two-dimensional array representation includes the following steps: Step 1: Use a circular calibration plate to calibrate the parameters of the stereo fisheye camera system, obtain the intrinsic parameters of the stereo fisheye camera, and acquire fisheye images under different relative camera poses. Step 2: Using the spherical imaging model as the basic imaging model, embed the unit regular icosahedron into the unit sphere to construct an extendable and combinable imaging model; Step 3: Based on the extensible combined imaging model, derive the azimuth angle using the formula of the equal solid angle projection model. With polar angle The calculation formula is used to obtain the mapping relationship between pixels in the fisheye image and points in the hemispherical image; Step 4: Calculate the mapping relationship between the points in the hemispherical image and each vertex of the geodesic plane of the icosahedron; Step 5: Calculate the one-to-one correspondence between hemispherical image points, fisheye image pixels, and each geodesic plane point using the positioning formula, and then perform PCA dimensionality reduction on the three-dimensional coordinates of each point on the geodesic plane. Step 6: Establish a two-dimensional array model and locate the array primitives. Based on the directional array representation, perform differentiated local representation according to the position information of each array primitive to generate analog transformation primitives. Step 7: Perform affine transformation on the analog transformation primitives to obtain an analog image. Use the SIFT algorithm to extract feature points on the analog image, generate descriptors, match the feature points, and then merge all matching point pairs into one group and remove duplicate matching point pairs. The specific implementation method of step 6 is as follows: Step 6.1: Select the reference geodesic plane, take one vertex of the reference geodesic plane as the origin of the coordinate system of the two-dimensional array model, and unfold the regular icosahedron model into a two-dimensional array model. Step 6.2: Calculate the optimal rigid transformation matrix based on the two-dimensional array model; Step 6.3: Establish the simulation transformation primitives using the optimal rigid transformation matrix; The implementation method of step 6.2 is as follows: After PCA dimensionality reduction, the coordinates of three-dimensional points on the geodesic plane are given. The three vertices are respectively , , , The corresponding array primitive is ,set up The three vertices are respectively , , In order to Convert to array primitives Calculate the optimal rigid transformation matrix using the following steps: First, [the matrix is ​​then...] , This can be expressed as follows:

[0008] set up , They are respectively , The center of mass, then , The coordinates are obtained by the following formula:

[0009]

[0010] To find the optimal rotation matrix, it is necessary to... , Redirect the coordinate data of all points and make the geodesic plane and array primitives The centroids of all elements are located at the origin, and the relocation matrix... Calculated using the following formula:

[0011] The matrix is ​​processed by the following formula. Perform singular value decomposition and calculate the optimal rotation matrix. :

[0012]

[0013] Using the optimal rotation matrix The optimal translation matrix is ​​calculated using the following formula based on the centroid coordinates. : ; Step 6.3: Establish the simulation transformation primitives using the optimal rigid transformation matrix; By representing the geodesic plane and its neighboring geodesic planes using a two-dimensional array, the corresponding analog transformation primitive array model is obtained; after PCA dimensionality reduction, the geodesic plane... All points , , … , The coordinates can form dimensional matrix :

[0014] in, ; Based on the analog transformation primitive array model, the geodesic plane Optimal rigid transformation is performed to obtain array elements. , The set of coordinates of all points Determined by the following formula:

[0015] Among them, matrix [ , , … , ]and equal.

[0016] Furthermore, the spatial point imaging process of the extended composite imaging model is as follows: Define For the camera coordinate system, spatial points Imaging on the image plane is achieved through the following two steps: First, spatial points... Along the vector Linear projection onto the imaging model and obtaining the projection points and At this time, along Vector in the positive direction of the axis with vector Forming polar angle along Vector in the positive direction of the axis and exist The projection vector on the plane forms the azimuth angle. Then, the projection point and Image points are obtained by nonlinearly projecting them onto the image plane. .

[0017] Furthermore, the specific implementation method of step 3 is as follows: Step 3.1: Based on the intrinsic parameters of the stereo fisheye camera, use an equal solid angle projection model. Define image point To the image center distance With polar angle The relationship between them The focal length of the camera; Step 3.2: Calculate the points on the hemispherical surface. corresponding azimuth angle With polar angle ; Step 3.3: Using azimuth angle With polar angle Calculate image pixels Coordinates and hemispherical image points The coordinates.

[0018] Furthermore, the calculation formula used in step 3.2 is as follows:

[0019]

[0020] in, For incident - X horn, For incident - Y horn; Step 3.3 calculates the image pixels using the following formula. Coordinates:

[0021] Step 3.3 uses the following formula to calculate the hemispherical image points. Coordinates:

[0022] in, Let be the radius of the hemisphere in the imaging model.

[0023] Furthermore, the specific implementation method of step 4 is as follows: Step 4.1, using the projection point Geodesic plane The three sides , , Calculate Plücker coordinates as a directed line segment; Step 4.2, for directed line segments and geodesic plane Three directed line segments , and Execute edge operation operators; Step 4.3: Determine the projection point The geodesic plane in which it is located.

[0024] Furthermore, the calculation method for step 4.1 is as follows: Let the projection point be The three vertices of the geodesic plane are respectively , , Then three directed line segments , and Plücker coordinates , and Calculate using the following formulas respectively:

[0025]

[0026]

[0027] The implementation method of step 4.2 is as follows: Let the origin of the imaging model camera coordinate system be... Directed line segment The Plücker coordinates are The edge operation operator is used to calculate the directed line segment. and , as well as Relationship , and :

[0028]

[0029]

[0030] The determination method in step 4.3 is as follows: if In the geodesic plane If the above occurs, the following three situations will exist: (1) Pass The three vertices must satisfy the following conditions: ; (2) Pass The three edges must satisfy the following conditions: ; (3) Pass Inside, the following conditions must be met: .

[0031] Furthermore, the specific implementation method of step 5 is as follows: Step 5.1: Calculate the geodesic plane normalized normal vector ; Step 5.2: Calculate the projection points 3D coordinates; Step 5.3: Using the fisheye image, the mapping relationship between the hemisphere and each geodesic plane point, perform PCA dimensionality reduction on the three-dimensional coordinates of the geodesic plane points.

[0032] Furthermore, step 5.1 uses the following formula to calculate the geodesic plane. normalized normal vector :

[0033]

[0034] ; The calculation method for step 5.2 is as follows: using the geodesic plane vertex Normalized normal vector and corresponding hemispherical points The projection point is obtained by the following formula. The three coordinate values , , :

[0035]

[0036] .

[0037] The advantages and positive effects of this invention are: 1. This invention establishes an extensible combined imaging model based on the nonlinear imaging method of fisheye cameras. Compared with the method of expanding fisheye images into equal rectangles based on traditional single spherical models, this invention uses an extensible model to expand fisheye images, which will not produce obvious image stretching and deformation problems, and is more conducive to improving feature matching accuracy.

[0038] 2. The present invention represents fisheye images in a two-dimensional array. Compared with perspective correction methods, the present invention does not produce scene loss problems. Whether in the central area of ​​the fisheye image disk with small distortion or the peripheral area with severe distortion, more widely distributed features can be extracted, which can make full use of the large scene advantage of fisheye cameras.

[0039] 3. This invention constructs a simulated transformation primitive through optimal rigid transformation, which can effectively solve the problem of not being able to effectively describe the features near the jagged edges due to insufficient neighborhood information. It performs affine transformation on the simulated transformation primitive and performs SIFT feature matching and removes duplicate feature matching on the simulated image after each parameter transformation. Compared with the existing fisheye image feature matching method, this invention can obtain a rich number of feature matching results, even up to thousands of pairs.

[0040] 4. The present invention is reasonably designed. It establishes an extensible combined imaging model and a two-dimensional array model, which reduces fisheye image distortion without causing significant scene loss and image stretching. It can effectively solve the problem of difficulty in extracting accurate features in the peripheral area due to severe nonlinear distortion, and improve the distribution range of feature matching. It constructs simulated transformation primitives and calculates feature matching on the simulated image after affine transformation, which can effectively improve the number of feature matches and the matching accuracy. Attached Figure Description

[0041] Figure 1 This is a flowchart of the large-scene feature stereo matching method based on two-dimensional array representation of the present invention; Figure 2 A schematic diagram of the extendable combined imaging model established in this invention; Figure 3 A schematic diagram of the two-dimensional array model established for this invention; Figure 4 This is a schematic diagram of the analog transformation primitive array model established in this invention; Figure 5 This is a schematic diagram of the matching results in the first scenario using the present invention; Figure 6 This is a schematic diagram illustrating the matching results of the present invention in the second scenario; Figure 7 This is a schematic diagram of the matching results in the third scenario using the present invention. Detailed Implementation

[0042] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] A method for stereo matching of large scene features based on two-dimensional array representation, such as Figure 1 As shown, it includes the following steps: Step 1: Use a circular calibration plate to calibrate the parameters of the stereo fisheye camera system and obtain the intrinsic parameters of the stereo fisheye camera. Then, acquire fisheye images under different relative camera poses.

[0044] Step 2: Establish an extendable combined imaging model based on the spherical imaging model.

[0045] In this step, a unit regular icosahedron is embedded within a unit sphere to construct an extendable composite imaging model based on the spherical imaging model, and the imaging model is defined. Since this invention is for fisheye cameras with a field of view of no more than 180°, the final extendable composite imaging model consists of half a unit regular icosahedron embedded within a hemisphere. Simultaneously, the spatial point imaging process of the extendable composite imaging model needs to be defined, such as... Figure 2 As shown: Definition This is the camera coordinate system. Spatial points. Imaging on the image plane is achieved through two steps: First along the vector Linear projection onto the imaging model and obtaining the projection points and At this time, along Vector in the positive direction of the axis with vector Forming polar angle along Vector in the positive direction of the axis and exist The projection vector on the plane forms the azimuth angle. Then click and Pixels are obtained by non-linearly projecting them onto the image plane. .

[0046] Step 3: Based on the extensible combined imaging model, derive the azimuth angle using the formula of the equal solid angle projection model. With polar angle The calculation formula is used to obtain the mapping relationship between fisheye image pixels and hemispherical image points.

[0047] In this embodiment, this step includes the following steps: Step 3.1: Based on the camera parameters, use an equal solid angle projection model. Define pixel To the image center distance With polar angle The relationship between them This refers to the camera's focal length.

[0048] Step 3.2: Calculate the points on the hemispherical surface. corresponding azimuth angle With polar angle The formula is as follows:

[0049]

[0050] in, For incident - X horn, For incident - Y horn.

[0051] Step 3.3: Using azimuth angle With polar angle Calculate the points of the hemispherical image respectively and image pixels The coordinates of, where, , The formula is as follows:

[0052]

[0053] in, Let be the radius of the hemisphere in the imaging model.

[0054] Step 4: Calculate the mapping relationship between the points in the hemispherical image and the vertices of each geodesic plane of the icosahedron.

[0055] In this embodiment, this step includes the following steps: Step 4.1, using the projection point Geodesic plane The three sides , , Calculate Plücker coordinates as a directed line segment.

[0056] Setting points The three vertices of the geodesic plane are respectively 、 、 Then three directed line segments , and Plücker coordinates , and Calculate using the following formulas respectively:

[0057]

[0058]

[0059] Step 4.2, for directed line segments and geodesic plane Three directed line segments , and Perform edge operation operators.

[0060] The origin of the camera coordinate system in the imaging model is Directed line segment Plücker coordinates The edge operation operator is used to calculate the directed line segment. and , as well as Relationship , and :

[0061]

[0062]

[0063] Step 4.3, Determine the point The geodesic plane in which it is located.

[0064] if In the geodesic plane If the above occurs, the following three situations will exist: 1) Pass The three vertices must satisfy the following conditions:

[0065]

[0066] 2) Pass The three edges must satisfy the following conditions:

[0067]

[0068]

[0069] 3) Pass Inside, the following conditions must be met:

[0070] .

[0071] Step 5: Calculate the one-to-one correspondence between hemispherical image points, fisheye image pixels, and each geodesic plane point using the positioning formula, and then perform PCA dimensionality reduction on the three-dimensional coordinates of each point on the geodesic plane.

[0072] In this embodiment, this step includes the following steps: Step 5.1: Calculate the geodesic plane normalized normal vector .

[0073] Geodesic plane normalized normal vector It can be obtained using the following formula:

[0074]

[0075]

[0076] Step 5.2, Calculation The three-dimensional coordinates.

[0077] Using geodesic plane vertex Normalized normal vector and corresponding hemispherical points The projection point is obtained by the following formula. The three coordinate values , , :

[0078]

[0079]

[0080] Step 5.3: Using the fisheye image and the mapping relationship between the points on the geodesic plane of each hemisphere and polyhedron, perform PCA dimensionality reduction on the three-dimensional coordinates of the points on the geodesic plane.

[0081] Step 6: Establish a two-dimensional array model and locate the array primitives. Based on the directional array representation, perform differentiated local representation according to the position information of each array primitive to generate analog transformation primitives.

[0082] In this embodiment, this step includes the following steps: Step 6.1: Select the reference geodesic plane, and use one vertex of the reference geodesic plane as the origin of the coordinate system of the two-dimensional array model to unfold the regular icosahedron model into a two-dimensional array model.

[0083] Assuming geodesic plane 1 is selected as the reference geodesic plane, a two-dimensional array model is established, such as... Figure 3 As shown, a vertex of the reference geodesic plane is taken as the origin of the coordinate system of the two-dimensional array model. The geodesic planes 1, 2, 3, 4, 5, 6, ..., 17, 18, 19, 20 in the regular icosahedron correspond to the array primitives 1', 2', 3', 4', 5', 6', ..., 17', 18', 19', 20' in the two-dimensional array model.

[0084] Step 6.2: Calculate the optimal rigid transformation matrix based on the two-dimensional array model.

[0085]

[0086]

[0087] After PCA dimensionality reduction, the coordinates of three-dimensional points on the geodesic plane are assumed to be... The three vertices are respectively , , The corresponding array primitive is ,set up The three vertices are respectively , , In order to Convert to array primitives The optimal rigid transformation matrix is ​​calculated using the following steps. First, [the matrix is ​​then...]. , This can be expressed as follows:

[0088]

[0089] set up , They are respectively , The center of mass, then , The coordinates can be obtained using the following formula:

[0090]

[0091] To find the optimal rotation matrix, it is necessary to... , Redirect all point coordinate data and make and The centroids of all elements are located at the origin, and the relocation matrix... It can be calculated using the following formula:

[0092] For matrix Perform singular value decomposition and calculate the optimal rotation matrix. :

[0093]

[0094] Using the optimal rotation matrix The optimal translation matrix is ​​calculated using the following formula based on the centroid coordinates. :

[0095] Step 6.3: Establish the simulation transformation primitives using the optimal rigid transformation matrix.

[0096] Jagged edges can occur during the two-dimensional array representation process. When calculating feature points near these jagged edges, features may be ignored or incorrectly extracted due to a lack of neighborhood information. To address this issue, this invention establishes analog transformation primitives based on a two-dimensional array model. Taking geodesic plane 1 as an example, by performing a two-dimensional array representation on geodesic plane 1 and its neighboring geodesic planes 2, 5, and 17, the corresponding analog transformation primitive array model is obtained, as follows: Figure 4 As shown.

[0097] After PCA dimensionality reduction All points , , … , The coordinates can be formed dimensional matrix :

[0098] Based on the analog transformation primitive array model, Optimal rigid transformation is performed to obtain array elements. , The set of coordinates of all points is determined by the following formula:

[0099] Among them, matrix , , … , and equal.

[0100] Using the same method, dimensionality reduction and optimal rigid transformation are performed on geodesic planes 2, 5, and 17, thereby establishing the analog transformation primitive for geodesic plane 1. Similarly, analog transformation primitives for other geodesic planes can be established.

[0101] Step 7: Perform an affine transformation on the analog transformation primitives to obtain an analog image. Use the SIFT algorithm to extract feature points on the analog image, generate descriptors, match the feature points, and then merge all matching point pairs into one group and remove duplicate matching point pairs.

[0102] The specific processing procedure for this step consists of two steps: Step 7.1: Perform an affine transformation on the analog transformation primitives: by changing two parameters that determine the camera axis orientation: latitude angle. and longitude angle Affine transformations are performed on the analog transformation primitives from multiple different viewpoints to obtain analog images. In this step, affine transformations are performed on the analog transformation primitives using the affine distortions caused by changes in the optical axis direction of the simulated camera. These affine distortions depend on the latitude angle. and longitude angle Different values ​​of . First, perform an angle transformation on the analog transformation primitive. Rotation, followed by absolute tilt value The tilt. Absolute tilt value. sampling interval Then the absolute tilt value The possible values ​​are as follows Accordingly, the latitude angle can be calculated. The possible values ​​of: Longitude angle sampling interval For each absolute tilt value The longitude angle is taken as .

[0103] Step 7.2: Execute the SIFT matching algorithm: Since the distortion in each analog transform primitive of the fisheye image is basically the same, it can be approximated as affine distortion. After performing an affine transformation on the analog transform primitive, a series of analog images are obtained. The SIFT algorithm is used on the analog images to extract feature points, generate descriptors, and match the feature points. Finally, all matching pairs are merged into one group, and duplicate point pairs are removed. The final matching result is as follows: Figure 5 , 6 As shown in Figure 7. Figure 5 , 6In the stereo fisheye images of the three scenarios shown in Figures 7, each scenario contains four pairs of fisheye images acquired using different camera poses. The two columns on the left are the original stereo fisheye images, and the two columns on the right are the matching results obtained by the stereo matching method for large scene features based on two-dimensional array representation provided by this invention. It can be observed from the figure that, regardless of whether it is in the central region with less distortion or the peripheral region with severe distortion, this invention can obtain a large number of accurate feature matching points.

[0104] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A method for stereo matching of large-scene features based on two-dimensional array representation, characterized in that: Includes the following steps: Step 1: Use a circular calibration plate to calibrate the parameters of the stereo fisheye camera system, obtain the intrinsic parameters of the stereo fisheye camera, and acquire fisheye images under different relative camera poses. Step 2: Using the spherical imaging model as the basic imaging model, embed the unit regular icosahedron into the unit sphere to construct an extendable and combinable imaging model; Step 3: Based on the extensible combined imaging model, derive the azimuth angle using the formula of the equal solid angle projection model. With polar angle The calculation formula is used to obtain the mapping relationship between pixels in the fisheye image and points in the hemispherical image; Step 4: Calculate the mapping relationship between the points in the hemispherical image and each vertex of the geodesic plane of the icosahedron; Step 5: Calculate the one-to-one correspondence between hemispherical image points, fisheye image pixels, and each geodesic plane point using the positioning formula, and then perform PCA dimensionality reduction on the three-dimensional coordinates of each point on the geodesic plane. Step 6: Establish a two-dimensional array model and locate the array primitives. Based on the directional array representation, perform differentiated local representation according to the position information of each array primitive to generate analog transformation primitives. Step 7: Perform affine transformation on the analog transformation primitives to obtain an analog image. Use the SIFT algorithm to extract feature points on the analog image, generate descriptors, match the feature points, and then merge all matching point pairs into one group and remove duplicate matching point pairs. The specific implementation method of step 6 is as follows: Step 6.1: Select the reference geodesic plane, take one vertex of the reference geodesic plane as the origin of the coordinate system of the two-dimensional array model, and unfold the regular icosahedron model into a two-dimensional array model. Step 6.2: Calculate the optimal rigid transformation matrix based on the two-dimensional array model; Step 6.3: Establish the simulation transformation primitives using the optimal rigid transformation matrix; The implementation method of step 6.2 is as follows: After PCA dimensionality reduction, the coordinates of three-dimensional points on the geodesic plane are given. The three vertices are respectively , , , The corresponding array primitive is ,set up The three vertices are respectively , , In order to Convert to array primitives Calculate the optimal rigid transformation matrix using the following steps: First, [the matrix is ​​then...] , This can be expressed as follows: set up , They are respectively , The center of mass, then , The coordinates are obtained by the following formula: To find the optimal rotation matrix, it is necessary to... , Redirect the coordinate data of all points and make the geodesic plane and array primitives The centroids of all elements are located at the origin, and the relocation matrix... Calculated using the following formula: The matrix is ​​processed by the following formula. Perform singular value decomposition and calculate the optimal rotation matrix. : Using the optimal rotation matrix The optimal translation matrix is ​​calculated using the following formula based on the centroid coordinates. : ; Step 6.3: Establish the simulation transformation primitives using the optimal rigid transformation matrix; By representing the geodesic plane and its neighboring geodesic planes using a two-dimensional array, the corresponding analog transformation primitive array model is obtained; after PCA dimensionality reduction, the geodesic plane... All points , , … , The coordinates can form dimensional matrix : in, ; Based on the analog transformation primitive array model, the geodesic plane Optimal rigid transformation is performed to obtain array elements. , The set of coordinates of all points Determined by the following formula: Among them, matrix [ , , ,…, , ]and equal.

2. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 1, characterized in that: The spatial point imaging process of the extended combined imaging model is as follows: Define For the camera coordinate system, spatial points Imaging on the image plane is achieved through the following two steps: First, spatial points... Along the vector Linear projection onto the imaging model and obtaining the projection points and At this time, along Vector in the positive direction of the axis with vector Forming polar angle along Vector in the positive direction of the axis and exist The projection vector on the plane forms the azimuth angle. Then, the projection point and Image points are obtained by nonlinearly projecting them onto the image plane. .

3. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 1, characterized in that: The specific implementation method of step 3 is as follows: Step 3.1: Based on the intrinsic parameters of the stereo fisheye camera, use an equal solid angle projection model. Define image point To the image center distance With polar angle The relationship between them The focal length of the camera; Step 3.2: Calculate the points on the hemispherical surface. corresponding azimuth angle With polar angle ; Step 3.3: Using azimuth angle With polar angle Calculate image pixels Coordinates and hemispherical image points The coordinates.

4. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 3, characterized in that: The calculation formula used in step 3.2 is as follows: in, For incident - X horn, For incident - Y horn; Step 3.3 calculates the image pixels using the following formula. Coordinates: Step 3.3 uses the following formula to calculate the hemispherical image points. Coordinates: in, Let be the radius of the hemisphere in the imaging model.

5. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 1, characterized in that: The specific implementation method of step 4 is as follows: Step 4.1, using the projection point Geodesic plane The three sides , , Calculate Plücker coordinates as a directed line segment; Step 4.2, for directed line segments and geodesic plane Three directed line segments , and Execute edge operation operators; Step 4.3: Determine the projection point The geodesic plane in which it is located.

6. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 5, characterized in that: The calculation method for step 4.1 is as follows: Let the projection point be The three vertices of the geodesic plane are respectively , , Then three directed line segments , and Plücker coordinates , and Calculate using the following formulas respectively: The implementation method of step 4.2 is as follows: Let the origin of the imaging model camera coordinate system be... Directed line segment The Plücker coordinates are The edge operation operator is used to calculate the directed line segment. and , as well as Relationship , and : The determination method in step 4.3 is as follows: if In the geodesic plane If the above occurs, the following three situations will exist: (1) Pass The three vertices must satisfy the following conditions: ; (2) Pass The three edges must satisfy the following conditions: ; (3) Pass Inside this, the following conditions must be met: 。 7. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 1, characterized in that: The specific implementation method of step 5 is as follows: Step 5.1: Calculate the geodesic plane normalized normal vector ; Step 5.2: Calculate the projection points 3D coordinates; Step 5.3: Using the fisheye image, the mapping relationship between the hemisphere and each geodesic plane point, perform PCA dimensionality reduction on the three-dimensional coordinates of the geodesic plane points.

8. The large-scene feature stereo matching method based on two-dimensional array representation according to claim 7, characterized in that: Step 5.1 uses the following formula to calculate the geodesic plane. normalized normal vector : ; The calculation method for step 5.2 is as follows: using the geodesic plane vertex Normalized normal vector and corresponding hemispherical points The projection point is obtained by the following formula. The three coordinate values , , : 。