A spherical panoramic image feature matching method, a storage medium and an electronic device

Abstract

This invention discloses a method for feature matching of spherical panoramic images, comprising: acquiring spherical panoramic images in ERP format; extracting features from the ERP format spherical panoramic images and generating a first feature descriptor; performing local image patch geometric correction on the ERP format spherical panoramic images using image patch reprojection; generating a second feature descriptor with the same dimension as the first feature descriptor using a CNN based on the corrected image patches; performing initial feature matching on two spherical panoramic images based on the second feature descriptor; and optimizing the initial feature matching using coplanar geometric constraints. This invention designs and implements a reliable spherical image feature matching algorithm by combining local geometric correction with descriptors learned by CNN.

Description

Technical Field

[0001] This invention relates to the fields of photogrammetry and computer vision, and particularly to a method for feature matching of spherical panoramic images, as well as a storage medium and electronic device. Background Technology

[0002] Urban streets and indoor environments have become crucial components of current real-scene 3D reconstruction. Spherical images, capable of capturing a complete scene with a single camera exposure, have become a vital data source for 3D reconstruction. Feature matching is a prerequisite for image-based 3D reconstruction. However, most existing algorithms are designed for perspective images, which differ from spherical images in camera imaging models. Therefore, a more reliable feature matching method is urgently needed to address the feature matching problem of spherical panoramic images.

[0003] Spherical panoramic cameras, also known as 360 cameras, can cover a 360°×180° field of view from the acquisition point, acquiring images with an all-around field of view. They have been widely used in security monitoring, street view mapping, robot navigation, and other fields. Compared with planar perspective cameras, spherical panoramic cameras have a larger field of view, enabling continuous view tracking in complex scenes and reducing data acquisition workload. Therefore, they are receiving increasing attention in the 3D reconstruction of urban streets and indoor environments. Existing methods include improvements to individual steps of panoramic image processing; some researchers have also attempted to use panoramic images for 3D reconstruction. However, current research mainly focuses on improving professional-grade panoramic cameras or integrating them with SLAM (Simultaneous Localization and Mapping) systems, aiming to meet the navigation and localization needs of robots. With the popularization of spherical panoramic cameras, such as consumer-grade Insta360 and Ricoh THETA, and the development of image processing technology, the demand for 3D reconstruction based on spherical panoramic images will increase significantly, while existing technical solutions cannot yet meet the needs of feature matching based on spherical panoramic images. Summary of the Invention

[0004] To address the above problems, this invention provides a spherical panoramic image feature matching method, comprising the following steps:

[0005] S1. Acquire a spherical panoramic image, and process the spherical panoramic image using equidistant cylindrical projection to obtain a spherical panoramic image in ERP format.

[0006] S2. Extract features from the spherical panoramic image in ERP format and generate a first feature descriptor;

[0007] S3. Use image block reprojection to perform local image block geometric correction on the spherical panoramic image in ERP format to obtain corrected image blocks;

[0008] S4. Based on the corrected image block, generate a second feature descriptor with the same dimension as the first feature descriptor;

[0009] S5. Perform the same processing as steps S1-S4 on the two spherical panoramic images to be matched. Based on the second feature descriptor, perform conventional feature matching on the two spherical panoramic images to obtain the initial feature matching of the two spherical panoramic images.

[0010] S6. Optimize the initial feature matching using coplanar geometric constraints. Coplanar geometric constraints require that the baseline vector connecting the centers of the two spherical panoramic images and the observation vector from the centers of the two spherical panoramic images to the object-side 3D point be coplanar. Two feature points that satisfy the coplanar geometric constraints are correctly matched to obtain the final feature matching result.

[0011] Furthermore, in step S2, the SIFT algorithm is used for feature extraction.

[0012] Furthermore, step S3 specifically includes:

[0013] S31. Select feature points as key points in the spherical panoramic image in ERP format.

[0014] S32. Convert the coordinates of the key points into latitude and longitude coordinates in a spherical coordinate system;

[0015] S33. Using the vector from the origin of the spherical coordinate system to the key point of the spherical surface as the normal vector of the tangent plane passing through the key point of the spherical surface, the tangent plane is obtained;

[0016] S34. Based on imaging geometry, project the local image blocks around key points in the spherical image of ERP format onto the tangent plane to obtain the corrected image blocks.

[0017] Furthermore, in step S34, the corrected image block is generated through inverse correction, specifically as follows:

[0018] S341. Calculate the pixel coordinates p of the image points in the corrected image block = (p x ,p y The homogeneous coordinates p corresponding to ) h =(p xh ,p yh ,1):

[0019]

[0020] Among them, (c xp ,c yp f is the principal point coordinate of the corrected image. p To correct the camera focal length corresponding to the image;

[0021] S342, change the homogeneous coordinates ph =(p xh ,p yh 1) Project the image block onto the unit spherical rectangular coordinate system O-X'Y'Z' corresponding to the local Cartesian coordinate system of the corrected image block to obtain the spherical point p. ls :

[0022]

[0023] Among them, ||p h ‖ represents point p h The distance to the origin O of the unit sphere rectangular coordinate system O-X'Y'Z';

[0024] S343, By transforming the matrix spherical point p ls Transform the local Cartesian coordinate system of the corrected image patch into the unit spherical rectangular coordinate system O-X'Y'Z', corresponding to the global spherical rectangular coordinate system O-XYZ, to obtain p s :

[0025] p s =R*p ls

[0026] Where R represents the rotation matrix, R y (θ) represents the rotation angle θ around the Y-axis. Indicates the rotation angle around the X-axis R z (ori) represents the rotation angle ori around the Z-axis;

[0027] S344, p s Image points in a spherical panoramic image converted to ERP format are then converted to grayscale values ​​and corrected to generate pixel coordinates p = (p... x ,p y The grayscale value of ) is used to obtain the corrected image block.

[0028] Further, in step S4, based on the corrected image patch, a second feature descriptor with the same dimension as the first feature descriptor is generated using a pre-trained HardNet network; wherein, the pre-training adopts a hard negative sampling strategy and a triplet boundary loss function to train the HardNet network.

[0029] Furthermore, in step S6, the relative orientation of the two spherical panoramic images to be matched is represented by relative rotation R and translation T, using the essential matrix E = [T]. × R encodes the relative orientation of two spherical panoramic images, [T] × The antisymmetric matrix of T is represented by the following formula:

[0030]

[0031] Where p1 and p2 are the spherical coordinates of the initial matching feature points on the two spherical panoramic images to be matched, respectively. This represents the transpose of p2;

[0032] Furthermore, in step S6, the hypothesis testing framework based on RANSAC achieves robust estimation of the essential matrix E. During the RANSAC iteration process, error metrics and error thresholds are used to determine correct and incorrect feature matches.

[0033]

[0034]

[0035] Where e is the spherical angle error measure, e p e is the error threshold in pixels. a The error threshold for the spherical angle is defined by abs(·), where abs(·) represents the absolute value and max(W,H) represents the pixel size of the long side of the spherical panoramic image in ERP format.

[0036] When e <e a At that time, the two feature points that were initially matched were the correct feature matches.

[0037] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described spherical panoramic image feature matching method.

[0038] The present invention also proposes an electronic device, including a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to execute the above-described spherical panoramic image feature matching method.

[0039] The beneficial effects of the technical solution provided by this invention are:

[0040] This invention proposes a reliable feature matching algorithm for spherical panoramic images. First, features are extracted from ERP format spherical panoramic images. Then, image patch reprojection is used to perform local image patch geometric correction on the ERP format spherical panoramic images, resulting in corrected image patches. Based on the corrected image patches, feature descriptors for feature matching are generated. Conventional feature matching is performed on two spherical panoramic images to obtain initial feature matching. Finally, coplanar geometric constraints are used to optimize the initial feature matching. The solution of this invention is applicable to feature matching of spherical panoramic images and incremental SFM (Structure from Motion). Attached Figure Description

[0041] Figure 1 This is a flowchart of a spherical panoramic image feature matching method according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the conversion between spherical panoramic images and equidistant cylindrical projection images according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating local image block geometric correction of an ERP format spherical panoramic image according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the coplanar geometric constraint principle of an embodiment of the present invention;

[0045] Figure 5 This is a block diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0047] A flowchart of a spherical panoramic image feature matching method according to an embodiment of the present invention is shown below. Figure 1 Specifically, it includes the following steps:

[0048] S1. Obtain the spherical panoramic image and process it using equirectangular projection (ERP) to obtain the spherical panoramic image in ERP format.

[0049] refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the conversion between a spherical panoramic image and an equidistant cylindrical projection image according to an embodiment of the present invention. Figure 2 In (a), P is a three-dimensional point on the object side, p is the spherical projection point of P, the coordinates of p in the spherical rectangular coordinate system O-XYZ are p(x,y,z), and the coordinates of p in the spherical latitude and longitude coordinate system are p(x,y,z). The coordinates in are The transformation relationship between the two coordinate systems is as follows:

[0050]

[0051] In the spherical rectangular coordinate system O-XYZ, the Y-axis points vertically downwards, and the Z-axis points to the principal point of the two-dimensional image; after... Figure 2 In (a), the plane of the north and south pole axes of the sphere and the projection point p intersects the equatorial plane at line l. The longitude θ and latitude of the projection point p are... Defined as the angle between the intersection line l and the Z-axis and OP respectively, with the sphere radius set to 1.

[0052] Using equidistant cylindrical projection, the spherical panoramic image is converted into a two-dimensional image, with spherical latitude and longitude coordinates. and the planar coordinates p(I) of the two-dimensional image x ,I y The conversion relationship is as follows:

[0053]

[0054] Among them, (c x ,c y ) represents the principal point coordinates of the two-dimensional image, and W and H represent the width and height of the two-dimensional image, respectively.

[0055] S2. Extract features from the spherical panoramic image in ERP format and generate a first feature descriptor.

[0056] This invention utilizes the SIFT (Scale Invariant Feature Transform) algorithm for feature extraction and description of spherical panoramic images. On one hand, most data acquisition uses fixed camera roll and pitch angles, such as in ground-based mobile surveying vehicle systems, ensuring consistent geometric structure of ERP images near the spherical equator. On the other hand, the classic SIFT algorithm exhibits excellent scale and rotation invariance, strong resistance to viewpoint changes, and has corresponding high-performance open-source libraries. Specifically, this invention utilizes SIFTGPU, the GPU (Graphics Processing Unit) version of the SIFT algorithm, for image feature extraction and descriptor generation. This technique is commonly used in digital photogrammetry and computer vision, and will not be elaborated upon here.

[0057] S3. Use image block reprojection to perform local image block geometric correction on the spherical panoramic image in ERP format to obtain corrected image blocks.

[0058] ERP format spherical panoramic images exhibit significant geometric distortion near the poles of the sphere, reducing the repeatability and distinguishability of extracted features. This invention utilizes image patch reprojection to achieve local geometric correction and mitigate geometric distortion, specifically:

[0059] S31. Select feature points as key points from the feature points extracted from the spherical panoramic image in ERP format, referring to... Figure 3 In (a), the coordinates of the key point are I(I) x ,I y );

[0060] S32, Reference Figure 3 In (b), the key point coordinates I(I)x ,I y Convert latitude and longitude coordinates to spherical coordinate system

[0061] S33. Using the vector from the origin of the spherical coordinate system to the key point of the spherical surface as the normal vector of the tangent plane passing through the key point of the spherical surface, the tangent plane is obtained;

[0062] S34. Based on imaging geometry, project the local image blocks around key points in the spherical image of ERP format onto the tangent plane to obtain the corrected image blocks.

[0063] To ensure the scale and rotation invariance of the descriptor, the scale parameter (scale) and orientation parameter (ori) must be determined to define the image patch during the reprojection process described above. In SIFT-based feature detection, a feature point f can be represented as f = (I x ,I y ,scale,ori), where (I x ,I y ) represents the pixel coordinates of the feature point, and scale and ori are the scale and orientation parameters of the feature point, respectively. Assume that at the original image scale, the width and height of the corrected image patch are W... p and H p The image patch size W of feature point p sp and H sp The following formula can be used to calculate, where S R S represents the scale ratio between the pyramid level containing feature point p and the original image. R Equal to scale / scale0, the SIFT used in this invention sets the scale0 of the original image to 1.6.

[0064]

[0065] To ensure the same spatial resolution as the original spherical image, a pinhole camera model corresponding to the corrected image patch can be defined based on the defined image patch size, where the focal length f... p For W / 4, principal point c xp and c yp W respectively sp / 2、H sp / 2. In this invention, the corrected image block is generated through inverse correction to ensure that the output block of the desired dimension is obtained. The generation steps are as follows:

[0066] S341. Calculate the pixel coordinates p of the image points in the corrected image block = (p x ,p y The homogeneous coordinates p corresponding to ) h =(p xh,p yh ,1):

[0067]

[0068] Among them, (c xp ,c yp f is the principal point coordinate of the corrected image. p To correct the camera focal length corresponding to the image;

[0069] S342, change the homogeneous coordinates p h =(p xh ,p yh 1) Project the image onto the unit spherical rectangular coordinate system OX′Y′Z′ corresponding to the local Cartesian coordinate system of the corrected image block using the normalization operation shown in the following formula to obtain the spherical point p. ls :

[0070]

[0071] Among them, ||p h ‖ represents point p h The distance to the origin O of the unit spherical rectangular coordinate system OX′Y′Z′; the unit spherical rectangular coordinate system O-X'Y'Z' corresponding to the local Cartesian coordinate system of the image patch, with the Z' axis pointing to point p. ls .

[0072] S343, By transforming the matrix spherical point p ls Transform the unit spherical rectangular coordinate system OX′Y′Z′ corresponding to the local Cartesian coordinate system of the corrected image patch into the global spherical rectangular coordinate system O-XYZ. This involves transforming the unit spherical rectangular coordinate system corresponding to the local Cartesian coordinate system of the corrected image patch into the global spherical rectangular coordinate system O-XYZ, i.e., the spherical rectangular coordinate system corresponding to the spherical panoramic image. (See [link to documentation]). Figure 2 In (a), O-XYZ yields p s :

[0073] p s =R*p ls

[0074] Where R represents the rotation matrix, R y (θ) represents the rotation angle θ around the Y-axis. Indicates the rotation angle around the X-axis R z (ori) represents the rotation angle ori around the Z-axis.

[0075] S344, p s Image points in a spherical panoramic image converted to ERP format are then converted to grayscale values ​​and corrected to generate pixel coordinates p = (p...x ,p y The grayscale value of ) is used to obtain the corrected image block.

[0076] Based on the above steps, a dimension of W is generated by reprojection of the tangent plane. sp and H sp The corrected image block was ultimately resized to a size of W. p and H p The image patch. Rotation transformation R around the Z-axis. z (ori) represents the transformation from the principal direction of the selected keypoint to the nominal direction in the Cartesian coordinate system. It is used to achieve orientation invariance in the subsequently generated descriptor.

[0077] S4. Based on the corrected image block, generate a second feature descriptor with the same dimension as the first feature descriptor;

[0078] The corrected image patches are converted into feature descriptors for feature matching. In this invention, separate detectors and descriptor learning networks are employed for two main reasons. Firstly, the network input is an image patch, so a joint detector and descriptor learning network with the entire image as input is not used. Secondly, this strategy can be easily integrated into existing workflows for subsequent feature matching and SFM-based image orientation.

[0079] Based on the corrected image patch, a second feature descriptor with the same dimension as the first feature descriptor is generated using a pre-trained HardNet network. The HardNet network structure used in this invention is the same as L2-Net, consisting of seven CNN layers with batch normalization. The last layer does not use an activation function, while the remaining layers use ReLU activation, and dilated convolutions are used in the third and fifth layers. For an input image patch of size 32×32 pixels, HardNet outputs a 128-dimensional descriptor with the same dimension as the SIFT descriptor. Unlike L2-Net, HardNet employs a hard negative sampling strategy and a triplet boundary loss function for network training.

[0080] S5. Perform the same processing steps S1-S4 on the two spherical panoramic images to be matched. Based on the second feature descriptor, perform conventional feature matching on the two spherical panoramic images to obtain the initial feature matching of the two spherical panoramic images.

[0081] To establish the correspondence between two images, initial matches are first obtained using a standard feature matching strategy. Nearest neighbor and second nearest neighbor searches are then performed between the two sets of second feature descriptors, and feature points that pass the ratio test are set as candidate matches. Simultaneously, a cross-validation strategy is used to further refine the initial matches.

[0082] S6. Due to the limited discriminative power of repetitive patterns and local descriptors in images, incorrect matches are inevitably retained in the initial matching. This embodiment of the invention utilizes coplanar geometric constraints to optimize the initial feature matching. The coplanar geometric constraint requires that the baseline vector connecting the centers of two spherical panoramic images and the observation vector from the respective centers of the two spherical panoramic images to the object-side 3D point be coplanar. Two feature points satisfying the coplanar geometric constraint are correctly matched, resulting in the final feature matching result. (Reference) Figure 4 .

[0083] The relative orientation of two spherical panoramic images to be matched is represented by relative rotation R and translation T. Relative orientation is a method to determine the relationship between relative left and right stereo images, using the essential matrix E = [T]. × R encodes the relative orientation of two spherical panoramic images, [T] × Let T be the antisymmetric matrix of vector T.

[0084] Coplanar geometric constraints are expressed by the following formula:

[0085]

[0086] Where p1 and p2 are the spherical coordinates of the initial matching feature points on the two spherical panoramic images to be matched, respectively. This represents the transpose of p2.

[0087] To achieve robust estimation of the essential matrix E, this embodiment of the invention uses a hypothesis testing framework based on RANSAC. During the RANSAC iteration process, an error metric e and an error threshold e are required. p To identify correct and incorrect matches.

[0088] according to Figure 4 The diagram showing the coplanar constraint principle indicates that ray O2p2, corresponding to O1p1 in the left image, is located at... The normal vector is on a circular plane with the projection center O2 of the right image as the center and the radius of the spherical image as the radius.

[0089]

[0090]

[0091] Where e is the spherical angle error measure, e p e is the error threshold in pixels. a is the error threshold for the spherical angle, abs(·) represents the absolute value, 2π / max(W,H) represents the scale factor of these two metrics, and max(W,H) represents the pixel size of the long side of the spherical panoramic image in ERP format.

[0092] When e <ea At that time, the two feature points that were initially matched were the correct feature matches.

[0093] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the above-described spherical panoramic image feature matching method.

[0094] Please see Figure 5 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.

[0095] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus to execute the above-mentioned spherical panoramic image feature matching method.

[0096] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for matching features of spherical panoramic images, characterized in that, The method comprises the following steps: S1, acquiring a spherical panoramic image, processing the spherical panoramic image using equidistant cylindrical projection to obtain an ERP format spherical panoramic image; S2, performing feature extraction on the ERP format spherical panoramic image and generating a first feature descriptor; S3, performing local image block geometric correction on the ERP format spherical panoramic image using image block re-projection to obtain a corrected image block; S4, generating a second feature descriptor with the same dimension as the first feature descriptor based on the corrected image block; S5, performing the same processing of steps S1-S4 on two spherical panoramic images to be matched, performing regular feature matching on the two spherical panoramic images based on the second feature descriptor to obtain initial feature matching of the two spherical panoramic images; S6, optimizing the initial feature matching using coplanar geometric constraints, the coplanar geometric constraints requiring that a baseline vector connecting the centers of the two spherical panoramic images and an observation vector from the center of each of the two spherical panoramic images to a three-dimensional point in the object space are coplanar, and the two feature points satisfying the coplanar geometric constraints are correctly matched to obtain a final feature matching result.

2. The method of claim 1, wherein, In step S2, the SIFT algorithm is used for feature extraction.

3. The method of claim 1, wherein, Step S3 is specifically: S31, selecting a feature point in the ERP format spherical panoramic image as a key point; S32, converting the key point coordinates into latitude and longitude coordinates of a spherical coordinate system; S33, taking the vector from the origin of the spherical coordinate system to the spherical key point as the normal vector of the tangent plane passing through the spherical key point to obtain the tangent plane; S34, projecting the local image block around the key point in the ERP format spherical image onto the tangent plane based on imaging geometry to obtain a corrected image block.

4. The method of claim 3, wherein, In step S34, the corrected image block is generated by inverse correction, specifically: S341. Calculate the pixel coordinates p of the image points in the corrected image block = (p x ,p y The homogeneous coordinates p corresponding to ) h =(p xh ,p yh ,1): wherein (c xp , c yp ) is the principal point coordinate of the rectified image, f p is the focal length of the camera corresponding to the rectified image; S342, change the homogeneous coordinates p h =(p xh ,p yh 1) Project the image block onto the unit spherical rectangular coordinate system O-X'Y'Z' corresponding to the local Cartesian coordinate system of the corrected image block to obtain the spherical point p. ls : wherein ||p h represents the distance from the point p h to the origin O of the rectangular coordinate system O-X'Y'Z' of the unit sphere. S343、by the transformation matrix Convert the spherical point p ls Convert the spherical point p s : p s = R * p ls where R denotes a rotation matrix, R y (θ) denotes a rotation angle θ around the Y-axis, denotes a rotation angle φ around the X-axis R z (ori) denotes a rotation angle ori around the Z-axis; S344, p s Image points in a spherical panoramic image converted to ERP format are then converted to grayscale values ​​and corrected to generate pixel coordinates p = (p... x ,p y The grayscale value of ) is used to obtain the corrected image block.

5. The method of claim 1, wherein, In step S4, based on the corrected image block, a second feature descriptor with the same dimension as the first feature descriptor is generated using a pre-trained HardNet network; wherein the pre-training uses a hard negative sampling strategy and a three-tuple boundary loss function to train the HardNet network.

6. The method of claim 1, wherein, In step S6, the relative orientation of the two spherical panoramic images to be matched is denoted by a relative rotation R and a translation T. The relative orientation is encoded using an essential matrix E = [T] × R encoding the relative orientation of the two spherical panoramic images, [T] × denoting the skew-symmetric matrix of T, the coplanar geometry constraint is expressed by the following equation: wherein p1, p2 are the spherical coordinates of the initial matching feature points on the two spherical panoramic images to be matched respectively, denotes the transpose of p2.

7. The method of claim 6, wherein, In step S6, the robust estimation of the essential matrix E is realized based on the hypothesis verification framework of RANSAC, and in the RANSAC iteration process, the error measurement and error threshold are used to judge correct feature matching and incorrect feature matching: where e is the spherical angular error metric, e p is the error threshold in pixels, e a is the error threshold for the spherical angle, abs(·) denotes the absolute value, and max(W, H) denotes the long side pixel size of the spherical panorama image in the ERP format. When e < e a The two initial matched feature points are correct feature matches.

8. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to realize the method of any one of claims 1-7.

9. An electronic device, comprising: The computer program comprises computer readable instructions, and the processor is configured to invoke the computer readable instructions to execute the method of any one of claims 1-7.

Images