Relative pose estimation methods, devices, terminal equipment, and storage media for semi-calibrated cameras
By identifying the relative motion of the semi-calibrated camera and evaluating translation or rotation, the number of feature point pairs is reduced, improving the computational efficiency of the relative pose of the semi-calibrated camera and solving the problem of high computational complexity in the prior art.
Patent Information
- Application Number
- CN202410783142.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-06-18
AI Technical Summary
In existing technologies, the relative pose estimation methods for semi-calibrated cameras have high computational complexity and low efficiency, making it difficult to perform feature point matching efficiently.
By identifying the relative motion of the semi-calibrated camera, it is assessed whether it is translational motion. If so, only a small number of feature point pairs are needed to calculate the translation matrix; otherwise, more feature point pairs are obtained to calculate the translation and rotation matrices, reducing the number of feature points and descriptors.
It improves the computational efficiency of relative pose calculation for semi-calibrated cameras and reduces computational complexity.
Smart Images

Figure CN118644559B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, terminal device, and storage medium for relative pose estimation of a semi-calibrated camera. Background Technology
[0002] In recent years, with the rapid development of mobile platform devices such as smartphones, drones, and robots, more and more mobile platform devices are being widely used in fields such as simultaneous localization and mapping (SLAM), motion reconstruction structures, and virtual reality. Achieving self-localization of the mobile platform is a prerequisite and key to meeting application requirements. Commonly used sensors installed on mobile platforms include Global Navigation Satellite System (GNSS), LiDAR, visual sensors (cameras), and inertial measurement units (IMUs). Among these, vision-based positioning, with its advantages of high accuracy, miniaturization, non-contact operation, and low cost, is widely used in aerospace engineering, medical services, agricultural production, and industrial production. Simultaneously, with the widespread adoption of inertial measurement elements and their advantages such as being unaffected by external factors, operating in all weather conditions, and having low data noise, many mobile platform devices simultaneously install both visual sensors (cameras) and inertial measurement units.
[0003] In practical applications, the assumption of a semi-calibrated camera is quite common. A semi-calibrated camera refers to a camera whose only intrinsic parameter is unknown—the focal length. This assumption is reasonable because ① the aspect ratio of pixels is usually 1:1, ② the principal point of a CCD or CMOS camera is located at the center of the image, and ③ if the field of view is narrow, distortion can be ignored. Since the camera and inertial measurement unit are fixed on the moving platform, the relative pose of a semi-calibrated camera only contains two degrees of freedom. However, currently, the main method for estimating the relative pose of a semi-calibrated camera is the feature point matching algorithm. This algorithm has high accuracy, but due to the need to process a large number of feature points and descriptors, it has high computational complexity and low efficiency.
[0004] Therefore, how to improve the efficiency of estimating the relative pose of a semi-calibrated camera based on feature point matching algorithms has become an urgent problem to be solved. Summary of the Invention
[0005] This invention provides a method, apparatus, terminal device, and storage medium for relative pose estimation of a semi-calibrated camera. By identifying the relative motion of the camera, the number of feature points and descriptors that need to be processed to calculate the relative pose of the semi-calibrated camera is reduced, thereby effectively improving the calculation efficiency of the relative pose of the semi-calibrated camera.
[0006] An embodiment of the present invention provides a relative pose estimation method for a semi-calibrated camera, comprising: Obtain the first intrinsic parameter matrix of the semi-calibrated camera at the first time point and the second intrinsic parameter matrix at the second time point; Wherein, the first time point is the time point preceding the second time point; The relative motion of the semi-calibrated camera at the first time and the second time is determined based on the inertial measurement unit calibrated with the semi-calibrated camera. When the relative motion is determined to be translational motion, a first number of first feature point pairs are extracted from the first image captured by the semi-calibrated camera at the first moment and the second image captured at the second moment. The first translation matrix of the semi-calibrated camera is calculated based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first feature point pair. When it is determined that the relative motion is not a translational motion, a second number of second feature point pairs are extracted from the first image and the second image; Wherein, the second quantity is greater than the first quantity; Based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second feature point pair, the second translation matrix and rotation matrix of the semi-calibrated camera are calculated.
[0007] Furthermore, calculating the first translation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first feature point pair includes: Based on the translation direction of the semi-calibrated camera, construct the first translation matrix to be determined for the semi-calibrated camera; When it is determined that the semi-calibrated camera has not adjusted its camera focal length at the second time, the first intrinsic parameter matrix and the second intrinsic parameter matrix are determined to be equal. Construct the first pose calculation model based on the first intrinsic parameter matrix and the first translation matrix to be calculated. According to the epipolar constraint formula, the first feature point pair is sequentially input into the first pose calculation model to solve the first translation matrix to be solved, thereby generating the first translation matrix of the semi-calibrated camera.
[0008] Furthermore, the step of calculating the first translation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first feature point pair further includes: When it is determined that the semi-calibrated camera has adjusted the camera focal length at the second time, it is determined that the first intrinsic parameter matrix and the second intrinsic parameter matrix are not equal; Construct a second pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined. According to the epipolar constraint formula, the first feature point pair is sequentially input into the second pose calculation model for calculation to generate the first translation matrix of the semi-calibrated camera and the first camera focal length of the semi-calibrated camera at the second time moment.
[0009] Furthermore, the step of calculating the second translation matrix and rotation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second feature point pair includes: Construct the second translation matrix to be determined for the semi-calibrated camera; Based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined, a third pose calculation model is constructed. According to the epipolar constraint formula, the second feature point pair is sequentially input into the third pose calculation model for calculation, generating the second translation matrix, rotation matrix, and second camera focal length of the semi-calibrated camera at the second time moment.
[0010] Furthermore, the step of constructing the first pose calculation model based on the first intrinsic parameter matrix and the first translation matrix to be calculated includes: Based on the first intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera, construct the first matrix to be determined; Based on the first matrix to be determined and the first translation matrix to be determined, the following first pose calculation model is constructed: ; in, This is the first pose calculation model. Let be the first matrix to be found. Let be the first translation matrix to be determined.
[0011] Furthermore, the step of constructing the second pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined includes: Based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera, construct the second matrix to be determined; Based on the second matrix to be determined and the first translation matrix to be determined, the following second pose calculation model is constructed: ; in, This is the second pose calculation model. The second matrix to be found. Let be the first translation matrix to be determined.
[0012] Furthermore, the step of constructing a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined includes: Based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera, a third matrix to be determined is constructed. Based on the third matrix to be determined and the second translation matrix to be determined, the following third pose calculation model is constructed: ; in, This is the third pose calculation model. The third matrix to be determined is... Let be the second translation matrix to be determined.
[0013] Another embodiment of the present invention provides a relative pose estimation device for a semi-calibrated camera, comprising: The parameter acquisition module is used to acquire the first intrinsic parameter matrix of the semi-calibrated camera at the first time point and the second intrinsic parameter matrix at the second time point; wherein, the first time point is the time point preceding the second time point; The motion evaluation module is used to determine the relative motion of the semi-calibrated camera at the first time and the second time based on the inertial measurement unit calibrated with the semi-calibrated camera. The first feature point generation module is used to extract a first number of first feature point pairs from the first image captured by the semi-calibrated camera at the first moment and the second image captured at the second moment when it is determined that the relative motion is a translational motion. The first pose estimation module is used to construct a first translation matrix to be determined for the semi-calibrated camera based on the translation direction of the semi-calibrated camera; when it is determined that the semi-calibrated camera has not adjusted its camera focal length at the second time moment, it determines that the first intrinsic parameter matrix and the second intrinsic parameter matrix are equal; based on the first intrinsic parameter matrix and the first translation matrix to be determined, it constructs a first pose calculation model; according to the epipolar constraint formula, it sequentially inputs the first feature point pairs into the first pose calculation model to solve the first translation matrix to be determined, generating the first translation matrix of the semi-calibrated camera; when it is determined that the semi-calibrated camera has adjusted its camera focal length at the second time moment, it determines that the first intrinsic parameter matrix and the second intrinsic parameter matrix are not equal; based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined, it constructs a second pose calculation model; according to the epipolar constraint formula, it sequentially inputs the first feature point pairs into the second pose calculation model for calculation, generating the first translation matrix of the semi-calibrated camera and the first camera focal length of the semi-calibrated camera at the second time moment; The second feature point generation module is used to extract a second number of second feature point pairs from the first image and the second image when it is determined that the relative motion is not a translational motion. The second pose estimation module is used to construct the second translation matrix to be determined for the semi-calibrated camera; construct a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined; and input the second feature point pair into the third pose calculation model in sequence according to the epipolar constraint formula to generate the second translation matrix, rotation matrix, and second camera focal length of the semi-calibrated camera at the second time step.
[0014] Another embodiment of the present invention provides a terminal device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a relative pose estimation method for a semi-calibrated camera as described in any of the embodiments.
[0015] Another embodiment of the present invention provides a storage medium, characterized in that the storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform a relative pose estimation method for a semi-calibrated camera as described in any of the above embodiments.
[0016] The following benefits can be obtained by implementing the present invention: This invention discloses a method, apparatus, terminal device, and storage medium for estimating the relative pose of a semi-calibrated camera. Before extracting feature points, the relative motion of the semi-calibrated camera at a first and second time point is evaluated. If the relative motion is determined to be translational, it can be determined that the semi-calibrated camera has not rotated. The translation matrix of the semi-calibrated camera can be obtained using only a few feature points, thus estimating the relative pose of the camera at previous and subsequent time points. Only when the relative motion is determined to be translational is more feature points acquired to further obtain the translation and rotation matrices of the semi-calibrated camera, thereby evaluating the relative pose of the camera at previous and subsequent time points. Therefore, this invention reduces the number of feature points and descriptors to be processed by identifying the relative motion of the camera, thereby effectively improving the computational efficiency of the relative pose of the semi-calibrated camera. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating a relative pose estimation method for a semi-calibrated camera according to an embodiment of the present invention.
[0018] Figure 2 This is a schematic diagram of the relative pose estimation device for a semi-calibrated camera provided in an embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0021] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0023] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0024] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0025] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0026] See Figure 1 This is a flowchart illustrating a relative pose estimation method for a semi-calibrated camera according to an embodiment of the present invention, comprising: S1. Obtain the first intrinsic parameter matrix of the semi-calibrated camera at the first time and the second intrinsic parameter matrix at the second time. Wherein, the first time point is the time point preceding the second time point; In a preferred embodiment of the present invention, the following first intrinsic parameter matrix and second intrinsic parameter matrix are obtained: ; ; in, This is the first intrinsic parameter matrix. The focal length of the semi-calibrated camera at the first moment. This is the second intrinsic parameter matrix. The focal length of the semi-calibrated camera at the second moment is denoted as .
[0027] S2. Determine the relative motion of the semi-calibrated camera at the first moment and the second moment based on the inertial measurement unit calibrated with the semi-calibrated camera; In a preferred embodiment of the present invention, the relative motion of the semi-calibrated camera at a first moment and a second moment is determined by an inertial measurement unit calibrated with the semi-calibrated camera.
[0028] It should be noted that, in this embodiment, before performing relative pose estimation of the semi-calibrated camera, it is necessary to calibrate the semi-calibrated camera and the inertial measurement unit. The calibration steps are as follows: S01. Input the GNSS timestamp into the IMU's raw data. Although the inertial navigation system's time may not be perfectly synchronized with the GNSS time to correspond to whole seconds, it can be estimated through interpolation. The camera sensor is synchronized by controlling the camera's triggering through the GNSS time, so that each frame of the image has a corresponding timestamp.
[0029] Understandably, time synchronization can unify the time of the inertial navigation and camera sensors into the same high-precision time system.
[0030] S02. The camera coordinate system and IMU coordinate system are calibrated using the open-source Kalibr scheme and software.
[0031] Understandably, given that the semi-calibrated camera and the inertial measurement unit have undergone time and space calibration, step S2 can accurately determine the relative motion of the semi-calibrated camera at the first and second moments using the inertial measurement unit.
[0032] S3. When it is determined that the relative motion is a translational motion, a first number of first feature point pairs are extracted from the first image captured by the semi-calibrated camera at the first moment and the second image captured at the second moment. In a preferred embodiment of the present invention, a first number of first feature point pairs are obtained through SIFT feature extraction and matching. It should be noted that a feature point pair includes a feature point and its coordinates in the first image, and a feature point and its coordinates in the second image that corresponds to the feature point in the first image.
[0033] S4. Calculate the first translation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first feature point pair. Preferably, calculating the first translation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first feature point pair includes: S41. Based on the translation direction of the semi-calibrated camera, construct the first translation matrix to be determined for the semi-calibrated camera; In a preferred embodiment of the present invention, taking a planar sweeping robot as an example, the following translation matrix is constructed: ; in, and These represent the components of the translation vector along the x-axis and z-axis, respectively.
[0034] S42. When it is determined that the semi-calibrated camera has not adjusted the camera focal length at the second time, the first intrinsic parameter matrix and the second intrinsic parameter matrix are determined to be equal. In a preferred embodiment of the present invention, based on the first intrinsic parameter matrix and the second intrinsic parameter matrix obtained in step S1, it can be clearly known that, Under the premise, .
[0035] S43. Construct the first pose calculation model based on the first intrinsic parameter matrix and the first translation matrix to be calculated; Preferably, constructing the first pose calculation model based on the first intrinsic parameter matrix and the first translation matrix to be determined includes: S431. Construct a first matrix to be determined based on the first intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera; S432. Based on the first matrix to be calculated and the first translation matrix to be calculated, construct the following first pose calculation model: ; in, This is the first pose calculation model. Let be the first matrix to be found. Let be the first translation matrix to be determined.
[0036] In a preferred embodiment of the present invention, the relationship between the fundamental matrix and the essential matrix of the semi-calibrated camera, as well as the essential matrix itself, are as follows: ; ; Where F is the basic matrix and E is the essential matrix. and These are the translation vector and the rotation matrix, respectively. This is the first intrinsic parameter matrix. This is the second intrinsic parameter matrix. Translation vector An antisymmetric matrix.
[0037] Furthermore, based on the relationship between the fundamental matrix and the essential matrix of the semi-calibrated camera, and the essential matrix itself, a 2x2 first matrix to be determined is constructed. .
[0038] S44. According to the epipolar constraint formula, the first feature point pair is sequentially input into the first pose calculation model to solve the first translation matrix to be solved, thereby generating the first translation matrix of the semi-calibrated camera.
[0039] In a preferred embodiment of the present invention, the preset first quantity is 2, that is: inputting two pairs of first feature points into the first pose calculation model to solve the first translation matrix to be solved.
[0040] The polar constraint formula is shown below: ; in, and Let F be the coordinates of the feature points in a pair of feature points, and let F be the basic matrix.
[0041] Furthermore, based on the relationship between the fundamental matrix and the essential matrix, the essential matrix, and the epipolar constraint formula, it can be determined that... ,Right now: ; because It has non-zero solutions, therefore Then, the focal length f can be calculated and substituted into the first pose calculation model to obtain the first translation matrix to be determined. Thus, the first translation matrix is obtained.
[0042] S45. When it is determined that the semi-calibrated camera has adjusted the camera focal length at the second moment, it is determined that the first intrinsic parameter matrix and the second intrinsic parameter matrix are not equal. In a preferred embodiment of the present invention, since the sweeping robot moving on the plane may have a situation where the distance between the camera and the target changes significantly, the camera adjusts its focal length to achieve a clearer image. In this case, the focal length of the camera is different at the first moment and the second moment, which in turn makes the first intrinsic parameter matrix and the second intrinsic parameter matrix unequal.
[0043] S46. Construct a second pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined. Preferably, the step of constructing the second pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined includes: S461. Construct a second matrix to be determined based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera; S462. Based on the second matrix to be calculated and the first translation matrix to be calculated, construct the following second pose calculation model: ; in, This is the second pose calculation model. The second matrix to be found. Let be the first translation matrix to be determined.
[0044] In a preferred embodiment of the present invention, a 2*2 second matrix to be determined is constructed based on the relationship between the fundamental matrix and the essential matrix of the semi-calibrated camera, and the essential matrix itself. .
[0045] S47. According to the epipolar constraint formula, the first feature point pair is sequentially input into the second pose calculation model for calculation to generate the first translation matrix of the semi-calibrated camera and the first camera focal length of the semi-calibrated camera at the second time.
[0046] In a preferred embodiment of the present invention, based on the relationship between the fundamental matrix and the essential matrix, the essential matrix, and the epipolar constraint formula, it can be determined that... ,Right now: ; because It has non-zero solutions, therefore Then, the focal length f can be calculated based on the two pairs of feature points, and substituted into the second pose calculation model to obtain the first translation matrix to be determined. Thus, the first translation matrix is obtained.
[0047] S5. When it is determined that the relative motion is not a translational motion, extract a second number of second feature point pairs from the first image and the second image; Wherein, the second quantity is greater than the first quantity; In a preferred embodiment of the present invention, when it is detected that the relative motion is not a translational motion, three pairs of second feature points are obtained.
[0048] It is understandable that for drones, when the distance between the camera and the target changes significantly, they may need to rotate and adjust the focus in order to obtain a clearer image.
[0049] S6. Calculate the second translation matrix and rotation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second feature point pair.
[0050] Preferably, the step of calculating the second translation matrix and rotation matrix of the semi-calibrated camera based on the epipolar constraint formula, the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second feature point pair includes: S61. Construct the second translation matrix to be determined for the semi-calibrated camera; In a preferred embodiment of the present invention, the second translation matrix to be determined is constructed as follows: ; in, These are the components of the translation vector along the x-axis, y-axis, and z-axis, respectively.
[0051] S62. Construct a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined. Preferably, the step of constructing the third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined includes: S621. Construct a third matrix to be determined based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera. S622. Based on the third matrix to be determined and the second translation matrix to be determined, construct the following third pose calculation model: ; in, This is the third pose calculation model. The third matrix to be determined is... Let be the second translation matrix to be determined.
[0052] In a preferred embodiment of the present invention, a 3*3 third matrix to be determined is constructed based on the relationship between the fundamental matrix and the essential matrix of the semi-calibrated camera, and the essential matrix itself. .
[0053] S63. According to the epipolar constraint formula, the second feature point pair is sequentially input into the third pose calculation model for calculation to generate the second translation matrix, rotation matrix and second camera focal length of the semi-calibrated camera at the second time.
[0054] In a preferred embodiment of the present invention, based on the relationship between the fundamental matrix and the essential matrix, the essential matrix, and the epipolar constraint formula, it can be determined that... ,Right now: ; because It has non-zero solutions, therefore Then, the focal length f can be calculated based on the three pairs of feature points, and substituted into the third pose calculation model to obtain the second translation matrix to be determined. This yields the second translation matrix.
[0055] This embodiment provides a method for estimating the relative pose of a semi-calibrated camera. Before extracting feature points, the relative motion of the semi-calibrated camera at a first time and a second time is evaluated. If the relative motion is determined to be translational, it can be determined that the semi-calibrated camera has not rotated. The translation matrix of the semi-calibrated camera can be obtained with only a few feature points, and then the relative pose of the camera at previous and next time moments can be estimated. When the relative motion is determined to be translational, more feature points are obtained to further obtain the translation and rotation matrices of the semi-calibrated camera, and then the relative pose of the semi-calibrated camera at previous and next time moments can be evaluated. Therefore, this invention reduces the number of feature points and descriptors that need to be processed by identifying the relative motion of the camera, thereby effectively improving the computational efficiency of the relative pose of the semi-calibrated camera.
[0056] See Figure 2 This is a schematic diagram of the structure of a relative pose estimation device for a semi-calibrated camera provided in an embodiment of the present invention, comprising: The parameter acquisition module is used to acquire the first intrinsic parameter matrix of the semi-calibrated camera at the first time point and the second intrinsic parameter matrix at the second time point; wherein, the first time point is the time point preceding the second time point; The motion evaluation module is used to determine the relative motion of the semi-calibrated camera at the first time and the second time based on the inertial measurement unit calibrated with the semi-calibrated camera. The first feature point generation module is used to extract a first number of first feature point pairs from the first image captured by the semi-calibrated camera at the first moment and the second image captured at the second moment when it is determined that the relative motion is a translational motion. The first pose estimation module is used to construct a first translation matrix to be determined for the semi-calibrated camera based on the translation direction of the semi-calibrated camera; when it is determined that the semi-calibrated camera has not adjusted its camera focal length at the second time moment, it determines that the first intrinsic parameter matrix and the second intrinsic parameter matrix are equal; based on the first intrinsic parameter matrix and the first translation matrix to be determined, it constructs a first pose calculation model; according to the epipolar constraint formula, it sequentially inputs the first feature point pairs into the first pose calculation model to solve the first translation matrix to be determined, generating the first translation matrix of the semi-calibrated camera; when it is determined that the semi-calibrated camera has adjusted its camera focal length at the second time moment, it determines that the first intrinsic parameter matrix and the second intrinsic parameter matrix are not equal; based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined, it constructs a second pose calculation model; according to the epipolar constraint formula, it sequentially inputs the first feature point pairs into the second pose calculation model for calculation, generating the first translation matrix of the semi-calibrated camera and the first camera focal length of the semi-calibrated camera at the second time moment; The second feature point generation module is used to extract a second number of second feature point pairs from the first image and the second image when it is determined that the relative motion is not a translational motion. The second pose estimation module is used to construct the second translation matrix to be determined for the semi-calibrated camera; construct a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined; and input the second feature point pair into the third pose calculation model in sequence according to the epipolar constraint formula to generate the second translation matrix, rotation matrix, and second camera focal length of the semi-calibrated camera at the second time step.
[0057] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0058] Those skilled in the art will clearly understand that, for convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0059] Another preferred embodiment of the present invention provides a terminal device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a relative pose estimation method for a semi-calibrated camera as described in any of the foregoing embodiments.
[0060] The terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The terminal device may include, but is not limited to, a processor and a memory.
[0061] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.
[0062] The memory can be used to store the computer program. The processor implements various functions of the terminal device by running or executing the computer program stored in the memory and calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0063] Another preferred embodiment of the present invention provides a storage medium, which is a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0064] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for relative pose estimation of a semi-calibrated camera, characterized in that, include: Obtain the first intrinsic parameter matrix of the semi-calibrated camera at the first time point and the second intrinsic parameter matrix at the second time point; Wherein, the first time point is the time point preceding the second time point; The relative motion of the semi-calibrated camera at the first time and the second time is determined based on the inertial measurement unit calibrated with the semi-calibrated camera. When the relative motion is determined to be translational motion, a first number of first feature point pairs are extracted from the first image captured by the semi-calibrated camera at the first moment and the second image captured at the second moment. Based on the translation direction of the semi-calibrated camera, construct the first translation matrix to be determined for the semi-calibrated camera; When it is determined that the semi-calibrated camera has not adjusted its focal length at the second time, the first intrinsic parameter matrix and the second intrinsic parameter matrix are determined to be equal; based on the first intrinsic parameter matrix and the first translation matrix to be solved, a first pose calculation model is constructed; according to the epipolar constraint formula, the first feature point pairs are sequentially input into the first pose calculation model to solve the first translation matrix to be solved, thereby generating the first translation matrix of the semi-calibrated camera; When it is determined that the semi-calibrated camera has adjusted its focal length at the second time step, it is determined that the first intrinsic parameter matrix and the second intrinsic parameter matrix are not equal; based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined, a second pose calculation model is constructed; according to the epipolar constraint formula, the first feature point pair is sequentially input into the second pose calculation model for calculation to generate the first translation matrix of the semi-calibrated camera and the first camera focal length of the semi-calibrated camera at the second time step; When it is determined that the relative motion is not a translational motion, a second number of second feature point pairs are extracted from the first image and the second image; Wherein, the second quantity is greater than the first quantity; Construct the second translation matrix to be determined for the semi-calibrated camera; construct a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined; input the second feature point pair into the third pose calculation model in sequence according to the epipolar constraint formula to generate the second translation matrix, rotation matrix, and second camera focal length of the semi-calibrated camera at the second time step.
2. The relative pose estimation method for a semi-calibrated camera as described in claim 1, characterized in that, The step of constructing the first pose calculation model based on the first intrinsic parameter matrix and the first translation matrix to be calculated includes: Based on the first intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera, construct the first matrix to be determined; Based on the first matrix to be determined and the first translation matrix to be determined, the following first pose calculation model is constructed: ; in, This is the first pose calculation model. Let be the first matrix to be found. Let be the first translation matrix to be determined.
3. The relative pose estimation method for a semi-calibrated camera as described in claim 1, characterized in that, The step of constructing a second pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined includes: Based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera, construct the second matrix to be determined; Based on the second matrix to be determined and the first translation matrix to be determined, the following second pose calculation model is constructed: ; in, This is the second pose calculation model. The second matrix to be found is... Let be the first translation matrix to be determined.
4. The relative pose estimation method for a semi-calibrated camera as described in claim 1, characterized in that, The step of constructing a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined includes: Based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, the fundamental matrix of the semi-calibrated camera, and the essential matrix of the semi-calibrated camera, a third matrix to be determined is constructed. Based on the third matrix to be determined and the second translation matrix to be determined, the following third pose calculation model is constructed: ; in, This is the third pose calculation model. The third matrix to be determined. Let be the second translation matrix to be determined.
5. A relative pose estimation device for a semi-calibrated camera, characterized in that, include: The parameter acquisition module is used to acquire the first intrinsic parameter matrix of the semi-calibrated camera at the first time point and the second intrinsic parameter matrix at the second time point; wherein, the first time point is the time point preceding the second time point; The motion evaluation module is used to determine the relative motion of the semi-calibrated camera at the first time and the second time based on the inertial measurement unit calibrated with the semi-calibrated camera. The first feature point generation module is used to extract a first number of first feature point pairs from the first image captured by the semi-calibrated camera at the first moment and the second image captured at the second moment when it is determined that the relative motion is a translational motion. The first pose estimation module is used to construct a first translation matrix to be determined for the semi-calibrated camera based on the translation direction of the semi-calibrated camera; when it is determined that the semi-calibrated camera has not adjusted its camera focal length at the second time moment, it determines that the first intrinsic parameter matrix and the second intrinsic parameter matrix are equal; based on the first intrinsic parameter matrix and the first translation matrix to be determined, it constructs a first pose calculation model; according to the epipolar constraint formula, it sequentially inputs the first feature point pairs into the first pose calculation model to solve the first translation matrix to be determined, generating the first translation matrix of the semi-calibrated camera; when it is determined that the semi-calibrated camera has adjusted its camera focal length at the second time moment, it determines that the first intrinsic parameter matrix and the second intrinsic parameter matrix are not equal; based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the first translation matrix to be determined, it constructs a second pose calculation model; according to the epipolar constraint formula, it sequentially inputs the first feature point pairs into the second pose calculation model for calculation, generating the first translation matrix of the semi-calibrated camera and the first camera focal length of the semi-calibrated camera at the second time moment; The second feature point generation module is used to extract a second number of second feature point pairs from the first image and the second image when it is determined that the relative motion is not a translational motion. The second pose estimation module is used to construct the second translation matrix to be determined for the semi-calibrated camera; construct a third pose calculation model based on the first intrinsic parameter matrix, the second intrinsic parameter matrix, and the second translation matrix to be determined; and input the second feature point pair into the third pose calculation model in sequence according to the epipolar constraint formula to generate the second translation matrix, rotation matrix, and second camera focal length of the semi-calibrated camera at the second time step.
6. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a relative pose estimation method for a semi-calibrated camera as described in any one of claims 1 to 4.
7. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform a relative pose estimation method for a semi-calibrated camera as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Attitude angle estimation method based on feature point set grouping
CN107798705A
Camera motion state-based binocular-inertial fusion pose estimation method, electronic equipment and storage medium
CN116205947A
