Multi-view camera pose updating method and system and storage medium

Abstract

The invention discloses a method for updating the pose of a multi-view camera. The method comprises the following steps: S1, acquiring an initial pose reference value of each camera in the multi-view camera; s2, at least one camera in the multi-view camera collects continuous time sequence images, and the two-dimensional picture offset between two continuous frames of time sequence images is calculated; s3, based on the two-dimensional picture offset, obtaining a three-dimensional pose increment by adopting an increment projection model; and S4, updating the camera pose in combination with the three-dimensional pose increment and the initial pose reference value. The invention further discloses a system for updating the pose of the multi-view camera to implement the method for updating the pose of the multi-view camera. Moreover, the invention further discloses a storage medium, a computer program is stored in the storage medium, and the multi-camera pose updating method is realized when the computer program is executed by a processor. Therefore, the multi-camera pose updating method and system designed by the invention can update the pose of the fixed monitoring camera in real time, and are particularly suitable for micro-vibration scenes such as a power transmission tower and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent monitoring, and particularly relates to a method and system for updating the pose of a multi-camera and a storage medium. Background Art

[0002] As is well known, a multi-camera is a very important stereo vision camera. The image pairs obtained by the existing multi-cameras must be corrected to be aligned. The correction of the image pairs depends on the relative pose between the cameras. Therefore, it is necessary to calibrate the internal and external parameters of the multi-camera to ensure its normal operation.

[0003] However, after the external parameter calibration of the multi-camera is completed, in some harsh situations, the external parameters of the above multi-camera generally change during application. For example, when the multi-camera is fixed on a transmission tower in the power transmission field, the transmission tower will experience micro-vibrations, resulting in changes in the camera's external parameters.

[0004] Therefore, in order to ensure the accuracy of the above multi-camera during subsequent multi-camera three-dimensional reconstruction, it is usually necessary to update the pose of the camera in real time. For example: Chinese Patent Invention with Publication No.: CN109087355A, Publication Date: December 25, 2018, and Title: A Monocular Camera Pose Measurement Device and Method Based on Iterative Update proposes a monocular camera pose measurement technology based on iterative update. This technology captures scene images through a monocular camera and accurately measures the camera pose in combination with an iterative algorithm, thereby realizing the accurate determination of the position of an object in three-dimensional space. Compared with traditional update methods, it has significant advantages in measurement accuracy and iterative update speed, improving the efficiency and accuracy of pose measurement.

[0005] However, there are still some defects in the existing technology that need to be urgently solved: First, the system robustness of the existing camera pose update method is poor; since it relies on the recognition of specific feature points in a two-dimensional image, when facing complex environments or scene changes, the extraction and matching of feature points may be interfered, resulting in incomplete detection results, thereby limiting the accuracy rate of pose measurement. Second, the existing technology fails to make full use of the time-series picture information. In a dynamic scene, the position and pose of an object may change over time. The method of updating the pose only relying on a single picture cannot effectively capture this dynamic change.

[0006] In addition, in some existing technical methods, it is usually necessary to rely on a mobile platform to collect images, which not only limits its application in micro-vibration scenarios but also results in low detection efficiency and is difficult to meet the requirements of application scenarios with high real-time requirements.

[0007] In summary, the existing camera pose update technology still has deficiencies in terms of robustness, adaptability to dynamic scenes, and detection efficiency. There is an urgent need for a new multi-camera pose update method to meet the growing application requirements of high precision, high efficiency, and high robustness. Summary of the Invention

[0008] The technical problem to be solved by the present invention is: The present invention discloses a multi-camera pose update method to solve the problems of poor robustness, poor adaptability to dynamic scenes, and low detection efficiency existing in the prior art when updating the camera pose.

[0009] To solve the above technical problem, the technical solution adopted by the present invention is: A multi-camera pose update method, which includes the steps: S1: Obtain the initial pose reference values of each camera in the multi-camera; S2: At least one camera in the multi-camera collects continuous time-series images and calculates the two-dimensional picture offset between two consecutive frames of the time-series images; S3: Based on the two-dimensional picture offset, obtain the three-dimensional pose increment; S4: Combine the three-dimensional pose increment and the initial pose reference value to update the camera pose.

[0010] In the above technical solution of the present invention, aiming at the problems of poor robustness, poor adaptability to dynamic scenes, and low detection efficiency existing in the prior art when updating the camera pose. The present invention discloses a new multi-camera pose update method. This multi-camera pose update method can effectively obtain the initial pose reference values of each camera in the multi-camera, and then control at least one camera to collect continuous time-series images to calculate the two-dimensional picture offset between two consecutive frames of the time-series images; based on these two-dimensional picture offsets, a mathematical relationship model with the three-dimensional pose increment can be constructed, so as to combine the three-dimensional pose increment and the initial pose reference value to update the camera pose in real time.

[0011] It can be seen that the multi-camera pose update method designed by the present invention is particularly suitable for micro-vibration scenarios such as electric towers. This multi-camera pose update method not only has high robustness, but also has a fast detection rate to update the pose of the multi-camera in the electric tower environment in real time, providing accurate external camera parameters for subsequent multi-camera three-dimensional reconstruction, thereby significantly improving the accuracy and reliability of the subsequent three-dimensional reconstruction results. It is easy to implement and has good promotion prospects and application value.

[0012] Furthermore, in the multi-camera pose update method of the present invention, in step S1, it specifically includes the steps: S11: Synchronously shoot a preset area using a multi-camera, and extract feature points from the images captured by each camera in the multi-camera; S12: Perform feature point matching between the images captured by different cameras, determine the best matching pairs, and establish a feature matching relationship; S13: Calibrate the external parameters of each camera in the multi-camera, and obtain the relative pose between the cameras to obtain the initial pose reference values of each camera in the multi-camera.

[0013] Furthermore, in the multi-camera pose update method of the present invention, in step S11, the SURF algorithm is applied to extract feature points from each image captured by each camera in the multi-camera.

[0014] In the above technical solution of the present invention, the SURF (Speeded-Up Robust Features) algorithm is a computer vision algorithm for image feature point detection and description, which aims to improve the efficiency of feature point detection and description while maintaining robustness to image scale, rotation, illumination changes, etc.

[0015] Of course, in some other embodiments, according to the specific usage situation, the SIFT (Scale-Invariant Feature Transform) algorithm and the ORB (Oriented FAST and Rotated BRIEF) algorithm can also be used to extract the feature points of the image.

[0016] Furthermore, in the multi-camera pose update method of the present invention, in step S13, a reprojection error energy function is established, and the reprojection error function is optimized to solve for the external parameters of each camera in the multi-camera; Among them, the reprojection error energy function is:

[0017] Among them, K represents the internal parameter of the camera; R d represents the external parameter of the camera, and R represents the rotation matrix, d represents the translation vector; t represents the time; X w represents the world point; x pi represents the i-th pixel point; . represents the distance between the i-th pixel point and the pixel point calculated by back-projecting the three-dimensional point coordinates.

[0018] Further, in the multi-camera pose update method of the present invention, in step S2, it specifically includes the steps: S21: At least one camera in the multi-camera collects continuous time-series images and extracts feature points in each frame of the image; S22: Between two consecutive frames of time-series images, obtain feature point pairs by matching; S23: Calculate the distance between the feature point pairs, construct a residual function, optimize the residual function to obtain the optimal homography matrix, and use the optimal homography matrix as the two-dimensional image offset of the adjacent two frames of time-series images.

[0019] Further, in the multi-camera pose update method of the present invention, the formula of the residual function is:

[0020] where, x i represents t the i-th feature point on the time-series image at time represents t the corresponding i-th feature point on the time-series image at time

[0021] In the above technical solution of the present invention, in step S23, when the distance between the feature point pairs is calculated, the reason for constructing the residual function and optimizing the residual function is that: constructing the residual function and optimizing it, and using the optimized optimal homography matrix as the two-dimensional image offset of the adjacent two frames of time-series images can effectively improve the camera pose estimation accuracy, which can enhance the algorithm robustness and effectively adapt to the dynamic environment of micro-vibrations.

[0022] Further, in the multi-camera pose update method of the present invention, in step S3, after obtaining the two-dimensional image offset of the adjacent two frames of time-series images, construct an incremental projection model to establish the connection between the two-dimensional image offset and the three-dimensional pose increment, and use the two-dimensional image offset to estimate the three-dimensional pose increment.

[0023] Further, in the multi-camera pose update method of the present invention, in step S4, the updated camera pose T new is:

[0024] where, R represents the camera rotation matrix before pose update; R Δ R represents the updated camera rotation matrix; drepresents the camera translation vector before pose update, d + R Δ d represents the camera translation vector after pose update.

[0025] Correspondingly, another object of the present invention is to disclose a multi-camera pose update system, which can simply execute the above multi-camera pose update method of the present invention to realize real-time update of the pose of the multi-camera, and specifically includes: A memory, a processor, and program instructions stored on the memory and executable on the processor, and when the processor executes the program instructions, each step in a multi-camera pose update method as described above in the present invention is realized.

[0026] At the same time, the present invention also discloses a storage medium, which stores a computer program, and when the above computer program is executed by a processor, each step in a multi-camera pose update method as described above in the present invention is realized.

[0027] The beneficial effects of the present invention are as follows: The present invention discloses a multi-camera pose update method, system and storage medium. The multi-camera pose update method can effectively obtain the initial pose reference values of each camera of the multi-camera, and then control at least one camera to collect continuous time-series images and calculate the two-dimensional picture offset between two consecutive frames of time-series images; based on these two-dimensional picture offsets, a mathematical relationship model with the three-dimensional pose increment can be constructed, so as to combine the three-dimensional pose increment and the initial pose reference value to perform real-time update of the camera pose. The multi-camera pose update method designed by the present invention is particularly suitable for micro-vibration scenarios such as electric towers, is easy to implement, and has the characteristics of high robustness and fast detection rate, and has good promotion prospects and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a block flow chart of the multi-camera pose update method of the present invention in one embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] To describe the technical content, the achieved object and the effect of the present invention in detail, the following is described in conjunction with the embodiments and with reference to the drawings.

[0030] See Figure 1 As shown, in this embodiment, the present invention discloses a multi-camera pose update method, which specifically includes the following steps S1-S4: S1: Obtain the initial pose reference values of each camera in the multi-camera; S2: At least one camera in the multi-camera collects continuous time-series images and calculates the two-dimensional picture offset between two consecutive frames of time-series images; S3: Obtain a 3D pose increment based on the above 2D image offset. S4: Update the camera pose by combining the above 3D pose increment and the initial pose reference value.

[0031] Based on this, when actually applying the above multi-camera pose update method of the present invention, the multi-camera pose update method can perform scene shooting based on a multi-camera, and is specifically applied to a transmission tower to be applicable to a micro-vibration environment. During actual application, the multi-camera pose update method can perform pose calculation on the multi-camera to obtain an initial pose reference value, and then control at least one camera in the multi-camera to collect consecutive time-series images, and calculate the 2D image offset between two consecutive frames of time-series images; based on these 2D image offsets, a mathematical relationship model with the 3D pose increment can be constructed, so as to combine the 3D pose increment and the initial pose reference value to perform real-time update of the camera pose.

[0032] Correspondingly, further refer to Figure 1 It can be seen that in this embodiment, in order to ensure that the multi-camera pose update method designed by the present invention can accurately obtain the initial pose reference value, when specifically performing the above step S1, the following steps S11-S13 can be sequentially included: S11: Use a multi-camera to synchronously shoot a preset area, and extract feature points from the images captured by each camera in the multi-camera.

[0033] Among them, in the above step S11, in order to effectively extract feature points, during actual application, the SURF algorithm can be specifically applied to extract feature points from each image captured by each camera in the multi-camera; among them, the above SURF (Speeded-Up Robust Features) algorithm is a computer vision algorithm for image feature point detection and description, which aims to improve the efficiency of feature point detection and description while maintaining robustness to image scale, rotation, illumination changes, etc.

[0034] Of course, in some other embodiments, according to the specific usage situation, the SIFT (Scale-Invariant Feature Transform) algorithm and the ORB (Oriented FAST and Rotated BRIEF) algorithm can also be correspondingly used to extract the feature points of the image.

[0035] S12: Perform feature point matching between the images captured by different cameras, determine the best matching pairs, and establish a feature matching relationship.

[0036] It should be noted that in the above step S11, when a feature point is detected, a descriptor will be generated for each feature point. The descriptor can be used to describe the local image content around the feature point. Therefore, when performing step S12, according to the obtained feature points, when performing feature point matching between the images captured by different cameras, by comparing the similarity between the descriptors, the best matching pairs can be determined to achieve feature point matching and establish a feature matching relationship.

[0037] S13: Calibrate the external parameters of each camera in the multi-camera system, and obtain the relative pose between the cameras to obtain the initial pose reference values of each camera in the multi-camera system.

[0038] In the above step S13, the external parameters of each camera in the multi-camera system can be specifically obtained by establishing a reprojection error energy function and optimizing the reprojection error function. Among them, the above reprojection error energy function is specifically:

[0039] Among them, K represents the internal parameter of the camera; R d represents the external parameter of the camera, and R represents the rotation matrix, d represents the translation vector; t represents the time; X w represents the world point; x pi represents the i-th pixel point; . represents the distance between the i-th pixel point and the pixel point calculated by the inverse of the three-dimensional point coordinates.

[0040] Thus, after obtaining the external parameters of each camera based on the above reprojection error energy function, the camera pose can be further initialized with the first image captured by each camera as the origin of the coordinate system to obtain the initial pose reference values of each camera in the multi-camera system.

[0041] Correspondingly, in the multi-camera pose update method of the present invention, after obtaining the initial pose reference values in the above step S1, the above step S2 is also required to calculate the two-dimensional image offset between the consecutive two-frame time series images for a single camera, which specifically includes the following steps S21 - S23: S21: At least one camera in the multi-camera system captures consecutive time series images and extracts the feature points in each frame of the image; S22: Between the consecutive two-frame time series images, match to obtain feature point pairs; S23: Calculate the distances between the above feature point pairs, construct a residual function, and optimize the residual function to obtain the optimal homography matrix, and use the above optimal homography matrix as the two-dimensional image offset of the time series images of two adjacent frames.

[0042] In the above technical solution of the present invention, the formula of the above residual function is specifically:

[0043] where x i represents t the i-th feature point on the time series image at time represents t the corresponding i-th feature point on the time series image at time + 1, and Δ H represents the change amount of the homography matrix.

[0044] Thus, in the present invention, by constructing a residual function and optimizing the residual function to obtain the optimal homography matrix, the optimal homography matrix is the two-dimensional image offset of two adjacent frames. Among them, in the above step S22, by comparing the feature points or regions in the time series images of two consecutive frames, the change of the camera pose caused by micro-vibration can be quantitatively analyzed, which is crucial for capturing the small change of the pose and provides necessary quantitative information for pose update.

[0045] Correspondingly, in the present invention, after obtaining the two-dimensional image offset of the time series images of two adjacent frames, an incremental projection model can also be constructed to establish the connection between the two-dimensional image offset and the three-dimensional pose increment, so that the three-dimensional pose increment can be estimated using the two-dimensional offset.

[0046] In the actual application process, the Taylor expansion approximation projection model can be used to obtain the following projection model ; based on this, using the Jacobian matrix J ΔH represents the relationship between the change amount of the homography matrix and the three-dimensional pose increment, and the set incremental projection model is specifically: .

[0047] Thus, the following residual function formula is used to define the residual function in the above step S2 to measure the difference between the observed two-dimensional image offset and the two-dimensional image offset predicted by the three-dimensional pose increment as:

[0048] where represents the observed change of the homography matrix of the residual function, represents the change of the homography matrix predicted by the three-dimensional pose increment Based on the above settings, when the residual function reaches the minimum value The three-dimensional pose increment at this time is the actual camera pose increment.

[0049] Therefore, based on the obtained three-dimensional pose increment and the initial pose reference value of the corresponding camera obtained in step S1, the camera pose can be effectively updated. The updated camera pose T new is:

[0050] where, R represents the camera rotation matrix before pose update; R Δ R represents the updated camera rotation matrix, which is specifically obtained by multiplying the rotation matrix before update by the rotation increment; d represents the camera translation vector before pose update, d + R Δ d represents the camera translation vector after pose update.

[0051] Therefore, through the above process, the two-dimensional picture offset can be mapped into the three-dimensional space to obtain the three-dimensional pose increment, realizing the accurate update of the pose, which takes into account the linear and non-linear components of the pose change to ensure the accuracy and robustness of the pose update in a micro-vibration environment.

[0052] Correspondingly, in the present invention, a multi-camera pose update system is also disclosed. The multi-camera pose update system can simply execute the above multi-camera pose update method of the present invention to realize the real-time update of the multi-camera pose, which specifically includes: a memory, a processor, and program instructions stored on the memory and executable on the processor. When the processor executes the program instructions, each step in a multi-camera pose update method as described above in the present invention is realized.

[0053] At the same time, the present invention also discloses a storage medium, which stores a computer program. When the above computer program is executed by a processor, each step in a multi-camera pose update method as described above in the present invention is realized.

[0054] As can be seen from the above description, the multi-camera pose update method designed by the present invention can first obtain an initial pose reference value, and then control at least one camera in the multi-camera to collect continuous time-series images, and calculate the two-dimensional image offset between two consecutive frames of time-series images; based on these two-dimensional image offsets, a mathematical relationship model with the three-dimensional pose increment can be constructed, so as to combine the three-dimensional pose increment and the initial pose reference value to realize the real-time update of the camera pose. It can be applied to micro-vibration scenarios such as electric towers, and has the advantages of high robustness, fast update rate and high accuracy, thus providing accurate external parameter information for subsequent multi-camera three-dimensional reconstruction, significantly improving the accuracy and reliability of subsequent three-dimensional reconstruction results, and having good promotion prospects and application values.

[0055] Correspondingly, the multi-camera pose update system and storage medium designed by the present invention can be used to execute the above multi-camera pose update method, and they also have the above advantages and beneficial effects.

[0056] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in the related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A multi-camera pose updating method, characterized in that: Includes steps: S1: Obtain the initial pose reference value of each camera in the multi-camera; S2: at least one camera in the multi-camera captures continuous time series images, and calculates a two-dimensional image offset between two consecutive frames of the time series images; S3: Obtaining a three-dimensional pose increment based on the two-dimensional image offset; S4: updating the camera pose by combining the three-dimensional pose increment and the initial pose reference value.

2. The multi-camera pose updating method according to claim 1, characterized in that: In step S1, the steps are specifically included: S11: using a multi-camera to synchronously shoot a preset area, and extracting feature points from images shot by each camera in the multi-camera; S12: performing feature point matching between images taken by different cameras, determining the best matching pair, and establishing a feature matching relationship; S13: Calibrate the external parameters of each camera in the multi-camera, and obtain the relative pose between the cameras to obtain the initial pose reference value of each camera in the multi-camera.

3. The multi-camera pose updating method according to claim 2, characterized in that: In step S11, the SURF algorithm is used to extract feature points from each image taken by each camera in the multi-camera.

4. The multi-camera pose updating method according to claim 2, characterized in that: In step S13, a reprojection error energy function is established, and the reprojection error function is optimized to obtain the external parameters of each camera in the multi-camera; Wherein, the reprojection error energy function is: in, K Indicates the internal parameters of the camera; R ] represents the external parameters of the camera, and R represents the rotation matrix, d represents the translation vector; t Indicates the moment; X w represents a world point; x pi Represents the i-th pixel; . Represents the distance between the i-th pixel and the pixel calculated by inverse calculation of the 3D point coordinates.

5. The multi-camera pose updating method according to claim 1, characterized in that: In step S2, the steps are specifically included: S21: at least one camera among the multi-cameras collects continuous time series images and extracts feature points in each frame of the image; S22: Matching between two consecutive frames of time series images to obtain feature point pairs; S23: Calculate the distance between the feature point pairs, construct a residual function, optimize the residual function to obtain an optimal homography matrix, and use the optimal homography matrix as a two-dimensional image offset of the time series images of two adjacent frames.

6. The multi-camera pose updating method according to claim 5, characterized in that: In step S23, the formula of the residual function is: in, x i express t The i-th feature point on the time series image at time , express t The i-th feature point corresponding to the time series image at time +1, ΔH represents the change in the homography matrix.

7. The multi-camera pose updating method according to claim 6, characterized in that: In step S3, after obtaining the two-dimensional image offset of the time series images of two adjacent frames, an incremental projection model is constructed to establish a connection between the two-dimensional image offset and the three-dimensional pose increment, and the three-dimensional pose increment is estimated using the two-dimensional image offset.

8. The multi-camera pose updating method according to claim 1, characterized in that: In step S4, the updated camera pose T new for: in, R Represents the camera rotation matrix before the pose is updated; R Δ R Represents the updated camera rotation matrix; d Represents the camera translation vector before the pose is updated, d + R Δ d Represents the camera translation vector after the pose is updated.

9. A multi-camera posture update system, comprising a memory, a processor, and program instructions stored in the memory and executable on the processor, characterized in that: The processor is configured to execute the multi-camera pose updating method as described in any one of claims 1-8 when running the program instructions.

10. A storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the multi-camera pose updating method as described in any one of claims 1 to 8.

Images