Multi-view camera pose determination method, system and device, medium and product

Abstract

The invention discloses a multi-view camera pose determination method, system and device, a medium and a product, and relates to the field of computer vision, and the method comprises the steps: constructing coordinate systems of a motion platform at different moments; determining transformation of each camera relative to a coordinate system and a direction vector and a distance vector of a Pluecker straight line corresponding to an image matching point; determining an initial essential matrix between the initial coordinate system and the transformed coordinate system based on a multi-camera pose solving linear method; determining an initial rotation matrix through the initial essential matrix; determining an initial translation vector based on a total least square method or a scaling factor recovery method; further determining an intermediate rotation matrix and an intermediate translation vector of each camera at different moments; finally, determining an optimized rotation matrix and an optimized translation vector between the initial coordinate system and the transformed coordinate system based on Simpson errors and an LM algorithm; according to the invention, the precision and robustness of determining the pose of the multi-view camera can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, system, device, medium and product for determining the position and posture of a multi-camera. Background Art

[0002] With the rapid development of computer vision and robotics, visual localization, as one of the core technologies in robotic perception systems, has been widely used in fields such as autonomous driving, drone navigation, and augmented reality. The main task of visual localization is to infer the relative position and pose of a camera or robot in an environment from image data captured by the camera. Traditional single-camera visual localization methods can achieve a certain level of accuracy, but in practical applications, especially in dynamic and complex environments, the positioning accuracy and robustness of single-camera systems are often limited.

[0003] To address this problem, visual localization methods based on multi-camera systems have emerged. By leveraging the collaborative work of multiple cameras, multi-camera systems can obtain more spatial information from different perspectives, thereby improving localization accuracy and robustness. This is particularly true in complex scenes, occluded environments, or dynamic environments. However, efficiently utilizing multi-camera systems to accurately determine position and pose remains a major challenge in computer vision.

[0004] Commonly used techniques for solving pose problems in multi-camera systems include the 6-point method, the 17-point method, and several optimization-based algorithms. Among these methods, the 17-point method, as a linear algorithm, is widely used due to its ease of implementation and high computational efficiency. However, in practical applications, the 17-point method still faces many challenges, particularly when solving for translation vectors, where computational instability often occurs, impacting overall positioning accuracy. To improve solution accuracy, optimization techniques such as least squares methods are often employed to refine the initial solution. However, the practical effects of least squares optimization based on the 17-point method often fail to meet positioning accuracy requirements, and the accuracy of the optimized results is even lower than that of the initial values ​​obtained using the robust estimation framework.

[0005] Therefore, based on the above problems, there is an urgent need to provide a new linear determination method for relative pose, which can overcome the instability of the 17-point method in solving the translation vector and improve the accuracy and robustness of multi-camera pose determination. Summary of the Invention

[0006] The purpose of this application is to provide a multi-camera pose determination method, system, device, medium and product that can improve the accuracy and robustness of multi-camera pose determination.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a method for determining the pose of a multi-camera, comprising:

[0009] Constructing coordinate systems of the motion platform at different moments; the different moments include: an initial moment and a transformed moment; the motion platform is provided with a plurality of cameras; the coordinate systems include: an initial coordinate system and a transformed coordinate system;

[0010] Determine the transformation of each camera relative to the coordinate system; and determine the direction vector and distance vector of the Plücker line corresponding to the image matching point;

[0011] Determine an initial essential matrix between the initial coordinate system and the transformed coordinate system based on a multi-camera pose solving linear method according to the transformation and the direction vector and the distance vector of the Plücker line; and determine an initial rotation matrix between the initial coordinate system and the transformed coordinate system using the initial essential matrix;

[0012] Determining an initial translation vector between the initial coordinate system and the transformed coordinate system based on the total least squares method or the scaling factor recovery method according to the initial essential matrix and the initial rotation matrix;

[0013] Determining an intermediate rotation matrix and an intermediate translation vector for each camera at different times based on the initial rotation matrix and the initial translation vector;

[0014] According to the intermediate rotation matrix and the intermediate translation vector, the initial rotation matrix and the initial translation vector are optimized based on Simpson error and LM algorithm to determine the optimized rotation matrix and optimized translation vector between the initial coordinate system and the transformed coordinate system.

[0015] Optionally, determining the transformation of each camera relative to the coordinate system; and determining the direction vector and distance vector of the Plücker line corresponding to the image matching point specifically includes:

[0016] Using the formula x i =(R i ·u i ) / ||R i ·u i ||Determine camera C i The direction vector x of the Plücker line corresponding to the matching point in the image taken at the initial moment i ;

[0017] Using the formula q i =t i ×x i Determine camera C i The distance vector q to the Plücker line corresponding to the matching point in the image taken at the initial moment i ;

[0018] Using the formula x j =(R j ·u j ) / ||R j ·u j ||Determine camera C j The direction vector x of the Plücker line corresponding to the matching point in the image taken at the time after the transformation j ;

[0019] Using the formula q j =t j ×x j Determine camera C j The distance vector q to the Plücker line corresponding to the matching point in the image taken at the time after the transformation j ;

[0020] Among them, R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, u i For camera C i Normalized coordinates of the matching points in the image captured at the initial time, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R j For camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, u j For camera C j Normalized coordinates of the matching points in the image captured at the time after the transformation, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed moment, · is the dot product between vectors, and × is the cross product between vectors.

[0021] Optionally, the process of determining the initial essential matrix includes:

[0022] Using the formula Determine the initial essential matrix E;

[0023] Among them, T is the transpose, x i For camera C i The direction vector of the Plücker line corresponding to the matching point in the image taken at the initial moment, x j For camera C j The direction vector of the Plücker line corresponding to the matching point of the image taken at the time after the transformation, R is the initial rotation matrix between the initial coordinate system and the transformed coordinate system, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, t j For camera C jThe translation vector relative to the transformed coordinate system at the transformed time instant.

[0024] Optionally, determining the initial translation vector between the initial coordinate system and the transformed coordinate system based on the total least squares method or the scaling factor recovery method according to the initial essential matrix and the initial rotation matrix specifically includes:

[0025] Using the formula Determine the total least squares method; where C is the construction matrix, A is the first coefficient matrix, b is the first coefficient vector, and T is the transpose;

[0026] Using the formula Determine the scaling factor recovery method; where, is the second coefficient matrix, used to determine the scaling factor, k is the scaling factor, b n×1 is the second coefficient vector, and n is the row dimension of the matrix and vector.

[0027] Optionally, determining the intermediate rotation matrix and intermediate translation vector of each camera at different times based on the initial rotation matrix and the initial translation vector specifically includes:

[0028] Using the formula and formula Determine the intermediate rotation matrix between each camera at different times and the intermediate translation vector

[0029] Among them, -1 is reversal, R is the initial rotation matrix, t is the initial translation vector, R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R j For camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed time instant.

[0030] Optionally, the Simpson error is determined as follows:

[0031] Using the formula Determine Simpson's error;

[0032] Where D is the Simpson error, T' is the optimized transformation matrix between the initial coordinate system and the transformed coordinate system, including the optimized rotation matrix and the optimized translation vector, p is the image matching point, is the intermediate essential matrix, x i For camera C i The direction vector of the Plücker line corresponding to the matching point in the image taken at the initial moment, x j For camera C j The direction vector of the Plücker line corresponding to the image matching point taken at the time after the transformation, T is the transpose, subscript 1 is the first element of the vector, subscript 2 is the second element of the vector, is the intermediate translation vector, To perform outer product operation Converted into an antisymmetric matrix, is the intermediate rotation matrix.

[0033] In a second aspect, the present application provides a multi-camera pose determination system, comprising:

[0034] A motion platform coordinate system determination module is used to construct a coordinate system of the motion platform at different moments; the different moments include: an initial moment and a transformed moment; the motion platform is provided with multiple cameras; the coordinate system includes: an initial coordinate system and a transformed coordinate system;

[0035] A Plücker line determination module is used to determine the transformation of each camera relative to the coordinate system; and determine the direction vector and distance vector of the Plücker line corresponding to the image matching point;

[0036] an initial rotation matrix determination module, configured to determine an initial essential matrix between the initial coordinate system and the transformed coordinate system based on the transformation and the direction vector and the distance vector of the Plücker line, and based on a multi-camera pose solving linear method; and determine an initial rotation matrix between the initial coordinate system and the transformed coordinate system using the initial essential matrix;

[0037] an initial translation vector determination module, configured to determine, based on the initial essential matrix and the initial rotation matrix, an initial translation vector between the initial coordinate system and the transformed coordinate system using a total least squares method or a scaling factor recovery method;

[0038] An intermediate rotation matrix and intermediate translation vector determination module, configured to determine an intermediate rotation matrix and an intermediate translation vector between different moments of each camera based on the initial rotation matrix and the initial translation vector;

[0039] The optimized rotation matrix and optimized translation vector determination module is used to optimize the initial rotation matrix and the initial translation vector based on the intermediate rotation matrix and the intermediate translation vector based on the Simpson error and LM algorithm, and determine the optimized rotation matrix and optimized translation vector between the initial coordinate system and the transformed coordinate system.

[0040] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the multi-camera pose determination methods described above.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the multi-camera pose determination methods described above.

[0042] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the multi-camera pose determination methods described above.

[0043] According to the specific embodiments provided in this application, this application has the following technical effects:

[0044] The present application provides a method, system, device, medium and product for determining the posture of a multi-camera. By introducing the Total Least Squares (TLS) method and the scaling factor recovery method, the stability of the initial translation vector determination is improved, the influence of image noise or numerical errors on the calculation results is effectively reduced, and the performance in accuracy is better than that of traditional methods. The intermediate rotation matrix and intermediate translation vector between each camera at different times are determined, and the Simpson error and LM algorithm are introduced to optimize the initial rotation matrix and initial translation vector, further improving the robustness and reliability of the determination results. The present application not only achieves improvement in accuracy, but also has stronger adaptability in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is a flow chart of a method for determining the position and posture of multiple cameras in one embodiment of the present application;

[0047] Figure 2 Schematic diagram of the process of determining the motion platform coordinate system and image matching points in one embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of a process for determining an intermediate rotation matrix and an intermediate translation vector in one embodiment of the present application;

[0049] Figure 4 Schematic diagram showing how the accuracy of the linear solution method varies with image noise in one embodiment of the present application ( Figure 4 Part (a) is a schematic diagram of rotation error. Figure 4 Part (b) is a schematic diagram of translation error);

[0050] Figure 5 Schematic diagram showing how the accuracy of the optimization method varies with image noise in an embodiment of the present application ( Figure 5 Part (a) is a schematic diagram of rotation error. Figure 5 Part (b) is a schematic diagram of translation error). DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0053] The traditional 17-point method suffers from poor stability and poor least-squares optimization when solving for translation vectors. The multi-camera pose determination method proposed in this application can better handle pose determination in multi-camera systems, offering greater robustness and reliability. This application provides a more reliable solution for multi-camera pose estimation in complex environments, not only improving the accuracy of translation vector calculations but also demonstrating greater adaptability in practical applications, providing a more accurate solution for determining multi-camera poses.

[0054] In an exemplary embodiment, Figure 1 As shown, a method for determining the position and posture of a multi-camera is provided, including the following S1 to S6.

[0055] S1: Construct the coordinate system of the motion platform at different times.

[0056] In an exemplary embodiment, a plurality of cameras are mounted on a motion platform to construct a coordinate system of the motion platform. The position of the motion platform changes from O1 to O2 at different moments. The different moments include: an initial moment and a transformed moment. That is, the coordinate system of the motion platform at the initial moment is O1, referred to as the initial coordinate system, and the coordinate system of the motion platform at the transformed moment is O2, referred to as the transformed coordinate system.

[0057] S2: Determine the transformation of each camera relative to the coordinate system; and determine the direction vector and distance vector of the Plücker line corresponding to the image matching point.

[0058] In an exemplary embodiment, Figure 2 As shown, let the initial transformation matrix between the initial coordinate system O1 and the transformed coordinate system O2 be T = [R t], including the initial rotation matrix R and the initial translation vector t, then the camera C i The transformation relative to the initial coordinate system at the initial moment is [R i t i ], Camera C j The transformation relative to the transformed coordinate system at the transformed moment is [R j t j ]; where R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R j For camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed time instant.

[0059] Assume that there is a set of cameras C at different times i and C j Matching points of captured images The normalized coordinates corresponding to the image matching points are The image matching point is a 3D point p in space. From a geometric point of view, the 3D point can be regarded as two Plücker lines L i and L j The intersection of Plücker line L i For camera C i The line connecting the optical center and the 3D point, the Plücker line L j For camera C j The line connecting the optical center and the 3D point.

[0060] Two Plücker lines L i and L j It can be expressed by direction vector and moment vector. The formula for determining the two Plücker lines is:

[0061] L i =[x i T q i T ] T .

[0062] L j =[x j T q j T ] T .

[0063] Camera C i The direction vector x of the Plücker line corresponding to the matching point in the image taken at the initial moment i The formula for determining is:

[0064] x i =(R i ·u i ) / ||R i ·u i ||.

[0065] Camera C i The distance vector q to the Plücker line corresponding to the matching point in the image taken at the initial moment i The formula for determining is:

[0066] q i =t i ×x i .

[0067] Camera C j The direction vector x of the Plücker line corresponding to the matching point in the image taken at the time after the transformation j The formula for determining is:

[0068] x j =(R j ·u j ) / ||R j ·u j ||.

[0069] Camera C j The distance vector q to the Plücker line corresponding to the matching point in the image taken at the time after the transformation j The formula for determining is:

[0070] q j =t j ×x j .

[0071] Among them, R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, u i For camera C i Normalized coordinates of the matching points in the image captured at the initial time, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R jFor camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, u j For camera C j Normalized coordinates of the matching points in the image captured at the time after the transformation, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed moment, · is the dot product between vectors, and × is the cross product between vectors.

[0072] S3: According to the transformation and the direction vector and distance vector of the Plücker line, an initial essential matrix between the initial coordinate system and the transformed coordinate system is determined based on a linear method for solving multi-camera poses; and an initial rotation matrix between the initial coordinate system and the transformed coordinate system is determined through the initial essential matrix.

[0073] Due to L i and L j are the Plücker line expressions in the initial coordinate system and the transformed coordinate system respectively, so it is necessary to convert L i and L j Convert to the same coordinate system and set the Plücker line L i The expression for transforming from the initial coordinate system O1 to the transformed coordinate system O2 is L' i =[x' i T q' i T ] T , the conversion relationship is as follows:

[0074]

[0075] Among them, x' i is the Plücker line L in the transformed coordinate system O2 i The direction vector, q' i is the Plücker line L in the transformed coordinate system O2 i The moment vector, L' i is the Plücker line L in the transformed coordinate system O2 i , R is the initial rotation vector, [t] × is the antisymmetric matrix of the initial translation vector, and T is the transpose.

[0076] In the same coordinate system O2, the two Plücker lines L' i and L j The necessary and sufficient conditions for intersection are:

[0077] x' i T q j +x j T q i'=0.

[0078] Through the transformation relationship of the Plücker line and the two Plücker lines L' i and L j The necessary and sufficient conditions for intersection and the generalized limit constraint formula are as follows:

[0079] (Rx i ) T q j +x j T R i +x j T [t] × Rx i =0.

[0080] The matrix form of the generalized limit constraint is as follows:

[0081]

[0082] Set the camera C i The distance vector q to the Plücker line corresponding to the matching point in the image taken at the initial moment i and Camera C j The distance vector q to the Plücker line corresponding to the matching point in the image taken at the time after the transformation j Substituting into the generalized limit constraint formula, we get the processed generalized limit constraint formula:

[0083]

[0084] Among them, E is the initial essential matrix.

[0085] The processed generalized limit constraint formula is converted into the following form to obtain the converted generalized limit constraint formula:

[0086] A E vec(E)+A R vec(R)=0.

[0087] Among them, vec() is the vector after stacking the matrix vertically by column, A E is the coefficient matrix corresponding to vec(E), A R is the coefficient matrix corresponding to vec(R).

[0088] The transformed generalized limit constraint formula is further simplified to obtain the simplified generalized limit constraint formula:

[0089] Bvec(E)=0.

[0090] Among them, B is the coefficient matrix, which is given by the formula Sure, A R The generalized inverse matrix of , I is the identity matrix.

[0091] Through SVD decomposition, the initial essential matrix E can be solved, and then the initial rotation matrix R can be decomposed. Unlike the initial rotation matrix R, the initial translation vector t is not decomposed from the essential matrix E.

[0092] Substituting the initial rotation matrix R into the generalized limit constraint, we obtain the equation based on the translation vector, which is calculated as follows:

[0093] At=b.

[0094] Where A is the first coefficient matrix, which is obtained by simplifying the generalized limit constraint formula and is used to recover the initial translation vector t, and b is the corresponding first coefficient vector.

[0095] By using SVD decomposition again, the initial translation vector t can be restored, and the restored initial translation vector t includes its scale information.

[0096] S4: According to the initial essential matrix and the initial rotation matrix, an initial translation vector between the initial coordinate system and the transformed coordinate system is determined based on the total least squares method or the scaling factor recovery method.

[0097] Due to the influence of image noise and numerical errors, the solution of the initial rotation matrix and initial translation vector is often unstable. The main reason is that during the solution process, the condition number of matrix A is large, and the solution of the equation based on the translation vector is unstable, causing matrix A to become ill-conditioned and the calculation accuracy to decrease.

[0098] The condition number κ(A) of matrix A is defined as follows:

[0099] κ(A)=||A||A -1 .

[0100] In order to address the problem of unstable translation vector solution, the following two methods are proposed to improve the stability of the calculation:

[0101] 1. Total least squares method.

[0102] Use the total least squares method to construct a new 4×4 coefficient matrix, called the construction matrix C, and use the SVD decomposition method to solve the minimum eigenvalue λ of the construction matrix C min and the corresponding eigenvector v min , satisfying Cv min =λ min v min , the expression for constructing matrix C is as follows:

[0103]

[0104] If the eigenvector v min If the last element of is not 0, the eigenvector is normalized, that is, the eigenvector is divided by its last element, and then the first three components are taken as the final value of the translation vector. The calculation formula is as follows:

[0105]

[0106] in, is the normalized vector, v min (4) is v min The fourth element of .

[0107] 2. Scaling factor recovery method.

[0108] When the condition number exceeds the preset threshold, the initial rotation matrix R and the homogeneous translation vector can be decomposed by the initial essential matrix E Let k be the scaling factor, and let The initial translation vector is The homogeneous translation vector Substituting into the formula At = b, we determine the scaling factor k to fully restore the initial translation vector. The calculation formula is as follows:

[0109]

[0110] in, is the second coefficient matrix, used to determine the scaling factor, b n×1 is the second coefficient vector, and n is the row dimension of the matrix and vector.

[0111] S5: Determine the intermediate rotation matrix and intermediate translation vector of each camera at different times based on the initial rotation matrix and the initial translation vector.

[0112] In an exemplary embodiment, Figure 3 As shown, the initial rotation matrix R and the initial translation vector t are used to determine the camera C at different times i and Camera C j The intermediate rotation matrix between and the intermediate translation vector The calculation formula is as follows:

[0113]

[0114] Among them, -1 is reversal, R is the initial rotation matrix, t is the initial translation vector, R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R jFor camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed time instant.

[0115] Using the intermediate rotation matrix and the intermediate translation vector Solve the corresponding intermediate essential matrix The calculation formula is as follows:

[0116]

[0117] in, To perform outer product operation Convert to an antisymmetric matrix.

[0118] S6: According to the intermediate rotation matrix and the intermediate translation vector, the initial rotation matrix and the initial translation vector are optimized based on the Simpson error and the LM algorithm to determine the optimized rotation matrix and the optimized translation vector between the initial coordinate system and the transformed coordinate system.

[0119] In the robust estimation framework, it is usually necessary to optimize the initial rotation matrix and initial translation vector based on the selected internal points. A common practice is to use the least squares method for global optimization to improve the accuracy of the solution. In addition, the optimization of the initial rotation matrix and initial translation vector can also be used for local optimization in the robust estimation framework, which helps to effectively reduce the running time and improve the efficiency of the algorithm. As a linear solution method, the 17-point method can be directly incorporated into the least squares framework without the need for additional steps. The only difference between the least squares based on the 17-point method and the 17-point method is that the number of points is expanded from 17 to n, and the dimensions of the matrix and vector in the formula At = b are increased from 17 to n.

[0120] In an exemplary embodiment, Figure 4 As shown in the figure, due to the instability of the least squares solution of the 17-point method, this application uses the Simpson error and LM algorithm to optimize the multi-camera pose determination process.

[0121] Calculate the Simpson error D corresponding to each image matching point. The calculation formula is as follows:

[0122]

[0123] Where T' is the optimized transformation matrix between the initial coordinate system and the transformed coordinate system, including the optimized rotation matrix and the optimized translation vector, p is the image matching point, subscript 1 is the first element of the vector, and subscript 2 is the second element of the vector.

[0124] The calculation formula for T' is as follows:

[0125]

[0126] Where w is the number of camera pairs involved in the generalized camera model, q is the camera number, is the set of inliers of the q-th camera pair.

[0127] The initial rotation matrix and initial translation vector are iteratively optimized by the LM algorithm to determine the optimized rotation matrix and optimized translation vector between the initial coordinate system and the transformed coordinate system.

[0128] In an exemplary embodiment, Figure 4 and Figure 5 As shown in the figure, compared with the traditional 17-point method, this application introduces the total least squares method, which significantly improves the numerical stability of solving the changing translation vector and effectively reduces the impact of image noise and numerical errors on the results. Compared with the least squares optimization method based on the 17-point method, the pose determination method proposed in this application shows better performance in terms of accuracy, can better handle pose solution problems in multi-camera systems, and has higher robustness and reliability.

[0129] In an exemplary embodiment, a multi-camera pose determination system is provided, the multi-camera pose determination system comprising:

[0130] A motion platform coordinate system determination module is used to construct a coordinate system of the motion platform at different moments; the different moments include: an initial moment and a transformed moment; the motion platform is provided with multiple cameras; the coordinate system includes: an initial coordinate system and a transformed coordinate system;

[0131] A Plücker line determination module is used to determine the transformation of each camera relative to the coordinate system; and determine the direction vector and distance vector of the Plücker line corresponding to the image matching point;

[0132] an initial rotation matrix determination module, configured to determine an initial essential matrix between the initial coordinate system and the transformed coordinate system based on the transformation and the direction vector and the distance vector of the Plücker line, and based on a multi-camera pose solving linear method; and determine an initial rotation matrix between the initial coordinate system and the transformed coordinate system using the initial essential matrix;

[0133] an initial translation vector determination module, configured to determine, based on the initial essential matrix and the initial rotation matrix, an initial translation vector between the initial coordinate system and the transformed coordinate system using a total least squares method or a scaling factor recovery method;

[0134] An intermediate rotation matrix and intermediate translation vector determination module, configured to determine an intermediate rotation matrix and an intermediate translation vector between different moments of each camera based on the initial rotation matrix and the initial translation vector;

[0135] The optimized rotation matrix and optimized translation vector determination module is used to optimize the initial rotation matrix and the initial translation vector based on the intermediate rotation matrix and the intermediate translation vector based on the Simpson error and LM algorithm, and determine the optimized rotation matrix and optimized translation vector between the initial coordinate system and the transformed coordinate system.

[0136] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multi-camera pose determination data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multi-camera pose determination method is implemented.

[0137] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0138] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0139] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0141] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0142] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0143] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for determining the pose of a multi-camera, characterized in that: include: Construct the coordinate system of the motion platform at different times; The different moments include: an initial moment and a transformed moment; the motion platform is provided with a plurality of cameras; the coordinate system includes: an initial coordinate system and a transformed coordinate system; Determine the transformation of each camera relative to the coordinate system; and determine the direction vector and distance vector of the Plücker line corresponding to the image matching point; Determine an initial essential matrix between the initial coordinate system and the transformed coordinate system based on a multi-camera pose solving linear method according to the transformation and the direction vector and the distance vector of the Plücker line; and determine an initial rotation matrix between the initial coordinate system and the transformed coordinate system using the initial essential matrix; Determining an initial translation vector between the initial coordinate system and the transformed coordinate system based on the total least squares method or the scaling factor recovery method according to the initial essential matrix and the initial rotation matrix; Determining an intermediate rotation matrix and an intermediate translation vector for each camera at different times based on the initial rotation matrix and the initial translation vector; According to the intermediate rotation matrix and the intermediate translation vector, the initial rotation matrix and the initial translation vector are optimized based on Simpson error and LM algorithm to determine the optimized rotation matrix and optimized translation vector between the initial coordinate system and the transformed coordinate system.

2. The method for determining the pose of a multi-camera according to claim 1, wherein: said determining a transformation of each camera relative to said coordinate system; And determine the direction vector and distance vector of the Plücker line corresponding to the image matching point, specifically including: Using the formula x i =(R i ·u i ) / ||R i ·u i ||Determine camera C i The direction vector x of the Plücker line corresponding to the matching point in the image taken at the initial moment i ; Using the formula q i =t i ×x i Determine camera C i The distance vector q to the Plücker line corresponding to the matching point in the image taken at the initial moment i ; Using the formula x j =(R j ·u j ) / ||R j ·u j ||Determine camera C j The direction vector x of the Plücker line corresponding to the matching point in the image taken at the time after the transformation j ; Using the formula q j =t j ×x j Determine camera C j The distance vector q to the Plücker line corresponding to the matching point in the image taken at the time after the transformation j ; Among them, R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, u i For camera C i Normalized coordinates of the matching points in the image captured at the initial time, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R j For camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, u j For camera C j Normalized coordinates of the matching points in the image captured at the time after the transformation, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed moment, · is the dot product between vectors, and × is the cross product between vectors.

3. The method for determining the pose of a multi-camera according to claim 1, wherein: The process of determining the initial essential matrix includes: Using the formula Determine the initial essential matrix E; Among them, T is the transpose, x i For camera C i The direction vector of the Plücker line corresponding to the matching point in the image taken at the initial moment, x j For camera C j The direction vector of the Plücker line corresponding to the matching point of the image taken at the time after the transformation, R is the initial rotation matrix between the initial coordinate system and the transformed coordinate system, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed time instant.

4. The method for determining the position and posture of a multi-camera according to claim 1, wherein: The determining, based on the initial essential matrix and the initial rotation matrix and based on a total least squares method or a scaling factor recovery method, an initial translation vector between the initial coordinate system and the transformed coordinate system specifically includes: Using the formula Determine the total least squares method; where C is the construction matrix, A is the first coefficient matrix, b is the first coefficient vector, and T is the transpose; Using the formula Determine the scaling factor recovery method; where, is the second coefficient matrix, used to determine the scaling factor, k is the scaling factor, b n×1 is the second coefficient vector, and n is the row dimension of the matrix and vector.

5. The method for determining the position and posture of a multi-camera according to claim 1, wherein: Determining the intermediate rotation matrix and the intermediate translation vector of each camera at different times based on the initial rotation matrix and the initial translation vector specifically includes: Using the formula and formula Determine the intermediate rotation matrix between each camera at different times and the intermediate translation vector Among them, -1 is reversal, R is the initial rotation matrix, t is the initial translation vector, R i For camera C i The rotation matrix relative to the initial coordinate system at the initial moment, t i For camera C i The translation vector relative to the initial coordinate system at the initial moment, R j For camera C j The rotation matrix relative to the transformed coordinate system at the transformed moment, t j For camera C j The translation vector relative to the transformed coordinate system at the transformed time instant.

6. The method for determining the position and posture of a multi-camera according to claim 1, wherein: The determination process of the Simpson error is: Using the formula Determine the Simpson error; Where D is the Simpson error, T' is the optimized transformation matrix between the initial coordinate system and the transformed coordinate system, including the optimized rotation matrix and the optimized translation vector, p is the image matching point, is the intermediate essential matrix, x i For camera C i The direction vector of the Plücker line corresponding to the matching point in the image taken at the initial moment, x j For camera C j The direction vector of the Plücker line corresponding to the image matching point taken at the time after the transformation, T is the transpose, subscript 1 is the first element of the vector, subscript 2 is the second element of the vector, is the intermediate translation vector, To perform outer product operation Converted into an antisymmetric matrix, is the intermediate rotation matrix.

7. A multi-camera pose determination system, characterized in that: include: The motion platform coordinate system determination module is used to construct the coordinate system of the motion platform at different times; The different moments include: an initial moment and a transformed moment; the motion platform is provided with a plurality of cameras; the coordinate system includes: an initial coordinate system and a transformed coordinate system; A Plücker line determination module is used to determine the transformation of each camera relative to the coordinate system; and determine the direction vector and distance vector of the Plücker line corresponding to the image matching point; an initial rotation matrix determination module, configured to determine an initial essential matrix between the initial coordinate system and the transformed coordinate system based on the transformation and the direction vector and the distance vector of the Plücker line, and based on a multi-camera pose solving linear method; and determine an initial rotation matrix between the initial coordinate system and the transformed coordinate system using the initial essential matrix; an initial translation vector determination module, configured to determine, based on the initial essential matrix and the initial rotation matrix, an initial translation vector between the initial coordinate system and the transformed coordinate system using a total least squares method or a scaling factor recovery method; An intermediate rotation matrix and intermediate translation vector determination module, configured to determine an intermediate rotation matrix and an intermediate translation vector between different moments of each camera based on the initial rotation matrix and the initial translation vector; The optimized rotation matrix and optimized translation vector determination module is used to optimize the initial rotation matrix and the initial translation vector based on the intermediate rotation matrix and the intermediate translation vector based on the Simpson error and LM algorithm, and determine the optimized rotation matrix and optimized translation vector between the initial coordinate system and the transformed coordinate system.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to implement the multi-camera pose determination method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining the pose of a multi-camera according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining the pose of a multi-camera according to any one of claims 1 to 6 is implemented.

Images