Multi-camera pose determination method, device, apparatus, vehicle and storage medium

Abstract

The application relates to a multi-camera pose determination method and device, equipment, a vehicle and a storage medium, and relates to the field of cameras. The method comprises the following steps: determining three groups of images according to image data; the first group of images are images with the same two-dimensional features captured by a target camera at different time points; the second group of images are images captured by the target camera and a co-view overlap camera at the same time point; and the third group of images are images captured by the target camera and a non-co-view overlap camera at different time points. Target absolute pose data of the target camera is determined based on the first group of images and external parameter data of the target camera. Absolute pose data of the co-view overlap camera is determined according to the second group of images, the target absolute pose data and external parameter data of the co-view overlap camera. Absolute pose data of the non-co-view overlap camera is determined according to the third group of images, the historically stored target absolute pose data and external parameter data of the non-co-view overlap camera. The multi-camera cooperation-based method improves the accuracy and robustness of pose estimation.

Description

Technical Field

[0001] This invention relates to the field of cameras, and more specifically to a method, apparatus, device, vehicle, and storage medium for determining the pose of multiple cameras. Background Technology

[0002] With the continuous advancement of vehicle sensor technology and the development of automated data acquisition technology, utilizing vehicle-mounted multi-camera systems for vehicle self-localization and 3D reconstruction has become one of the key technologies in the field of autonomous driving. By working collaboratively with multiple cameras, detailed information about the surrounding environment can be acquired, providing accurate spatial perception and decision support for autonomous driving.

[0003] Because multi-camera systems are often affected by vibrations or sudden events during operation, the images captured by the cameras may be affected, leading to distortion or loss of feature points. This abnormal motion may cause the camera pose to be incorrectly estimated, resulting in errors in the pose calculation of the image data.

[0004] Therefore, improving the positioning accuracy of each camera in a multi-camera system is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, device, vehicle, and storage medium for determining the pose of multiple cameras, which improves the accuracy of pose calculation for each camera in a multi-camera system.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, this application provides a method for determining the pose of multiple cameras, the method comprising:

[0008] Acquire image data from multiple cameras and extrinsic parameter data from each camera;

[0009] Based on the image data, a first group of images, a second group of images, and a third group of images are determined. The first group of images consists of a set of images with the same two-dimensional features taken by the target camera at different times. The second group of images consists of a set of images with the same two-dimensional features taken by the target camera and a co-viewing overlapping camera at the same time. The co-viewing overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera. The third group of images consists of a set of images with the same two-dimensional features taken by the target camera and a non-co-viewing overlapping camera at different times. The target camera is any one of the multiple cameras.

[0010] Based on the first set of images and the extrinsic data of the target camera, the target absolute pose data of the target camera is determined, and the target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system;

[0011] The absolute pose data of the common-view overlapping camera is determined based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the common-view overlapping camera.

[0012] The absolute pose data of the non-co-view overlapping camera is determined based on the third set of images, the historically stored target absolute pose data, and the extrinsic parameter data of the non-co-view overlapping camera.

[0013] By employing the aforementioned techniques, multi-view information from a multi-camera system can be fully utilized to improve the accuracy and robustness of pose estimation. Joint analysis of the multi-camera system not only leverages the extrinsic parameter data acquired by each camera itself but also allows for cross-calibration using pose data from the same camera at different times and from other cameras, thereby improving the absolute pose accuracy of each camera. Especially in complex scenes, the combination of multiple perspectives helps reduce errors that may arise from a single camera.

[0014] Further, the step of determining the target absolute pose data of the target camera based on the first set of images and the extrinsic parameter data of the target camera, wherein the target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system, includes:

[0015] Based on the first set of images, determine the first image of the target camera at the current time and the second image at a reference time, wherein the reference time is any time before the current time;

[0016] Based on the first image and the second image, determine the relative pose data of the target camera at the current time relative to the reference time;

[0017] The target absolute pose data is determined based on the relative pose data, the extrinsic parameter data of the target camera at the reference time, and the extrinsic parameter data of the target camera at the current time.

[0018] By combining image data acquired at different time points with extrinsic parameter data using the aforementioned techniques, the absolute pose of the target camera can be effectively determined by utilizing historical camera information. This method, through the calculation of relative pose data, ensures that pose estimation not only relies on observations at a single moment but also enhances the stability of pose estimation by using data from a reference time, reducing error accumulation and drift. This contributes to improving the positioning accuracy and temporal consistency of various cameras in a multi-camera system.

[0019] Further, determining the target absolute pose data based on the relative pose data, the extrinsic parameter data of the target camera at the reference time, and the extrinsic parameter data of the target camera at the current time includes:

[0020] Based on the relative pose data and the extrinsic parameter data of the target camera at the reference time, the reference absolute pose data is determined;

[0021] The target absolute pose data is determined based on the reference absolute pose data and the extrinsic data of the target camera at the current time.

[0022] By combining the relative pose data with the external parameter data of the reference time and the current time through the above technical means, the impact of relying solely on external measurement errors on the final result is reduced, and the robustness of camera pose estimation is improved.

[0023] Further, determining the target absolute pose data based on the reference absolute pose data and the extrinsic parameter data of the target camera at the current time includes:

[0024] The target absolute pose data is determined by averaging the reference absolute pose data and the extrinsic parameters of the target camera at the current time.

[0025] By employing the aforementioned techniques and averaging the values ​​to determine the target's absolute pose data, noise introduced by local errors or sudden factors can be further smoothed and eliminated. Using the average value for correction can stabilize pose estimation, reduce drastic fluctuations caused by instantaneous errors, and ensure that pose data remains more consistent and reliable over long periods.

[0026] Furthermore, the number of feature point pairs with the same two-dimensional features in the current time image and the reference time image is greater than 8.

[0027] By employing the aforementioned techniques, it is ensured that the image contains at least eight feature points, providing sufficient geometric information to accurately calculate the camera's relative pose. A small number of feature points may fail to accurately describe the geometric relationships of the image, leading to unreliable pose estimation. Ensuring a sufficient number of feature points effectively improves the accuracy of pose calculation.

[0028] Further, determining the absolute pose data of the co-view overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the co-view overlapping camera includes:

[0029] Based on the second set of images, determine the third image of the co-viewing overlapping camera at the current moment and the fourth image of the target camera at the current moment;

[0030] Based on the third image and the fourth image, the relative pose data of the co-viewing overlapping camera relative to the target camera is determined;

[0031] The absolute pose data of the common-view overlapping camera is determined based on the relative pose data of the common-view overlapping camera relative to the target camera, the absolute pose data of the target camera, and the extrinsic parameter data of the common-view overlapping camera.

[0032] By employing the aforementioned techniques and combining them with the pose information of the target camera, the absolute pose of overlapping cameras can be accurately calculated. This method enhances the collaborative effect between multiple cameras, improves the accuracy of pose estimation through data fusion from different perspectives, and is particularly effective in complex environments by increasing the redundancy of information from multiple perspectives, thus contributing to a more stable and accurate calculation of camera pose.

[0033] Further, determining the absolute pose data of the non-common-view overlapping camera based on the third set of images, historically stored target absolute pose data, and extrinsic parameter data of the non-common-view overlapping camera includes:

[0034] Based on the third set of images, determine the fifth image of the non-co-viewing overlapping camera at the current time and the sixth image of the target camera at the co-viewing time, where the co-viewing time is any time before the current time;

[0035] Based on the fifth image and the sixth image, determine the relative pose data of the non-co-viewing overlapping camera relative to the target camera;

[0036] The target absolute pose data of the target camera at the common viewing moment is determined from the historically stored target absolute pose data;

[0037] The absolute pose data of the non-co-view overlapping camera is determined based on the relative pose data of the non-co-view overlapping camera relative to the target camera, the target absolute pose data of the target camera at the co-view time, and the extrinsic parameter data of the non-co-view overlapping camera at the current time.

[0038] Using the aforementioned techniques, this method combines historically stored target absolute pose data with current image data to accurately calculate the pose of non-common-view overlapping cameras. By utilizing the target camera's historical pose data, the motion trajectories of non-common-view overlapping cameras can be correlated with those of other cameras, improving the stability and accuracy of multi-camera systems during long-term operation.

[0039] Furthermore, acquire the extrinsic data of each camera, including:

[0040] Obtain the vehicle's first yaw angle at the current time, the vehicle's rotation matrix relative to the world coordinate system at the current time, and the second yaw angle at any time during the vehicle's travel before the current time;

[0041] The change in yaw angle is determined based on the first yaw angle and the second yaw angle;

[0042] The theoretical yaw angle is determined based on the rotation matrix.

[0043] Based on the theoretical yaw angle and the change in yaw angle, determine the new yaw angle after correction;

[0044] Based on the new yaw angle and the initial extrinsic parameters of each camera at the current moment, the extrinsic parameter data of each camera are determined.

[0045] By employing the aforementioned technical means and considering the vehicle's yaw angle and rotation matrix, yaw errors caused by vehicle motion can be accurately corrected, thereby improving the accuracy of camera extrinsic data and further enhancing the accuracy of camera pose.

[0046] Secondly, this application provides a multi-camera pose determination device, the device comprising:

[0047] The acquisition module is used to acquire image data from multiple cameras and extrinsic parameter data from each camera.

[0048] The grouping determination module is used to determine a first group of images, a second group of images, and a third group of images based on the image data. The first group of images consists of a set of images with the same two-dimensional features captured by the target camera at different times. The second group of images consists of a set of images with the same two-dimensional features captured by the target camera and a co-viewing overlapping camera at the same time. The co-viewing overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera. The third group of images consists of a set of images with the same two-dimensional features captured by the target camera and a non-co-viewing overlapping camera at different times. The target camera is any one of the multiple cameras.

[0049] The target camera determination module is used to determine the target absolute pose data of the target camera based on the first set of images and the extrinsic parameter data of the target camera. The target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system.

[0050] A common-view overlapping camera determination module is used to determine the absolute pose data of the common-view overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the common-view overlapping camera.

[0051] The non-common-view overlapping camera determination module is used to determine the absolute pose data of the non-common-view overlapping camera based on the third set of images, historically stored target absolute pose data, and the extrinsic parameter data of the non-common-view overlapping camera.

[0052] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0053] The memory stores computer-executed instructions;

[0054] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any of the first aspects.

[0055] Fourthly, this application provides a vehicle including a controller for use as described in any of the first aspects.

[0056] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0057] The beneficial effects of this invention are:

[0058] This approach fully leverages the multi-view information from a multi-camera system to improve the accuracy and robustness of pose estimation. Through joint analysis of the multi-camera system, it not only utilizes the extrinsic parameter data acquired by each camera but also performs cross-calibration using pose data from the same camera at different times and from other cameras, thereby improving the absolute pose accuracy of each camera. Especially in complex scenes, the combination of multiple perspectives helps reduce errors that may arise from a single camera. Compared to traditional single-camera localization methods, this multi-camera collaborative approach not only provides higher accuracy but also enhances the system's adaptability to sudden camera anomalies, thus achieving more reliable pose estimation and target tracking. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0060] Figure 1 This is a flowchart illustrating an embodiment of a method for determining the pose of multiple cameras proposed in this application.

[0061] Figure 2 The co-view situation of the same camera at different times provided in this application;

[0062] Figure 3 A schematic diagram of the feature points of the essential matrix provided in this application;

[0063] Figure 4 A schematic diagram illustrating the co-viewing situation of cameras at the same moment as provided in this application;

[0064] Figure 5 A schematic diagram illustrating the co-viewing situation of different cameras at different times for this application;

[0065] Figure 6 A schematic diagram of yaw angle variation provided for this application;

[0066] Figure 7 A schematic diagram of the structure of a multi-camera pose determination device provided in this application;

[0067] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0068] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0069] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0070] Currently, multi-camera systems are widely used in vehicles to acquire environmental data. However, in practice, the poses of different cameras in a multi-camera system may become abnormal during driving, leading to pose misalignment issues when using image data subsequently acquired by the cameras.

[0071] In view of the above problems, this application proposes a method for determining the pose of multiple cameras. First, the pose of a single camera is determined based on images taken at different times. Then, based on the single camera with the determined pose, the poses of other cameras with co-view relationships at the same time and cameras with co-view relationships at different times are calibrated separately. This method can reduce the error propagation of a single camera and improve the calibration accuracy of the overall system.

[0072] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0073] Figure 1 This is a flowchart illustrating an embodiment of a multi-camera pose determination method proposed in this application, as shown below. Figure 1 As shown, it includes the following steps:

[0074] S101. Acquire image data from multiple cameras and extrinsic parameter data from each camera.

[0075] In this step, image data is acquired simultaneously using multiple cameras. Each camera captures the surrounding environment from different angles and positions, generating multiple image frames. These images are used for subsequent image processing, feature matching, and camera pose estimation. The image data includes timestamps of the image frames, camera calibration information, and image content.

[0076] For the extrinsic data of each camera, the camera's installation position and orientation are known (e.g., the specific installation position and orientation of the camera on the vehicle). Based on the vehicle's yaw angle, the extrinsic parameters of the camera at each moment can be calculated through coordinate transformation relationships.

[0077] S102. Based on the image data, determine the first set of images, the second set of images, and the third set of images. The first set of images is a set of images with the same two-dimensional features taken by the target camera at different times. The second set of images is a set of images with the same two-dimensional features taken by the target camera and a co-viewing overlapping camera at the same time. The co-viewing overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera. The third set of images is a set of images with the same two-dimensional features taken by the target camera and a non-co-viewing overlapping camera at different times. The target camera is any one of the multiple cameras.

[0078] Image data is subjected to feature matching to determine feature matching relationships, identifying a group of images with the same two-dimensional features. Then, based on the image timestamps, the images with the same two-dimensional features are divided into three groups.

[0079] The first set of images consists of images taken by the target camera at different times, all possessing the same two-dimensional features (such as the same object or a point in a scene). These images are used to track the positional changes of the target camera at different times. The target camera is the first camera that needs to be calibrated; it can be the forward-looking camera of a vehicle's multi-camera system, or another camera.

[0080] The second set of images consists of images taken by the target camera and the co-viewing overlapping camera at the same time. The co-viewing overlapping camera refers to another camera with an overlapping viewpoint with the target camera. The shooting perspectives of these two types of cameras overlap, and they can observe the same feature points in the scene. The feature points in the second set of images will be used to calculate the relative pose between the cameras.

[0081] The third set of images consists of images taken by the target camera and a non-overlapping camera at different times. A non-overlapping camera is one whose field of view does not overlap with that of the target camera. In this case, although there is no direct overlap in the field of view between the cameras, the same feature points captured by the two types of cameras can still be matched to calculate the relative pose.

[0082] S103. Based on the first set of images and the external parameter data of the target camera, determine the target absolute pose data of the target camera. The target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system.

[0083] In this step, using two images from the first set of images from the target camera, the relative pose data of the two cameras at a later time step relative to the previous time step is obtained by calculating the essential matrix between the two images; that is, the rotation matrix and translation vector. To determine the absolute pose at the current time step, the first image from the target camera at the current time step is needed, and the other image is the second image from the target camera at a reference time step, which is any time step in the first set of images prior to the current time step.

[0084] There are at least the following methods for determining the absolute pose data of a target:

[0085] In one implementation, the target's absolute pose data is determined based on two images. Specifically, based on the first and second images, the relative pose data of the target camera at the current moment relative to a reference moment is determined; based on the relative pose data and the extrinsic parameter data of the target camera at the reference moment, the reference absolute pose data at the current moment can be determined. Then, by combining the reference absolute pose data and the extrinsic parameter data at the current moment, the target's absolute pose data at the current moment is determined. By combining the extrinsic parameter and relative pose data from the previous moment, the accuracy, stability, and robustness of the target's absolute pose estimation can be significantly improved. This method avoids errors caused by a single image, reduces error accumulation, optimizes the estimation process using temporal continuity, and achieves accurate positioning through visual data even when external sensors cannot be relied upon. This method is particularly effective in dynamic environments and helps improve the overall performance and reliability of visual positioning systems that operate for extended periods.

[0086] In one implementation, multiple images are used to determine the target's absolute pose data. Specifically, images from multiple time points are selected sequentially as second images, and combined with the first image to determine multiple reference absolute pose data. Based on the pose determined by multiple references and the extrinsic parameter data at the current time point, the target's absolute pose data can be obtained by averaging. By combining multiple images, the accuracy and robustness of pose estimation are further improved, effectively reducing the error or noise that might be introduced by a single image. Furthermore, averaging multiple reference poses helps smooth out instantaneous errors, avoiding large deviations caused by instantaneous image matching errors. In addition, images from multiple time points provide more viewpoint and motion information, making the target pose estimation more stable, especially in dynamic environments.

[0087] In one implementation, when calculating the target absolute pose based on the reference absolute pose data and the target camera's extrinsic data at the current moment, the weight ratios of the reference absolute pose data and the target camera's extrinsic data at the current moment are different.

[0088] It should be noted that the camera's extrinsic data and the camera's absolute pose data refer to the same thing.

[0089] S104. Determine the absolute pose data of the common-view overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the common-view overlapping camera.

[0090] After determining the absolute pose data of the target camera, the pose of the overlapping camera with the same field of view is calibrated based on the target camera. The second set of images includes multiple images with the same feature points taken by the overlapping camera and the target camera at the same time. The third image from the overlapping camera at the current time and the fourth image from the target camera at the current time are selected from the second set of images.

[0091] There are several ways to determine the absolute pose data of the co-viewing overlapping cameras:

[0092] In one implementation, the relative pose data of the co-viewing overlapping camera relative to the target camera is determined based on the third and fourth images. Specifically, this can be obtained by solving for the essential matrix and then decomposing it. Based on the relative pose data of the co-viewing overlapping camera relative to the target camera, the target's absolute pose data, and the extrinsic parameters of the co-viewing overlapping camera, the absolute pose data of the co-viewing overlapping camera is determined. Specifically, based on the target's absolute pose data and the relative pose data of the co-viewing overlapping camera, reference absolute pose data can be calculated. Combined with the extrinsic parameters of the co-viewing overlapping camera, the absolute pose data of the co-viewing overlapping camera at the current moment is determined. This can be obtained by averaging the two values.

[0093] In another implementation, multiple fourth images can be selected and combined with the third image to obtain multiple reference absolute pose data. Then, the average value of the multiple reference absolute pose data and the extrinsic parameter data of the co-viewing overlapping camera can be calculated and used as the absolute pose data of the co-viewing overlapping camera at the current time.

[0094] S105. Based on the third set of images, the historically stored target absolute pose data, and the extrinsic parameter data of the non-common-view overlapping camera, determine the absolute pose data of the non-common-view overlapping camera.

[0095] For cameras that do not share a common field of view with the target camera, calibration based on the target camera and determination of its pose data can only be achieved using a third set of images with the same features taken at different times. This third set of images includes multiple images with the same feature points taken by the non-common-view overlapping camera and the target camera at different times. The fifth image from the non-common-view overlapping camera at the current time and the sixth image from the target camera at the common-view moment are selected from the third set of images. The common-view moment is any time prior to the current time.

[0096] There are several ways to determine the absolute pose data of the co-viewing overlapping cameras:

[0097] In one implementation, the relative pose data of the non-co-viewing overlapping camera relative to the target camera is determined based on the fifth and sixth images. The target absolute pose data of the target camera at the time of co-view is determined from historically stored target absolute pose data. The absolute pose data of the non-co-viewing overlapping camera is determined based on the relative pose data of the non-co-viewing overlapping camera relative to the target camera, the target absolute pose data of the target camera at the time of co-view, and the extrinsic parameter data of the non-co-viewing overlapping camera at the current time. Specifically, reference absolute pose data can be calculated based on the target absolute pose data of the target camera at the time of co-view and the relative pose data of the non-co-viewing overlapping camera. Combined with the extrinsic parameter data of the non-co-viewing overlapping camera, the absolute pose data of the non-co-viewing overlapping camera at the current time is determined. This can be achieved by averaging the two data.

[0098] In another implementation, multiple sixth images can be selected and combined with the fifth image to obtain multiple reference absolute pose data. Then, the average value of the multiple reference absolute pose data and the extrinsic parameter data of the non-co-viewing overlapping camera can be calculated and used as the absolute pose data of the co-viewing overlapping camera at the current time.

[0099] This embodiment provides a method for determining the pose of multiple cameras. The method includes: acquiring image data and extrinsic parameter data of each camera from multiple cameras; determining a first set of images, a second set of images, and a third set of images based on the image data. The first set of images consists of images with the same two-dimensional features captured by a target camera at different times; the second set of images consists of images with the same two-dimensional features captured by the target camera and a co-viewing overlapping camera at the same time; the co-viewing overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera; the third set of images consists of images with the same two-dimensional features captured by the target camera and a non-co-viewing overlapping camera at different times; the target camera is any one of the multiple cameras; determining the target absolute pose data of the target camera based on the first set of images and the extrinsic parameter data of the target camera; the target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system; determining the absolute pose data of the co-viewing overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the co-viewing overlapping camera; and determining the absolute pose data of the non-co-viewing overlapping camera based on the third set of images, historically stored target absolute pose data, and the extrinsic parameter data of the non-co-viewing overlapping camera. By incorporating images from multiple cameras and images from the same camera at different times, pose is determined through calculations between the cameras, reducing the impact of individual camera errors and improving the accuracy of pose determination.

[0100] Furthermore, by combining image information from multiple cameras, this method can maintain the robustness of the system even if cameras malfunction or temporarily fail. For example, if the view of one camera is temporarily obstructed or cannot provide reliable image information, other cameras (especially co-overlapping and non-co-overlapping cameras) can still provide the information needed for pose estimation, avoiding the risk of the entire system crashing.

[0101] The following examples will be used to provide a detailed introduction to the pose determination process of the target camera, the co-view overlapping camera, and the non-co-view overlapping camera.

[0102] Figure 2 This application provides co-viewing scenarios of the same camera at different times. For example... Figure 2 As shown, with the vehicle's forward-facing camera as the target camera, the figure shows four consecutive moments of the vehicle, with images A, B, C, and D captured at each moment. The images captured at any two moments are mutually identical, meaning they both contain multiple identical feature points.

[0103] In one scenario, if images A, B, C, and D may all contain the same features, then A, B, C, and D can be considered as the first set of images.

[0104] In a case where A, B, and C contain the same features, and B, C, and D contain the same features, then A, B, and C are considered as a first group of images, and B, C, and D are also considered as a first group of images.

[0105] For example, in the first case described above, the pose data of the forward-looking camera at the current moment is calculated based on four images A, B, C, and D that have the same feature points. Assuming that the moment of image D is the current moment, firstly, each pair of images is formed into an image pair: A and D, B and D, C and D.

[0106] For each image pair, the method for calculating the reference absolute pose data is the same, taking images C and D as examples.

[0107] Based on the two images, solve for the essential matrix. The formula for calculating the essential matrix E is:

[0108] P D T EP C =0

[0109] In the formula P C It is an n-dimensional, 2-column matrix, where each row represents the coordinates of a common feature point in the C image of the current frame, P. D It is an n-dimensional, 2-column matrix, where each row represents the coordinates of a common feature point in the CD image in the current frame of the D image.

[0110] Figure 3 A schematic diagram of the feature points of the essential matrix provided in this application, such as Figure 3 As shown, the number of feature points involved in the calculation of the essential matrix must be greater than 8, i.e., P. C and P D For the calculated essential matrix to be valid, n must be greater than 8. Singular Value Decomposition (SVD) of the essential matrix yields the relative pose data of the forward-looking camera at time B relative to time A, namely the camera's rotation matrix and displacement vector.

[0111] Based on the extrinsic parameter data at time C, combined with the relative pose data at time D relative to time C, the absolute pose data at time D can be determined.

[0112] Assume the absolute pose of the camera at time C is W. A The relative pose of the camera at time D with respect to time C is W. rel Then the reference absolute pose at time D can be obtained through the following relationship: W B =W rel ×W A .

[0113] W rel It is the relative pose obtained through SVD (i.e., the combination of rotation matrix and translation vector).

[0114] The above process yields the reference absolute pose based on C. There are also multiple matching pairs, namely A and D, and B and D. Following the same steps, we can obtain the reference absolute pose based on A and the reference absolute pose based on B.

[0115] In one implementation, the target absolute pose data at time D is obtained by adding the target camera's extrinsic parameters to the three reference absolute poses and then dividing by 4.

[0116] In actual calculations, there may be more images. All absolute pose data and the current extrinsic parameter data are treated as a Gaussian distribution, and the average value of this distribution is taken as the pose data W of the forward-looking camera. final The calculation formula is:

[0117]

[0118] Treating the absolute pose data at each moment as a Gaussian distribution and taking its average not only handles uncertainty but also enhances the robustness and accuracy of the system. This approach is particularly suitable for applications requiring continuous pose estimation where the system may be affected by dynamic environmental changes or noise.

[0119] The above process determines the pose of the vehicle's forward-looking camera. The pose determination process for the co-viewing overlapping camera is described below.

[0120] Figure 4 This is a schematic diagram illustrating the co-viewing situation of cameras at the same moment, as provided in this application. Figure 4 As shown, there is already a shared field of view between the side camera and the front-view camera. After determining the absolute pose of the front-view camera, the decision pose of the overlapping camera can be determined based on the absolute pose of the front-view camera.

[0121] The second set of images includes images with the same feature points taken by the forward-looking camera and the co-view overlapping camera at the same time. Based on the two images taken by the forward-looking camera and the co-view overlapping camera in the second set of images, the essential matrix of the co-view overlapping camera relative to the forward-looking camera is calculated, and then the relative pose of the co-view overlapping camera is calculated. Finally, the absolute pose of the co-view overlapping camera is solved.

[0122] The pose of each co-viewing overlapping camera is calculated separately. The specific methods for calculating the essential matrix, relative pose, and absolute pose are as described above. Figure 3 The methods used in the embodiments are similar and will not be described again.

[0123] In a vehicle's multi-camera system, in addition to the co-viewing overlapping camera that shares the same view as the forward-viewing camera, there is also a non-co-viewing overlapping camera that does not share the same view as the forward-viewing camera.

[0124] Figure 5 This application provides a schematic diagram illustrating the co-viewing situation of different cameras at different times, such as... Figure 5 As shown, the forward-view camera at the front of the vehicle and the rear-view camera at the rear of the vehicle share a common viewing area at different times during vehicle movement. After determining the absolute pose of the forward-view camera, the decision pose of the non-co-viewing overlapping camera can be determined based on the absolute pose of the forward-view camera and the third set of images.

[0125] The third set of images includes images with the same feature points taken by the forward-looking camera and the non-co-viewing overlapping camera at different times. Based on the two images taken by the forward-looking camera and the non-co-viewing overlapping camera in the third set of images, the essential matrix of the non-co-viewing overlapping camera relative to the forward-looking camera is calculated, and then the relative pose of the non-co-viewing overlapping camera is calculated. Finally, the absolute pose of the non-co-viewing overlapping camera is solved.

[0126] The absolute pose of each non-common-view overlapping camera is calculated separately. The specific methods for calculating the essential matrix, relative pose, and absolute pose are as described above. Figure 3 The methods used in the embodiments are similar and will not be described again.

[0127] The three methods described above allow for the calibration of the absolute pose of the forward-looking camera based on its extrinsic parameter data and data from different time points. The absolute pose of each camera on the vehicle, i.e., its extrinsic parameters, is determined based on the absolute pose data of the forward-looking camera and the extrinsic parameter data of each camera. Therefore, to obtain accurate pose data, the extrinsic parameter data of the forward-looking camera needs to be calibrated before determining the absolute pose data of each camera to improve data accuracy.

[0128] Figure 6 The diagram illustrating the yaw angle variation provided in this application is as follows: Figure 6 As mentioned above, the yaw angle changes smoothly during vehicle movement. There is an angular error between the theoretical heading angle (i.e., theoretical yaw angle) and the actual trajectory heading angle. Therefore, it is necessary to correct the theoretical heading angle to obtain a more accurate vehicle heading angle. The vehicle's external parameter data is then updated based on the corrected vehicle heading angle.

[0129] At each moment of a trajectory process, the yaw angle (i.e., the first yaw angle) P at the current moment is first obtained through the sensor. cur_time The yaw angle P at the previous moment (i.e., the second yaw angle) pre_time The time interval between each moment is preset, such as 1 second, 5 seconds or 10 seconds.

[0130] Next, calculate the change in yaw angle. The calculation method is as follows:

[0131]

[0132] Obtain the rotation matrix R of the vehicle body relative to the world coordinate system using the IMU:

[0133]

[0134] The yaw angle ψ of the vehicle body is calculated as follows:

[0135]

[0136] Calculate the new yaw angle ψ after correction at each point. new :

[0137] ψ new =ψ+Δ ψ

[0138] Based on the new yaw angle and the initial extrinsic parameters of each camera at the current moment, the extrinsic parameter data of each camera is determined. Specifically, the yaw angle at the current moment is incorporated into the camera's extrinsic parameter matrix using a rotation matrix, thereby obtaining the updated extrinsic parameter data.

[0139] Figure 7 A schematic diagram of a multi-camera pose determination device provided in this application is shown below. Figure 7 As shown, the multi-camera pose determination device 40 includes:

[0140] The acquisition module 401 is used to acquire image data and external parameter data of each camera from multiple cameras;

[0141] The grouping determination module 402 is used to determine a first group of images, a second group of images, and a third group of images based on the image data. The first group of images is a group of images with the same two-dimensional features taken by the target camera at different times. The second group of images is a group of images with the same two-dimensional features taken by the target camera and a co-view overlapping camera at the same time. The co-view overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera. The third group of images is a group of images with the same two-dimensional features taken by the target camera and a non-co-view overlapping camera at different times. The target camera is any one of the multiple cameras.

[0142] The target camera determination module 403 is used to determine the target absolute pose data of the target camera based on the first set of images and the extrinsic parameter data of the target camera. The target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system.

[0143] The co-view overlapping camera determination module 404 is used to determine the absolute pose data of the co-view overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the co-view overlapping camera.

[0144] The non-co-view overlapping camera determination module 405 is used to determine the absolute pose data of the non-co-view overlapping camera based on the third set of images, historically stored target absolute pose data, and the extrinsic parameter data of the non-co-view overlapping camera.

[0145] Optionally, the target camera determination module 403 is specifically used for:

[0146] Based on the first set of images, determine the first image of the target camera at the current time and the second image at a reference time, wherein the reference time is any time before the current time;

[0147] Based on the first image and the second image, determine the relative pose data of the target camera at the current time relative to the reference time;

[0148] The target absolute pose data is determined based on the relative pose data, the extrinsic parameter data of the target camera at the reference time, and the extrinsic parameter data of the target camera at the current time.

[0149] Optionally, the target camera determination module 403 is specifically used for:

[0150] Based on the relative pose data and the extrinsic parameter data of the target camera at the reference time, the reference absolute pose data is determined;

[0151] The target absolute pose data is determined based on the reference absolute pose data and the extrinsic data of the target camera at the current time.

[0152] Optionally, the target camera determination module 403 is specifically used for:

[0153] The target absolute pose data is determined by averaging the reference absolute pose data and the extrinsic parameters of the target camera at the current time.

[0154] Optionally, the number of feature point pairs with the same two-dimensional features in the current time image and the reference time image is greater than 8.

[0155] Optionally, the co-viewing overlapping camera determination module 404 is specifically used for:

[0156] Based on the second set of images, determine the third image of the co-viewing overlapping camera at the current moment and the fourth image of the target camera at the current moment;

[0157] Based on the third image and the fourth image, the relative pose data of the co-viewing overlapping camera relative to the target camera is determined;

[0158] The absolute pose data of the common-view overlapping camera is determined based on the relative pose data of the common-view overlapping camera relative to the target camera, the absolute pose data of the target camera, and the extrinsic parameter data of the common-view overlapping camera.

[0159] Optionally, the non-co-viewing overlapping camera determination module 405 is specifically used for:

[0160] Based on the third set of images, determine the fifth image of the non-co-viewing overlapping camera at the current time and the sixth image of the target camera at the co-viewing time, where the co-viewing time is any time before the current time;

[0161] Based on the fifth image and the sixth image, determine the relative pose data of the non-co-viewing overlapping camera relative to the target camera;

[0162] The target absolute pose data of the target camera at the common viewing moment is determined from the historically stored target absolute pose data;

[0163] The absolute pose data of the non-co-view overlapping camera is determined based on the relative pose data of the non-co-view overlapping camera relative to the target camera, the target absolute pose data of the target camera at the co-view time, and the extrinsic parameter data of the non-co-view overlapping camera at the current time.

[0164] The multi-camera pose determination device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0165] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application, which may be a vehicle controller. Figure 8 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0166] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0167] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0168] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0169] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0170] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0171] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0172] This application also provides a vehicle including a controller for implementing the above-described method.

[0173] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0174] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0175] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0176] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0181] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for determining the pose of multiple cameras, characterized in that, The method includes: Acquire image data from multiple cameras and extrinsic parameter data from each camera; Based on the image data, a first group of images, a second group of images, and a third group of images are determined. The first group of images consists of a set of images with the same two-dimensional features taken by the target camera at different times. The second group of images consists of a set of images with the same two-dimensional features taken by the target camera and a co-viewing overlapping camera at the same time. The co-viewing overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera. The third group of images consists of a set of images with the same two-dimensional features taken by the target camera and a non-co-viewing overlapping camera at different times. The target camera is any one of the multiple cameras. Based on the first set of images and the extrinsic data of the target camera, the target absolute pose data of the target camera is determined, and the target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system; The absolute pose data of the common-view overlapping camera is determined based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the common-view overlapping camera. The absolute pose data of the non-co-view overlapping camera is determined based on the third set of images, the historically stored target absolute pose data, and the extrinsic parameter data of the non-co-view overlapping camera.

2. The method according to claim 1, characterized in that, The step of determining the target absolute pose data of the target camera based on the first set of images and the extrinsic parameter data of the target camera, wherein the target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system, includes: Based on the first set of images, determine the first image of the target camera at the current time and the second image at a reference time, wherein the reference time is any time before the current time; Based on the first image and the second image, determine the relative pose data of the target camera at the current time relative to the reference time; The target absolute pose data is determined based on the relative pose data, the extrinsic parameter data of the target camera at the reference time, and the extrinsic parameter data of the target camera at the current time.

3. The method according to claim 2, characterized in that, The step of determining the target absolute pose data based on the relative pose data, the extrinsic parameter data of the target camera at the reference time, and the extrinsic parameter data of the target camera at the current time includes: Based on the relative pose data and the extrinsic parameter data of the target camera at the reference time, the reference absolute pose data is determined; The target absolute pose data is determined based on the reference absolute pose data and the extrinsic data of the target camera at the current time.

4. The method according to claim 3, characterized in that, The step of determining the target absolute pose data based on the reference absolute pose data and the target camera's extrinsic parameter data at the current time includes: The target absolute pose data is determined by averaging the reference absolute pose data and the extrinsic parameters of the target camera at the current time.

5. The method according to any one of claims 2, characterized in that, The number of feature point pairs with the same two-dimensional features in the current time image and the reference time image is greater than 8.

6. The method according to any one of claims 2-5, characterized in that, The step of determining the absolute pose data of the co-view overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the co-view overlapping camera includes: Based on the second set of images, determine the third image of the co-viewing overlapping camera at the current moment and the fourth image of the target camera at the current moment; Based on the third image and the fourth image, the relative pose data of the co-viewing overlapping camera relative to the target camera is determined; The absolute pose data of the common-view overlapping camera is determined based on the relative pose data of the common-view overlapping camera relative to the target camera, the absolute pose data of the target camera, and the extrinsic parameter data of the common-view overlapping camera.

7. The method according to any one of claims 2-5, characterized in that, The step of determining the absolute pose data of the non-common-view overlapping camera based on the third set of images, historically stored target absolute pose data, and extrinsic parameter data of the non-common-view overlapping camera includes: Based on the third set of images, determine the fifth image of the non-co-viewing overlapping camera at the current time and the sixth image of the target camera at the co-viewing time, where the co-viewing time is any time before the current time; Based on the fifth image and the sixth image, determine the relative pose data of the non-co-viewing overlapping camera relative to the target camera; The target absolute pose data of the target camera at the common viewing moment is determined from the historically stored target absolute pose data; The absolute pose data of the non-co-view overlapping camera is determined based on the relative pose data of the non-co-view overlapping camera relative to the target camera, the target absolute pose data of the target camera at the co-view time, and the extrinsic parameter data of the non-co-view overlapping camera at the current time.

8. The method according to any one of claims 2-5, characterized in that, Obtain the extrinsic data for each camera, including: Obtain the vehicle's first yaw angle at the current time, the vehicle's rotation matrix relative to the world coordinate system at the current time, and the second yaw angle at any time during the vehicle's travel before the current time; The change in yaw angle is determined based on the first yaw angle and the second yaw angle; The theoretical yaw angle is determined based on the rotation matrix. Based on the theoretical yaw angle and the change in yaw angle, determine the new yaw angle after correction; Based on the new yaw angle and the initial extrinsic parameters of each camera at the current moment, the extrinsic parameter data of each camera are determined.

9. A device for determining the pose of multiple cameras, characterized in that, The device includes: The acquisition module is used to acquire image data from multiple cameras and extrinsic parameter data from each camera. The grouping determination module is used to determine a first group of images, a second group of images, and a third group of images based on the image data. The first group of images consists of a set of images with the same two-dimensional features captured by the target camera at different times. The second group of images consists of a set of images with the same two-dimensional features captured by the target camera and a co-viewing overlapping camera at the same time. The co-viewing overlapping camera refers to a camera whose shooting angle overlaps with that of the target camera. The third group of images consists of a set of images with the same two-dimensional features captured by the target camera and a non-co-viewing overlapping camera at different times. The target camera is any one of the multiple cameras. The target camera determination module is used to determine the target absolute pose data of the target camera based on the first set of images and the extrinsic parameter data of the target camera. The target absolute pose data represents the transformation relationship between the camera coordinate system and the world coordinate system. A common-view overlapping camera determination module is used to determine the absolute pose data of the common-view overlapping camera based on the second set of images, the target absolute pose data, and the extrinsic parameter data of the common-view overlapping camera. The non-common-view overlapping camera determination module is used to determine the absolute pose data of the non-common-view overlapping camera based on the third set of images, historically stored target absolute pose data, and the extrinsic parameter data of the non-common-view overlapping camera.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A vehicle, characterized in that, The vehicle includes a controller for use in the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

Images