Vehicle-mounted camera calibration method and device, terminal equipment and computer program product

Abstract

The invention relates to a vehicle-mounted camera calibration method and apparatus, a terminal device and a computer program product. The method comprises the steps of obtaining three-dimensional point cloud data obtained by performing three-dimensional scanning on a target vehicle by an electronic device; constructing a three-dimensional model and a vehicle coordinate system of the target vehicle based on the three-dimensional point cloud data; obtaining a first video stream obtained by shooting the target vehicle by the electronic equipment and a second video stream collected by a plurality of vehicle-mounted cameras; based on the first video stream and the three-dimensional model, determining a target pose stream of the target vehicle in a world coordinate system; based on the target pose stream, the first video stream and the second video stream, determining a first shooting parameter of each vehicle-mounted camera in a vehicle coordinate system; and performing bundle adjustment optimization on the first camera parameter of each vehicle-mounted camera to obtain a second camera parameter of each vehicle-mounted camera in the vehicle coordinate system. According to the method and the device, the problem of relatively low efficiency of calibrating the vehicle-mounted multi-path camera system by depending on a physical target in related technologies is solved.

Description

Technical Field

[0001] This application belongs to the field of camera calibration technology, and in particular relates to a method, apparatus, terminal equipment and computer program product for calibrating a vehicle-mounted camera. Background Technology

[0002] Road patrol vehicles, autonomous vehicles, or driverless vehicles are typically equipped with multiple cameras. In these vehicle-mounted multi-camera systems, each camera needs to be precisely calibrated to obtain its intrinsic and extrinsic parameters. Traditional camera calibration methods rely on physical targets such as checkerboard patterns, which require careful arrangement in a controlled environment. This is extremely inconvenient for on-site implementation, has low calibration efficiency, and is not suitable for rapid roadside deployment and retesting.

[0003] Currently, no effective solution has been proposed to address the low efficiency of calibrating vehicle-mounted multi-channel camera systems using physical targets in related technologies. Summary of the Invention

[0004] This application provides a method, apparatus, terminal device, and computer program product for calibrating vehicle-mounted cameras, in order to at least solve the problem of low efficiency in calibrating vehicle-mounted multi-channel camera systems by relying on physical targets in related technologies.

[0005] In a first aspect, embodiments of this application provide a method for calibrating a vehicle-mounted camera, comprising: acquiring three-dimensional point cloud data obtained by an electronic device performing a three-dimensional scan of a target vehicle, wherein the target vehicle includes multiple vehicle-mounted cameras; constructing a three-dimensional model and a vehicle coordinate system of the target vehicle based on the three-dimensional point cloud data; acquiring a first video stream obtained by the electronic device capturing images of the target vehicle, and a second video stream captured by the multiple vehicle-mounted cameras, after the target vehicle starts and drives; determining a target pose flow of the target vehicle in a world coordinate system based on the first video stream and the three-dimensional model; determining a first camera parameter of each vehicle-mounted camera in the vehicle coordinate system based on the target pose flow, the first video stream, and the second video stream; and performing bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain a second camera parameter of each vehicle-mounted camera in the vehicle coordinate system.

[0006] In some embodiments, the first camera parameters include a first intrinsic parameter, a first distortion parameter, a first time offset, a first extrinsic parameter, and a first plane parameter; determining the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system based on the target pose stream, the first video stream, and the second video stream includes: inputting image frames from the second video stream acquired by each vehicle-mounted camera into a depth calibration model to obtain the first intrinsic parameter and the first distortion parameter of each vehicle-mounted camera; determining the first time offset between the second video stream acquired by each vehicle-mounted camera and the first video stream based on the target pose stream; and determining the first extrinsic parameter of each vehicle-mounted camera in the vehicle coordinate system, and the first plane parameter of the road surface on which the target vehicle is traveling in the vehicle coordinate system, based on the road surface assumption, the target pose stream, and the first intrinsic parameter and the first time offset of each vehicle-mounted camera, according to the road surface assumption, wherein the road surface on which the target vehicle is traveling is assumed to be a mathematical plane.

[0007] In some embodiments, determining the first extrinsic parameters of each vehicle-mounted camera in the vehicle coordinate system, and the first planar parameters of the road surface traveled by the target vehicle in the vehicle coordinate system, based on the road surface plane assumption, the target pose stream, and the first intrinsic parameters and the first time offset of each vehicle-mounted camera, includes: for each vehicle-mounted camera, selecting a current image frame and a next image frame from the second video stream acquired by the vehicle-mounted camera; determining, based on the first time offset of the vehicle-mounted camera, the current pose data corresponding to the current image frame and the next pose data corresponding to the next image frame from the target pose stream; constructing a planar homography matrix between the current image frame and the next image frame based on the road surface plane assumption, the current pose data, the next pose data, and the first intrinsic parameters of the vehicle-mounted camera; solving the planar homography matrix corresponding to each vehicle-mounted camera to calculate the first extrinsic parameters of each vehicle-mounted camera in the vehicle coordinate system and the first planar parameters of the road surface traveled by the target vehicle in the vehicle coordinate system.

[0008] In some embodiments, performing bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system includes: for each vehicle-mounted camera, constructing a matrix residual based on the planar homography matrix corresponding to the vehicle-mounted camera; constructing an objective function based on the matrix residual; using the target pose flow as a fixed value, and using the first intrinsic parameter, the first distortion parameter, the first time offset, the first extrinsic parameter, and the first planar parameter of the vehicle-mounted camera as optimization variables, performing bundle adjustment optimization by minimizing the objective function to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system.

[0009] In some embodiments, determining the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the 3D model includes: acquiring the first pose flow of the electronic device in the world coordinate system during the recording of the first video stream; inputting the first video stream and the 3D model into a pose estimation model to obtain a second pose flow of the target vehicle relative to the electronic device, wherein the first pose flow and the second pose flow have the same scale as the 3D model; and determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the second pose flow.

[0010] In some embodiments, determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the second pose flow includes: detecting a first key point from a first image frame in the first video stream, and using the first key point to perform feature matching on the 3D model to obtain a second key point; optimizing the second pose flow based on the first key point and the second key point corresponding to each of the first image frames in the first video stream, with the goal of minimizing reprojection error, to obtain a third pose flow of the target vehicle in the world coordinate system; and determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the third pose flow.

[0011] In some embodiments, after performing bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system, the method further includes: calculating a calibration quality index for each vehicle-mounted camera based on the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system; and outputting the calibration result of the vehicle-mounted camera based on the comparison result of the calibration quality index and a preset threshold.

[0012] Secondly, embodiments of this application provide a vehicle-mounted camera calibration device, comprising: a first acquisition module, configured to acquire three-dimensional point cloud data obtained by an electronic device performing a three-dimensional scan of a target vehicle, wherein the target vehicle includes multiple vehicle-mounted cameras; a construction module, configured to construct a three-dimensional model and a vehicle coordinate system of the target vehicle based on the three-dimensional point cloud data; a second acquisition module, configured to acquire a first video stream obtained by the electronic device capturing images of the target vehicle after the target vehicle starts and drives, and a second video stream captured by the multiple vehicle-mounted cameras; a first determination module, configured to determine the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the three-dimensional model; a second determination module, configured to determine a first camera parameter of each vehicle-mounted camera in the vehicle coordinate system based on the target pose flow, the first video stream, and the second video stream; and an optimization module, configured to perform bundle adjustment optimization on the first camera parameter of each vehicle-mounted camera to obtain a second camera parameter of each vehicle-mounted camera in the vehicle coordinate system.

[0013] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the vehicle camera calibration method of any of the first aspects described above.

[0014] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the vehicle camera calibration method described in any one of the first aspects to be executed.

[0015] Compared to related technologies, the vehicle-mounted camera calibration method, apparatus, terminal device, and computer program product provided in this application can acquire 3D point cloud data obtained by an electronic device performing a 3D scan of a target vehicle including multiple vehicle-mounted cameras; can construct a 3D model of the target vehicle and a vehicle coordinate system based on the 3D point cloud data; can acquire a first video stream captured by an electronic device of the target vehicle and a second video stream collected by multiple vehicle-mounted cameras after the target vehicle starts and drives; can determine the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the 3D model; can determine the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system based on the target pose flow, the first video stream, and the second video stream; and can perform bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system. In this way, a 3D model of the target vehicle is obtained by scanning the target vehicle with electronic devices, and the target vehicle's pose flow is generated by continuously tracking the target vehicle's motion using electronic devices. This dynamic reference can replace traditional physical targets and naturally bridges all non-overlapping field-of-view multi-channel vehicle cameras. Furthermore, by jointly optimizing the first camera parameters of each vehicle camera through bundle adjustment to determine the second camera parameters, calibration of the multi-channel vehicle camera system can be achieved without any hardware synchronization or physical target placement, thereby improving the efficiency of vehicle multi-channel camera system calibration. This application solves the problem of low efficiency in calibrating multi-channel vehicle camera systems using physical targets in related technologies, achieving the technical effect of improving the efficiency of vehicle multi-channel camera system calibration.

[0016] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of an in-vehicle camera calibration system according to an embodiment of this application; Figure 2 This is a flowchart of a vehicle-mounted camera calibration method according to an embodiment of this application; Figure 3 This is a flowchart illustrating the determination of first camera parameters for a vehicle-mounted camera according to one embodiment of this application; Figure 4 This is a flowchart illustrating the determination of second camera parameters for a vehicle-mounted camera according to one embodiment of this application; Figure 5 This is a schematic diagram of the structure of a vehicle-mounted camera calibration device according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] Road patrol vehicles, autonomous vehicles, or driverless vehicles are typically equipped with multiple cameras. In these vehicle-mounted multi-camera systems, each camera needs to be precisely calibrated to obtain its intrinsic and extrinsic parameters. Traditional camera calibration methods rely on physical targets such as checkerboard patterns, which require careful arrangement in a controlled environment. This is extremely inconvenient for on-site implementation, has low calibration efficiency, and is not suitable for rapid roadside deployment and retesting.

[0026] Currently, no effective solution has been proposed to address the low efficiency of calibrating vehicle-mounted multi-channel camera systems using physical targets in related technologies.

[0027] In view of this, embodiments of this application provide a vehicle-mounted camera calibration method, which can acquire 3D point cloud data obtained by an electronic device performing 3D scanning on a target vehicle including multiple vehicle-mounted cameras; can construct a 3D model of the target vehicle and a vehicle coordinate system based on the 3D point cloud data; can acquire a first video stream captured by the electronic device and a second video stream captured by multiple vehicle-mounted cameras after the target vehicle starts and moves; can determine the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the 3D model; can determine the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system based on the target pose flow, the first video stream, and the second video stream; and can perform bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system. In this way, a 3D model of the target vehicle is obtained by scanning the target vehicle with electronic devices, and the target vehicle's pose flow is generated by continuously tracking the target vehicle's motion using electronic devices. This dynamic reference can replace traditional physical targets and naturally bridges all non-overlapping field-of-view multi-channel vehicle cameras. Furthermore, by jointly optimizing the first camera parameters of each vehicle camera through bundle adjustment to determine the second camera parameters, calibration of the multi-channel vehicle camera system can be achieved without any hardware synchronization or physical target placement, thereby improving the efficiency of vehicle multi-channel camera system calibration. This application solves the problem of low efficiency in calibrating multi-channel vehicle camera systems using physical targets in related technologies, achieving the technical effect of improving the efficiency of vehicle multi-channel camera system calibration.

[0028] The following describes an exemplary application architecture of the vehicle-mounted camera calibration system provided in this application embodiment. (See also...) Figure 1 , Figure 1 This is a schematic diagram of a vehicle-mounted camera calibration system according to an embodiment of this application. The system includes an electronic device 110 and a server 120. The electronic device 110 performs a 3D scan of a target vehicle 130 to obtain 3D point cloud data of the target vehicle 130. Furthermore, the electronic device 110 also captures images of the target vehicle 130 to obtain a first video stream. The electronic device 110 can establish a connection with the server 120 via a wired or wireless network. The target vehicle 130 may include multiple vehicle-mounted cameras with non-overlapping fields of view.

[0029] The electronic device 110 can be a terminal device with three-dimensional scanning and shooting functions. For example, the electronic device 110 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as wearable devices, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) devices, etc., which can install application software. The embodiments of this application do not impose special limitations on the specific form of the electronic device 110.

[0030] In this embodiment, the user can hold the electronic device 110 and start the 3D scanning application software in the electronic device 110. When the target vehicle 130 stops moving, the user can slowly walk around the vehicle to keep the electronic device 110 taking stable pictures of the target vehicle 130 and obtain 3D point cloud data covering all angles of the target vehicle 130. Then, the server 120 can process the 3D point cloud data of the target vehicle 130 from all angles to generate a 3D model with real physical scale (unit: meter).

[0031] After the target vehicle 130 starts or moves, the user can use a handheld electronic device 110 to capture the movement trajectory of the target vehicle 130 in real time to obtain the first video stream.

[0032] Server 120 can be a single server, or it can be a server cluster consisting of multiple servers (or microservers). The server cluster can also be a distributed cluster. This application embodiment does not limit the specific implementation of server 120.

[0033] In this embodiment, server 120 can acquire 3D point cloud data obtained by electronic device 110 performing 3D scanning on target vehicle 130, and construct a 3D model and vehicle coordinate system of target vehicle 130 based on the 3D point cloud data; server 120 can also acquire a first video stream captured by electronic device 110 of target vehicle 130 and a second video stream captured by multiple vehicle-mounted cameras on target vehicle 130 after target vehicle 130 starts and drives; and determine the target pose flow of target vehicle 130 in world coordinate system based on the first video stream and 3D model; and determine the first camera parameters of each vehicle-mounted camera in vehicle coordinate system based on the target pose flow, the first video stream and the second video stream; and perform bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in vehicle coordinate system.

[0034] In practical implementation, both server 120 and electronic device 110 can adopt... Figure 6 The composition results shown may include Figure 6 The components shown. Figure 6 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. When the terminal device 6 has the function of the server 120 described in the embodiment of this application, the terminal device 6 can be the server 120 or a chip or system-on-a-chip in the server 120; when the terminal device 6 has the function of the electronic device 110 described in the embodiment of this application, the terminal device 6 can be the electronic device 110 or a chip or system-on-a-chip in the electronic device 110.

[0035] In some embodiments, the server 120 provided in this application can be implemented in software, for example, it can be mounted on... Figure 6 The server 120, housed in the memory 61 of the terminal device 6 shown, can be software in the form of programs and plug-ins, including: a first acquisition module 50, a construction module 51, a second acquisition module 52, a first determination module 53, a second determination module 54, and an optimization module 55. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.

[0036] The functions of each module will be explained below.

[0037] The following will combine Figure 1 This application describes a method for calibrating a vehicle-mounted camera according to one embodiment. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart of a vehicle-mounted camera calibration method according to an embodiment of this application, such as... Figure 2 As shown, the method includes: Step S201: Obtain 3D point cloud data obtained by the electronic device performing a 3D scan of the target vehicle, the target vehicle including multiple vehicle-mounted cameras.

[0038] In this embodiment, the multiple vehicle-mounted cameras can be multi-channel cameras with non-overlapping fields of view. Users can use handheld electronic devices (e.g., smartphones) to walk around the stationary target vehicle and perform a rapid 3D scan, obtaining 3D point cloud data of the target vehicle.

[0039] Step S202: Based on the 3D point cloud data, construct a 3D model of the target vehicle and a vehicle coordinate system.

[0040] In this embodiment, a three-dimensional model (i.e., a reference vehicle model) of the target vehicle can be constructed based on the three-dimensional point cloud data obtained by rapidly scanning the target vehicle using an electronic device. This three-dimensional model has a true physical scale (unit: meter).

[0041] Then, a vehicle coordinate system can be constructed based on the 3D model of the target vehicle. As an example, the center of the front axle of the target vehicle can be used as the origin of the vehicle coordinate system. The x-axis of the vehicle coordinate system represents the direction perpendicularly forward from the origin, the y-axis represents the direction perpendicularly to the left from the origin, and the z-axis represents the direction perpendicularly upward from the origin.

[0042] Step S203: After the target vehicle starts and drives, acquire the first video stream obtained by the electronic device capturing the target vehicle, and the second video stream captured by multiple vehicle-mounted cameras.

[0043] In this embodiment, after the three-dimensional model and vehicle coordinate system of the target vehicle are constructed, the calibration session can begin. The driver can enter the target vehicle, start the target vehicle and start driving. Visual time difference is created through vehicle movement (e.g., turning, braking, acceleration, etc.) to facilitate subsequent calibration.

[0044] During the movement of the target vehicle, the Visual-Inertial Odometry (VIO) / Simultaneous Localization and Mapping (SLAM) application software (e.g., ARKit / ARCore) on the electronic device can be activated to capture the movement trajectory of the target vehicle in real time, thus obtaining the first video stream; at the same time, multiple on-board cameras on the target vehicle can be activated to capture a second video stream with different fields of view.

[0045] Step S204: Based on the first video stream and the 3D model, determine the target pose flow of the target vehicle in the world coordinate system.

[0046] In this embodiment, the target vehicle's pose flow in the world coordinate system can be obtained by inputting the first video stream and the 3D model into an object 6D (6 Degrees of Freedom) pose estimator (e.g., MegaPose) trained on large-scale synthetic data.

[0047] As an example, step S204 above may include: acquiring the first pose flow of the electronic device in the world coordinate system during the capture of the first video stream; inputting the first video stream and the three-dimensional model into the pose estimation model to obtain the second pose flow of the target vehicle relative to the electronic device, wherein the first pose flow and the second pose flow have the same scale as the three-dimensional model; and determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the second pose flow.

[0048] In this embodiment, after launching the VIO / SLAM application software on the electronic device, the first attitude flow of the electronic device in the world coordinate system can be obtained in real time. Then, a pose estimation model (e.g., MegaPose) can be run, inputting the first video stream and the 3D model into the pose estimation model to obtain the coarse pose of the target vehicle relative to the electronic device, i.e., the second pose stream, output by the pose estimation model. .

[0049] It should be noted that both the first and second pose flows are at the same scale as the 3D model. This means that the scale of both the first and second pose flows can be the real physical scale (unit: meters). Since the pose output by the VIO / SLAM application software on electronic devices is usually of undefined scale (unknown scale factor), the Umeyama algorithm can be used to align the scale with the 3D model. Assuming the scan point set of the 3D model is... The set of observation points output by the VIO / SLAM application software is Then the similarity transformation can be solved: ; constraint: (Rotation group) The scaling factor can be obtained by solving for it. Rotational transformation amount Translation and shift transformation quantities .

[0050] In obtaining scale factor Rotational transformation amount Translation and shift transformation quantities These parameters can then be applied to the pose output by the VIO / SLAM application software, thereby obtaining the first and second pose flows with the same scale as the 3D model. By unifying the pose scale of the electronic device and the target vehicle with the scale of the 3D model, it can be ensured that all physical quantities related to the vehicle coordinate system have the same real physical scale during subsequent calibration, thus achieving scale anchoring.

[0051] In one embodiment, the second pose flow can also be analyzed using keypoint detection and feature matching methods. Optimization is performed to improve the accuracy of the acquired target vehicle's attitude, thereby providing high-quality input for subsequent calibration.

[0052] As an example, determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the second pose flow includes: detecting a first key point from a first image frame in a first video stream, and using the first key point to perform feature matching on a 3D model to obtain a second key point; optimizing the second pose flow based on the first key point and the second key point corresponding to each first image frame in the first video stream, with the goal of minimizing the reprojection error, to obtain the third pose flow of the target vehicle in the world coordinate system; and determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the third pose flow.

[0053] In this embodiment, a keypoint detection model (e.g., SuperPoint) can be used to detect a first keypoint on a first image frame in the first video stream, and a deep learning feature matching method (e.g., LightGlue) can be used to match the first keypoint in a pre-rendered reference map of the 3D model (which may be a vehicle image that visually most closely approximates the viewpoint of the first image frame). The corresponding second key point This allows us to obtain the 2D to 3D correspondence of the first image frame. .

[0054] Then, based on the 2D to 3D correspondences of each first image frame in the first video stream... The second pose flow is then optimized using Perspective-n-Point (PnP) optimization. For example, robust PnP or EPro-PnP algorithms can be used to optimize the second pose flow with the goal of minimizing reprojection error. ; in, For the third pose flow, This is a special Euclidean group. Use a robust kernel function (e.g., Cauchy, Huber, or Tukey loss function). For projection function, The intrinsic parameters of the camera in the electronic device are known and fixed quantities.

[0055] In obtaining the third pose flow Then, the third pose can be flowed. With the first posture flow By combining the data, the target vehicle's pose flow in the world coordinate system is obtained. : .

[0056] It should be noted that the above... Indicates from vehicle logo system To the world coordinate system The rigid body transformation matrix, whose physical meaning is the precise 6D pose of the target vehicle in the world coordinate system (including 3D rotation and 3D translation). Represents the coordinate system of the electronic device To the world coordinate system The rigid body transformation matrix, whose physical meaning is the 6D pose of the electronic device in the world coordinate system; Represents the coordinate system of the electronic device To vehicle coordinate system The rigid body transformation matrix is ​​physically represented by the 6D pose of the target vehicle relative to the electronic equipment.

[0057] It should be noted that during the subsequent calibration process, the camera's intrinsic parameters... Distortion parameters and target pose flow These parameters are all considered as known quantities from Augmented Reality (AR) technology in electronic devices, and are assumed to be correct and fixed. Therefore, these parameters are not involved in subsequent Bundle Adjustment (BA) optimization.

[0058] Step S205: Based on the target pose stream, the first video stream, and the second video stream, determine the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system.

[0059] In this embodiment, the first camera parameters of each vehicle-mounted camera include a first intrinsic parameter, a first distortion parameter, a first time offset, a first extrinsic parameter, and a first planar parameter. The first intrinsic parameter, the first distortion parameter, the first time offset, or the first extrinsic parameter of each vehicle-mounted camera may be different, while the first planar parameter applies to all vehicle-mounted cameras, that is, the first planar parameter of all vehicle-mounted cameras is the same.

[0060] In this embodiment, the intrinsic parameters of all vehicle cameras can be initialized using a depth calibration model to obtain the first intrinsic parameters and the first distortion parameters of each vehicle camera; the planar feature tracking method can also be used to perform feature tracking of the road surface area of ​​the second video stream corresponding to each vehicle camera, and solve for the first extrinsic parameters and the first planar parameters of each vehicle camera.

[0061] Figure 3 This is a flowchart illustrating the determination of first camera parameters for a vehicle-mounted camera according to one embodiment of this application, as shown below. Figure 3 As shown, step S205 above may include: Step S301: Input the image frames in the second video stream acquired by each vehicle-mounted camera into the depth calibration model to obtain the first intrinsic parameter and the first distortion parameter of each vehicle-mounted camera.

[0062] In this embodiment, a depth calibration model (e.g., DeepCalib) can be used to estimate the first intrinsic parameters of the vehicle camera corresponding to the second video stream from a single image in the second video stream. and the first distortion parameter .

[0063] Step S302: Based on the target pose stream, determine the first time offset between the second video stream acquired by each vehicle camera and the first video stream.

[0064] In this embodiment, the first time offset of each vehicle camera can be initially estimated by utilizing the cross-correlation between the motion proxy signal of the electronic device and the planar flow motion signal of each vehicle camera.

[0065] Specifically, the yaw rate of the target vehicle can be determined from the target pose flow. The linear acceleration is then used to estimate the target vehicle's in-field yaw rate using optical flow estimation methods (e.g., RAFT (Recurrent All-Pairs Field Transforms)). Then, the cross-correlation between the two parameters is calculated: ; The calculated peak position is the location of the vehicle-mounted camera. First time offset Vehicle-mounted camera Local time Global time corresponding to electronic devices .

[0066] Step S303: Based on the road surface plane assumption, the first extrinsic parameter of each vehicle camera in the vehicle coordinate system is determined based on the target pose flow, the first intrinsic parameter and the first time offset of each vehicle camera, and the first plane parameter of the road surface on which the target vehicle is traveling in the vehicle coordinate system. The road surface plane assumption means: assuming that the road surface on which the target vehicle is traveling is a mathematical plane.

[0067] In this embodiment, it can be assumed that the road surface on which the target vehicle is traveling is a mathematical plane. Based on this road surface plane assumption and the first time offset of the vehicle camera, the matching frame pair between the vehicle camera and the electronic device is obtained. Based on the matching frame pair, a planar homography matrix is ​​constructed, and the first extrinsic parameter and the first planar parameter of the vehicle camera are solved.

[0068] As an example, step S303 above may include: for each vehicle-mounted camera, selecting the current image frame and the next image frame from the second video stream acquired by the vehicle-mounted camera; based on the first time offset of the vehicle-mounted camera, determining the current pose data corresponding to the current image frame and the next pose data corresponding to the next image frame from the target pose stream; based on the road surface plane assumption, constructing a planar homography matrix between the current image frame and the next image frame based on the current pose data, the next pose data, and the first intrinsic parameters of the vehicle-mounted camera; solving the planar homography matrix corresponding to each vehicle-mounted camera to calculate the first extrinsic parameters of each vehicle-mounted camera in the vehicle coordinate system, and the first planar parameters of the road surface on which the target vehicle is traveling in the vehicle coordinate system.

[0069] In this embodiment, for each vehicle-mounted camera, data can be obtained from the vehicle-mounted camera. Select the current image frame (corresponding to local time) from the acquired second video stream. ) and the next image frame (corresponding to local time) ), and based on the first time offset Determine the current pose data corresponding to the current image frame from the target pose stream. and the next pose data corresponding to the next image frame. .

[0070] Then, based on the above parameters, vehicle motion constraints (known and correct) are constructed: ; And construct motion constraints for the vehicle-mounted camera (depending on an unknown first extrinsic parameter). ): ; And construct road surface plane constraints: Assume the road surface is: Therefore, in the vehicle-mounted camera coordinate system: , ; Based on all the above constraints, construct the planar homography matrix between the current image frame and the next image frame: ( )= ; Solving for the aforementioned planar homography matrix allows us to determine the first extrinsic parameter of the vehicle-mounted camera in the vehicle coordinate system. And the first plane parameters of the road surface where the target vehicle travels in the vehicle coordinate system. , .

[0071] Step S206: Perform bundle adjustment optimization on the first camera parameters of each vehicle camera to obtain the second camera parameters of each vehicle camera in the vehicle coordinate system.

[0072] In this embodiment, the first internal parameter can be used. and the first distortion parameter As a priori, the first camera parameters are jointly optimized across multiple vehicle-mounted cameras.

[0073] Figure 4 This is a flowchart illustrating the determination of second camera parameters for a vehicle-mounted camera according to one embodiment of this application, as shown below. Figure 4 As shown, step S206 above includes: Step S401: For each vehicle-mounted camera, construct the matrix residual based on the planar homography matrix corresponding to the vehicle-mounted camera.

[0074] In this embodiment, for each vehicle-mounted camera, a matrix residual can be constructed based on the planar homography matrix corresponding to the vehicle-mounted camera: .

[0075] in, homogeneous coordinates Convert to image coordinates .

[0076] Step S402: Construct the objective function based on the matrix residual.

[0077] Then, the objective function can be constructed based on the matrix residuals described above: ; The objective function includes matrix residuals, regularized priors, and may also include optional reprojection and rolling shutter terms.

[0078] in, For matrix residuals; This is a robust kernel function used to reduce the impact of outliers; Point weights (determined by tracking confidence, etc.); Prior regularization for the first intrinsic parameter (anchored to the first intrinsic parameter) and the first distortion parameter ); Regularization of the first time offset (anchored to the first time offset) ), Prior to the road surface plane parameters (e.g., the normal is close to vertically upward).

[0079] In addition, if there is common viewing between the vehicle-mounted cameras, cross-projection residuals can be added; if the vehicle-mounted camera is a rolling shutter camera, the time per line can be introduced and jointly optimized: the rolling shutter camera exposes line by line, which will cause different lines in the same frame to correspond to different acquisition times.

[0080] Assuming the second video stream of the vehicle-mounted camera contains the first... Frame image frame timestamp (Usually corresponds to the middle row), normalized row coordinates [0,1] (0 represents the top row, 1 represents the bottom row), then the first row... The exposure time for the line is: + ; in, The readouttime is the time span from the first line to the last line (in milliseconds, which needs to be estimated or obtained from the configuration of the vehicle camera).

[0081] In bundle adjustment optimization: for each residual point, interpolation can be performed based on its image row coordinates to obtain... And based on the third pose flow corresponding to the interpolation. And calculate the corresponding planar homography matrix. If Unknown variables can also be used as optimization variables for joint solution.

[0082] Additionally, a learned RS correction stream (e.g., Caoetal.2024) can be used to pre-distort images or provide line-time priors.

[0083] Step S403: Using the target pose flow as a fixed value, and the first intrinsic parameter, first distortion parameter, first time offset, first extrinsic parameter, and first plane parameter of the vehicle camera as optimization variables, bundle adjustment optimization is performed by minimizing the objective function to obtain the second camera parameters of each vehicle camera in the vehicle coordinate system.

[0084] In this embodiment, the matrix residuals constructed based on the planar homography matrix can directly correlate the observation errors at the image level with the underlying geometric parameters (extrinsic and planar parameters) and intrinsic parameters, providing precise gradient guidance for joint optimization. Furthermore, fixing the third pose flow of the target vehicle as an invariant reference avoids ill-conditioned convergence caused by simultaneous fluctuations in all parameters. Moreover, the joint optimization based on bundle adjustment can overcome the cumulative effect of error propagation in step-by-step calibration, enabling the intrinsic, extrinsic, and temporal offset parameters of the vehicle-mounted camera to mutually correct each other under global constraints. This ultimately achieves high-precision calibration of the vehicle-mounted camera while ensuring the consistency of the second imaging parameters of each vehicle-mounted camera within a unified geometric framework.

[0085] In one embodiment, after step S206 above, the method may further include: calculating the calibration quality index of each vehicle camera based on the second camera parameters of each vehicle camera in the vehicle coordinate system; and outputting the calibration result of the vehicle camera based on the comparison result of the calibration quality index and a preset threshold.

[0086] In this embodiment, the calibration quality index may include reprojection error (root mean square error, RMS): ; The total number of samples represents the total number of feature points successfully tracked and included in the calculation by the vehicle-mounted camera across all matching frame pairs used for optimization. This is the reprojection error.

[0087] The calibration quality indicators may also include time offset consistency indicators: these can detect the second time offset in the second camera parameters. offset from the first time Differences between them: It is recommended that this value be less than 50 milliseconds; at the same time, the sharpness (signal-to-noise ratio) of the cross-correlation peak can be detected, and a low signal-to-noise ratio will prompt a resampling.

[0088] Calibration quality metrics can also include plane normal stability metrics: these can be used to divide the data into several subsets and independently estimate the plane parameter normals. Calculate the variance of the angle between subsets, and suggest <2° (indicating that the plane assumption is consistent with the extrinsic parameters).

[0089] The calibration quality indicators can also include coverage indicators: the proportion of effective matching frames and the spatial distribution density of road feature points can be statistically analyzed. It is recommended that the coverage be >50% to ensure uniform spatial distribution.

[0090] The calibration quality index can also include the parameter uncertainty index: the covariance is approximated by the Hessian matrix, and extracted... and The confidence interval / standard deviation is given, along with a threshold; high uncertainty suggests the need for additional stimulus motion.

[0091] Based on the above calibration quality indicators, different preset thresholds can be set. For example, if the reprojection error is less than 0.6 pixels (sub-pixel accuracy), the plane normal stability index is less than 2°, the coverage index is greater than 50%, and the time offset consistency index (also known as time offset) is less than 50 milliseconds, then the calibration is considered successful, and a calibration package and quality inspection report can be output. If the reprojection error is between 0.6 pixels and 1 pixel, or the coverage is less than 50%, it is recommended to collect additional data and recalibrate. If the reprojection error is greater than 1 pixel or the plane normal stability index is greater than 5°, it is recommended to recalibrate and improve the motion excitation.

[0092] By setting calibration quality indicators and preset thresholds, it is easy to test the calibration quality of the vehicle-mounted multi-channel camera system, so as to control and retest on site.

[0093] In addition, the vehicle-mounted camera calibration method provided in this application has strong scalability and low cost. It only requires short-term recording using electronic devices to complete the calibration of the vehicle-mounted multi-channel camera system without any hardware synchronization or physical target placement. Therefore, the calibration efficiency of the vehicle-mounted multi-channel camera system is high.

[0094] Through the above steps S201 to S206, three-dimensional point cloud data obtained by the electronic device through three-dimensional scanning of a target vehicle including multiple vehicle-mounted cameras can be acquired; a three-dimensional model of the target vehicle and a vehicle coordinate system can be constructed based on the three-dimensional point cloud data; after the target vehicle starts and moves, a first video stream captured by the electronic device and a second video stream captured by multiple vehicle-mounted cameras can be acquired; the target pose flow of the target vehicle in the world coordinate system can be determined based on the first video stream and the three-dimensional model; the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system can be determined based on the target pose flow, the first video stream, and the second video stream; the first camera parameters of each vehicle-mounted camera can be optimized by bundle adjustment to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system. In this way, a 3D model of the target vehicle is obtained by scanning the target vehicle with electronic devices, and the target vehicle's pose flow is generated by continuously tracking the target vehicle's motion using electronic devices. This dynamic reference can replace traditional physical targets and naturally bridges all non-overlapping field-of-view multi-channel vehicle cameras. Furthermore, by jointly optimizing the first camera parameters of each vehicle camera through bundle adjustment to determine the second camera parameters, calibration of the multi-channel vehicle camera system can be achieved without any hardware synchronization or physical target placement, thereby improving the efficiency of vehicle multi-channel camera system calibration. This application solves the problem of low efficiency in calibrating multi-channel vehicle camera systems using physical targets in related technologies, achieving the technical effect of improving the efficiency of vehicle multi-channel camera system calibration.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] Corresponding to the vehicle-mounted camera calibration method described in the above embodiments, Figure 5 A schematic diagram of a vehicle-mounted camera calibration device according to an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0097] Please see Figure 5 The vehicle-mounted camera calibration device 5 includes: a first acquisition module 50, used to acquire three-dimensional point cloud data obtained by an electronic device performing a three-dimensional scan of a target vehicle, the target vehicle including multiple vehicle-mounted cameras; a construction module 51, used to construct a three-dimensional model of the target vehicle and a vehicle coordinate system based on the three-dimensional point cloud data; a second acquisition module 52, used to acquire a first video stream obtained by the electronic device capturing the target vehicle after the target vehicle starts and moves, and a second video stream captured by multiple vehicle-mounted cameras; a first determination module 53, used to determine the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the three-dimensional model; a second determination module 54, used to determine the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system based on the target pose flow, the first video stream, and the second video stream; and an optimization module 55, used to perform bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system.

[0098] In one embodiment, the first camera parameters include a first intrinsic parameter, a first distortion parameter, a first time offset, a first extrinsic parameter, and a first plane parameter; the second determining module 54 is further configured to input the image frames in the second video stream acquired by each vehicle-mounted camera into the depth calibration model to obtain the first intrinsic parameter and the first distortion parameter of each vehicle-mounted camera; based on the target pose stream, determine the first time offset between the second video stream acquired by each vehicle-mounted camera and the first video stream; according to the road surface plane assumption, based on the target pose stream, and the first intrinsic parameter and the first time offset of each vehicle-mounted camera, determine the first extrinsic parameter of each vehicle-mounted camera in the vehicle coordinate system, and the first plane parameter of the road surface on which the target vehicle is traveling in the vehicle coordinate system, wherein the road surface plane assumption means: assuming that the road surface on which the target vehicle is traveling is a mathematical plane.

[0099] In one embodiment, the second determining module 54 is further configured to, for each vehicle-mounted camera, select the current image frame and the next image frame from the second video stream acquired by the vehicle-mounted camera; determine the current pose data corresponding to the current image frame and the next pose data corresponding to the next image frame from the target pose stream based on the first time offset of the vehicle-mounted camera; construct a planar homography matrix between the current image frame and the next image frame based on the road surface plane assumption, the current pose data, the next pose data, and the first intrinsic parameters of the vehicle-mounted camera; solve the planar homography matrix corresponding to each vehicle-mounted camera to calculate the first extrinsic parameters of each vehicle-mounted camera in the vehicle coordinate system, and the first planar parameters of the road surface on which the target vehicle is traveling in the vehicle coordinate system.

[0100] In one embodiment, the optimization module 55 is further configured to construct a matrix residual based on the planar homography matrix corresponding to each vehicle camera; construct an objective function based on the matrix residual; take the target pose flow as a fixed value, and take the first intrinsic parameter, first distortion parameter, first time offset, first extrinsic parameter and first planar parameter of the vehicle camera as optimization variables, and perform bundle adjustment optimization by minimizing the objective function to obtain the second camera parameters of each vehicle camera in the vehicle coordinate system.

[0101] In one embodiment, the first determining module 53 is further configured to acquire the first pose flow of the electronic device in the world coordinate system during the capture of the first video stream; input the first video stream and the three-dimensional model into the pose estimation model to obtain the second pose flow of the target vehicle relative to the electronic device, wherein the first pose flow and the second pose flow have the same scale as the three-dimensional model; and determine the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the second pose flow.

[0102] In one embodiment, the first determining module 53 is further configured to detect a first key point from a first image frame in the first video stream, and use the first key point to perform feature matching on a 3D model to obtain a second key point; based on the first key point and the second key point corresponding to each first image frame in the first video stream, optimize the second pose flow with the goal of minimizing the reprojection error to obtain a third pose flow of the target vehicle in the world coordinate system; and determine the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the third pose flow.

[0103] In one embodiment, the vehicle-mounted camera calibration device 5 further includes an evaluation module for calculating the calibration quality index of each vehicle-mounted camera based on the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system; and outputting the calibration result of the vehicle-mounted camera based on the comparison result of the calibration quality index and a preset threshold.

[0104] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] Figure 6 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Figure 6 As shown, the terminal device 6 includes: at least one processor 60 ( Figure 6 (Only one is shown in the image) a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on at least one processor 60. When the processor 60 executes the computer program 62, it implements the steps in any of the above embodiments of the vehicle camera calibration method.

[0107] Terminal device 6 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 6 and does not constitute a limitation on terminal device 6. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0108] The processor 60 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0109] In some embodiments, memory 61 may be an internal storage unit of terminal device 6, such as a hard disk or memory of terminal device 6. In other embodiments, memory 61 may be an external storage device of terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on terminal device 6. In other embodiments, memory 61 may include both internal and external storage units of terminal device 6. Memory 61 is used to store operating system, applications, bootloader, data, and other programs, such as the program code of computer program 62. Memory 61 may also be used to temporarily store data that has been output or will be output.

[0110] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above embodiments of the vehicle-mounted camera calibration methods.

[0111] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described vehicle camera calibration method embodiments.

[0112] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a vehicle-mounted camera calibration device or terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calibrating a vehicle-mounted camera, characterized in that, include: Acquire 3D point cloud data obtained by electronic devices performing 3D scanning of a target vehicle, wherein the target vehicle includes multiple vehicle-mounted cameras; Based on the three-dimensional point cloud data, a three-dimensional model of the target vehicle and a vehicle coordinate system are constructed. After the target vehicle starts and drives, a first video stream obtained by the electronic device capturing the target vehicle, and a second video stream captured by multiple vehicle-mounted cameras are acquired. Based on the first video stream and the 3D model, the target pose flow of the target vehicle in the world coordinate system is determined; Based on the target pose stream, the first video stream, and the second video stream, determine the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system; Bundle adjustment optimization is performed on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system.

2. The method according to claim 1, characterized in that, The first camera parameters include a first intrinsic parameter, a first distortion parameter, a first time offset, a first extrinsic parameter, and a first planar parameter; determining the first camera parameters of each vehicle-mounted camera in the vehicle coordinate system based on the target pose stream, the first video stream, and the second video stream includes: The image frames in the second video stream acquired by each of the vehicle-mounted cameras are input into the depth calibration model to obtain the first intrinsic parameter and the first distortion parameter of each of the vehicle-mounted cameras; Based on the target pose stream, determine the first time offset between the second video stream acquired by each of the vehicle-mounted cameras and the first video stream; Based on the road surface plane assumption, and based on the target pose flow, as well as the first intrinsic parameter and the first time offset of each vehicle-mounted camera, the first extrinsic parameter of each vehicle-mounted camera in the vehicle coordinate system, and the first plane parameter of the road surface on which the target vehicle travels in the vehicle coordinate system are determined. The road surface plane assumption means that the road surface on which the target vehicle travels is assumed to be a mathematical plane.

3. The method according to claim 2, characterized in that, The step of determining the first extrinsic parameters of each vehicle-mounted camera in the vehicle coordinate system based on the road surface plane assumption, the target pose flow, and the first intrinsic parameters and the first time offset of each vehicle-mounted camera, as well as the first planar parameters of the road surface on which the target vehicle travels in the vehicle coordinate system, includes: For each of the vehicle-mounted cameras, the current image frame and the next image frame are selected from the second video stream captured by the vehicle-mounted camera; Based on the first time offset of the vehicle-mounted camera, the current pose data corresponding to the current image frame and the next pose data corresponding to the next image frame are determined from the target pose stream. Based on the road surface plane assumption, and using the current pose data, the next pose data, and the first intrinsic parameter of the vehicle camera, a planar homography matrix between the current image frame and the next image frame is constructed. Solve the planar homography matrix corresponding to each vehicle-mounted camera to calculate the first extrinsic parameter of each vehicle-mounted camera in the vehicle coordinate system, and the first planar parameter of the road surface on which the target vehicle travels in the vehicle coordinate system.

4. The method according to claim 3, characterized in that, The step of performing bundle adjustment optimization on the first camera parameters of each vehicle-mounted camera to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system includes: For each of the vehicle-mounted cameras, a matrix residual is constructed based on the planar homography matrix corresponding to the vehicle-mounted camera; Construct an objective function based on the matrix residuals; Using the target pose flow as a fixed value, and the first intrinsic parameter, the first distortion parameter, the first time offset, the first extrinsic parameter, and the first planar parameter of the vehicle-mounted camera as optimization variables, the bundle adjustment optimization is performed by minimizing the objective function to obtain the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the 3D model includes: Acquire the first pose flow of the electronic device in the world coordinate system during the capture of the first video stream; The first video stream and the 3D model are input into the pose estimation model to obtain the second pose stream of the target vehicle relative to the electronic device. The first pose stream and the second pose stream are at the same scale as the 3D model. Based on the first pose flow and the second pose flow, the target pose flow of the target vehicle in the world coordinate system is determined.

6. The method according to claim 5, characterized in that, The step of determining the target pose flow of the target vehicle in the world coordinate system based on the first pose flow and the second pose flow includes: A first key point is detected from a first image frame in the first video stream, and a second key point is obtained by feature matching on the 3D model using the first key point. Based on the first key points and the second key points corresponding to each first image frame in the first video stream, the second pose flow is optimized with the goal of minimizing the reprojection error, to obtain the third pose flow of the target vehicle in the world coordinate system. Based on the first pose flow and the third pose flow, the target pose flow of the target vehicle in the world coordinate system is determined.

7. The method according to any one of claims 1 to 4, characterized in that, After performing bundle adjustment optimization on the first camera parameters of each of the vehicle-mounted cameras to obtain the second camera parameters of each of the vehicle-mounted cameras in the vehicle coordinate system, the method further includes: Based on the second camera parameters of each vehicle-mounted camera in the vehicle coordinate system, calculate the calibration quality index of each vehicle-mounted camera; Based on the comparison results between the calibration quality index and the preset threshold, the calibration result of the vehicle-mounted camera is output.

8. A vehicle-mounted camera calibration device, characterized in that, include: The first acquisition module is used to acquire three-dimensional point cloud data obtained by the electronic device performing a three-dimensional scan of the target vehicle, wherein the target vehicle includes multiple vehicle-mounted cameras. A construction module is used to construct a 3D model and vehicle coordinate system of the target vehicle based on the 3D point cloud data; The second acquisition module is used to acquire, after the target vehicle starts and drives, a first video stream obtained by the electronic device shooting the target vehicle, and a second video stream collected by multiple vehicle-mounted cameras; The first determining module is used to determine the target pose flow of the target vehicle in the world coordinate system based on the first video stream and the three-dimensional model; The second determining module is used to determine the first camera parameters of each of the vehicle cameras in the vehicle coordinate system based on the target pose stream, the first video stream and the second video stream; An optimization module is used to perform bundle adjustment optimization on the first camera parameters of each of the vehicle-mounted cameras to obtain the second camera parameters of each of the vehicle-mounted cameras in the vehicle coordinate system.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the vehicle-mounted camera calibration method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The system includes a computer program that, when executed, causes the vehicle-mounted camera calibration method as described in any one of claims 1 to 7 to be performed.

Images