Vehicle surround view camera self-calibration method and device suitable for underground garage environment

Abstract

The application provides a vehicle surround view camera self-calibration method and device suitable for a garage environment, relates to the technical field of intelligent driving application, and comprises the following steps: constraining an inertial navigation equation under an ESKF filtering model framework through wheel speed of a current vehicle, and determining a navigation state value of the current vehicle; correlating prior information of a gravity direction of the current vehicle with a vertical line element in a camera structure line element, and estimating a vertical main direction of a current scene; estimating a horizontal main direction of the current scene based on the vertical main direction of the current scene and projection clustering of line features in the horizontal main direction; and performing self-calibration on a camera pose of the current vehicle based on a constraint relationship of the camera pose on the inertial navigation equation, the vertical main direction of the current scene and the horizontal main direction, thereby relieving the technical problem that it is difficult to calibrate an external parameter of a surround view camera to a vehicle body based on a track in the garage environment.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent driving applications, and in particular to a self-calibration method and apparatus for vehicle surround-view cameras in underground parking environments. Background Technology

[0002] In fields such as intelligent driving navigation and high-precision maps, cameras provide rich perceptual information about the vehicle's surroundings. Coupled with their low cost, they have become standard sensors for high-level intelligent driving systems. Based on their distribution on the vehicle body, they are categorized as forward-looking, surround-view, and panoramic cameras, used to perceive dynamic and static scene elements at different distances and orientations around the vehicle. Among these, surround-view cameras, providing a 360-degree field of view at close range, are particularly suitable for applications requiring close-range panoramic perception in low-speed scenarios, making them a core sensor for high-end APA / AVP technology solutions in the automatic parking segment.

[0003] Due to the imaging characteristics of cameras, they do not have the distance attributes of LiDAR. In order to effectively integrate the perception information of the front, rear, left and right cameras of the surround view camera, it is necessary to pre-calibrate the pose relationship of each camera relative to the vehicle body, unify the perception information from the 2D camera perspective to the BEV overhead view, and establish a BEV image with distance attributes from the panoramic perception overhead view, so as to better serve the advanced intelligent driving applications such as intelligent vehicle mapping, positioning and path planning.

[0004] The trajectories of each camera in the surround view camera are calculated using visual odometry. The effectiveness of monocular visual odometry is related to the richness of scene texture. However, the uneven texture distribution, repetitive textures, and changes in lighting conditions in underground parking environments have a significant impact on the accuracy and reliability of traditional feature-point-based monocular visual odometry. In addition, the scale drift of monocular visual odometry itself and the influence of near-plane motion of ground vehicles on the observability of external parameter estimation all make it difficult to calibrate the external parameters of the surround view camera to the vehicle body based on trajectory in underground parking environments. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a self-calibration method and device for vehicle surround view cameras in underground parking environments, which fully considers the geometric constraints of Manhattan scenes in underground parking environments and alleviates the technical problem of difficult trajectory-based extrinsic parameter calibration of surround view cameras to vehicle bodies in underground parking environments.

[0006] In a first aspect, the embodiments provide a self-calibration method for a vehicle surround-view camera suitable for underground parking environments, the method comprising:

[0007] The inertial navigation equations under the ESKF filtering model framework are constrained by the current vehicle's wheel speed to determine the current vehicle's navigation state value; wherein, the navigation state value is based on prior information about the current vehicle's gravity direction.

[0008] The prior information of the current vehicle's gravity direction is associated with the vertical line elements in the camera structure line elements to determine the vertical principal direction of the current scene; wherein, the camera structure line elements are features extracted from the current frame image captured by the current vehicle; the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction are clustered to determine the horizontal principal direction of the current scene.

[0009] Based on the inertial navigation equation and the constraints of the vertical and horizontal principal directions of the current scene on the camera attitude, the camera attitude of the current vehicle is self-calibrated.

[0010] In an optional implementation, the step of determining the navigation state value of the current vehicle by constraining the inertial navigation equations under the ESKF filtering model framework using the current vehicle's wheel speed includes:

[0011] Add a proportional coefficient corresponding to the wheel speed to the inertial navigation equation under the ESKF filtering model framework corresponding to the current vehicle;

[0012] The error state measurement equation for the current vehicle is determined based on the wheel speed proportionality coefficient as a constraint condition.

[0013] Based on the error state measurement equation, the current navigation state value of the vehicle is determined.

[0014] In an optional implementation, the step of determining the vertical principal direction of the current scene by associating the prior information of the current vehicle's gravity direction with the vertical line elements in the camera's structural line elements includes:

[0015] Extract line features from the captured image of the current vehicle in the current scene;

[0016] Based on the angle between the current vehicle's gravity direction and the normal vector of the line element on the camera's Gaussian ellipsoid projection plane at a preset time, the vertical line element constrained by the vertical principal direction in the current scene is determined.

[0017] Based on the vertical line elements, constraint equations are constructed to obtain the optimized gravity component of the current scene in the vertical principal direction.

[0018] In an optional implementation, the step of clustering the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction to determine the horizontal principal direction of the current scene includes:

[0019] The vertical principal direction and the horizontal principal direction are parameterized to determine the parametric equation of the horizontal principal direction;

[0020] The numerical distribution range of the horizontal principal direction is determined based on the normal vector corresponding to the projection of the line features of the camera structure line elements in the horizontal principal direction onto the Gaussian sphere.

[0021] Based on the geometric constraints between the horizontal principal direction and the coordinate axes, the approximate value of the current scene in the horizontal principal direction is clustered.

[0022] In an optional implementation, the step of self-calibrating the camera attitude of the current vehicle based on the inertial navigation equations and the constraints of the vertical and horizontal principal directions of the current scene on the camera attitude includes:

[0023] Based on the vertical and horizontal principal directions of the current scene, an attitude cost function is established to calculate the distance from the line features perceived in the current frame to the corresponding vanishing point, thereby optimizing the single-camera attitude of the current vehicle.

[0024] In an optional implementation, the step of self-calibrating the camera attitude of the current vehicle based on the inertial navigation equations and the constraints of the vertical and horizontal principal directions of the current scene on the camera attitude further includes:

[0025] Based on the correlation between the vertical principal direction and the gravity direction, and the relative attitude estimation of the vehicle body at two adjacent time points, a cost function is established to optimize the multi-camera attitude of the current vehicle.

[0026] In an optional implementation, before determining the navigation state value of the current vehicle by constraining the inertial navigation equations under the ESKF filtering model framework using the current vehicle's wheel speed, the method further includes:

[0027] The current vehicle status is statically initialized.

[0028] Secondly, the embodiment provides a vehicle surround view camera self-calibration device suitable for underground parking environments, the device comprising:

[0029] The determination module uses the current vehicle's wheel speed to constrain the inertial navigation equations under the ESKF filtering model framework, thereby determining the current vehicle's navigation state value; wherein, the navigation state value is based on prior information about the current vehicle's gravity direction;

[0030] The first estimation module associates the prior information of the current vehicle's gravity direction with the vertical line elements in the camera structure line elements to determine the vertical principal direction of the current scene; wherein, the camera structure line elements are features extracted from the current frame image captured by the current vehicle.

[0031] The second estimation module clusters the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction to determine the horizontal principal direction of the current scene.

[0032] The optimization module performs self-calibration of the camera attitude of the current vehicle based on the inertial navigation equation and the constraint relationship between the vertical principal direction and the horizontal principal direction of the current scene on the camera attitude.

[0033] Thirdly, an embodiment provides an electronic device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method described in any of the foregoing embodiments.

[0034] Fourthly, an embodiment provides a machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the steps of the method described in any of the foregoing embodiments.

[0035] This invention provides a self-calibration method and device for vehicle surround-view cameras in underground parking garage environments. It enhances the vehicle's trajectory accuracy by constructing an enhanced odometer with wheel speed scale correction using IMU-assisted wheel speed. Simultaneously, by tracking the gravity direction in the inertial device, it quickly associates this with the vertical lines perceived by the Manhattan geometry scene in the parking garage. Furthermore, based on this prior constraint, it quickly associates the scene's horizontal principal direction. This avoids the shortcomings of traditional methods that rely heavily on strict geometric constraints when estimating the principal direction of the Manhattan geometry framework, and the high computational resource consumption and difficulty in real-time calculation of RANSAC search traversal and mean-shift clustering. This ensures the reliability and real-time performance of camera attitude estimation in underground parking garage scenarios.

[0036] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0037] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0038] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 A flowchart of a vehicle surround view camera self-calibration method suitable for underground parking environments is provided in an embodiment of the present invention;

[0040] Figure 2 Flowchart of another vehicle surround view camera self-calibration method applicable to underground parking environments provided by embodiments of the present invention;

[0041] Figure 3 A functional block diagram of a vehicle surround view camera self-calibration device suitable for underground parking environments provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Establishing a unified geometric reference between the coordinate systems of all surround-view cameras and the vehicle coordinate system is a crucial foundation for surround-view camera perception fusion. The pose accuracy of each camera relative to the vehicle is key to determining the geometric accuracy of the BEV panoramic image. Traditionally, surround-view camera calibration requires a dedicated calibration site. This involves calibrating the poses of different cameras relative to manually collaborating markers within the calibration site. Assuming the position of the rear wheel center is known, a vehicle coordinate system is typically established with the rear wheel center as the origin, using the vehicle's forward direction, normal direction, and vertically upward direction from the ground. This yields the poses of different cameras relative to the vehicle. While vehicles typically undergo standardized calibration in a calibration workshop before leaving the factory, the complex environment of actual roads during driving can easily cause relative deflection of the cameras relative to the vehicle. This reduces the accuracy of the original factory calibration parameters, affecting the quality of the final fused perception. Furthermore, traditional calibration methods have specific requirements for the calibration site and the operation of technical personnel, making timely updates difficult and reducing the usability of advanced intelligent driving functions.

[0045] To address the shortcomings of traditional calibration methods, another approach is trajectory correlation based on sensor rigid body motion constraints. This type of method does not require prior preparation of a calibration site. First, the motion trajectories of each sensor on the motion platform are calculated. Then, the relative pose relationships of different sensors are calculated based on hand-eye calibration theory in robotics applications. This type of hand-eye calibration method does not require a dedicated control field, but generally requires sufficient perturbation of the motion platform in three-dimensional translation and rotation directions. The calibration accuracy is related to the accuracy of the sensor motion mode and motion trajectory. Here, the vehicle trajectory is calculated by combining the pulse information of the wheel encoder with the Ackerman dynamics principle of the vehicle. When the radii and spacing of the left and right wheels are known, the pose of the vehicle relative to the starting position is further calculated through a differential wheel speed model. Obviously, due to the measurement errors of the wheel radius and spacing, the scale error of the vehicle trajectory calculation will be caused. In addition, the influence of wheel slippage means that the mileage calculation accuracy based solely on the differential wheel signal model is limited.

[0046] Traditional target calibration requires advance site setup and professional technical personnel to guide the operation. To ensure the accuracy and reliability of self-calibration methods related to motion trajectories, there are high requirements for the complexity of the vehicle's own motion mode and the accuracy of the individual sensor trajectories. However, specific scenarios such as underground parking garages often cannot meet these requirements. It can be seen that current calibration methods cannot simultaneously meet the needs of automation, accuracy, and reliability, and still have certain limitations in practical applications, posing a significant challenge to the usability and safety of mass-produced intelligent driving applications.

[0047] Based on this, the present invention provides a vehicle surround-view camera self-calibration method suitable for underground parking environments. By further integrating the IMU measurement unit on the vehicle body, an enhanced odometer fused with wheel speed measurement IMU is constructed for vehicle trajectory estimation. At the same time, it effectively perceives the scene geometry under the daily operating conditions of intelligent driving. Based on the absolute attitude estimation of the camera in the Manhattan geometry scene, it effectively constrains the attitude drift of the vehicle in the underground parking scene, assists in online monitoring of the quality of surround-view camera extrinsic parameter calibration, and solves the problem of online update when the original camera extrinsic parameters fail during vehicle operation.

[0048] To facilitate understanding of this embodiment, a detailed description of a vehicle surround-view camera self-calibration method suitable for underground parking environments, as disclosed in this embodiment, is provided first. This method can be applied to intelligent control devices such as host computers, servers, and controllers. The inertial navigation equation in this embodiment can be used to characterize the motion constraints of the current vehicle. Based on this inertial navigation equation, the navigation state value of the current vehicle can be determined. This navigation state value provides a reference for the current vehicle trajectory. Modeling is performed using this reference navigation state value, and the model parameters are estimated and adjusted to optimize the optimal extrinsic parameter settings of the camera corresponding to the current vehicle driving state. This enables the self-calibration of the vehicle surround-view camera. This self-calibration can overcome the limitations of signal transmission accuracy in the relatively enclosed underground parking environment, ensuring the reliability of vehicle driving in underground parking environments.

[0049] Figure 1 This is a flowchart of a self-calibration method for a vehicle surround-view camera in an underground parking environment, provided by an embodiment of the present invention.

[0050] like Figure 1 As shown, the method includes the following steps:

[0051] Step S102: The inertial navigation equations under the ESKF filtering model framework are constrained by the current wheel speed of the vehicle to determine the current navigation state value of the vehicle. This navigation state value is based on pre-defined prior information about the current vehicle's gravity direction.

[0052] It should be noted that before step S102, the current vehicle state needs to be statically initialized. Assuming the vehicle is stationary, based on the state constraints under the stationary condition, the initial position, velocity, attitude of the vehicle platform, as well as the zero bias of the gyroscope and accelerometer and the direction of gravity are initialized.

[0053] For example, the gyroscope and accelerometer readings of the six-axis sensor IMU are recorded for 3-5 seconds in a stationary state, at which time the initial drift b of the gyroscope is recorded. g The average measurement value for this interval is set as the mean value. Since the accelerometer readings are mainly affected by local gravity, the direction of gravity in the IMU coordinate system is obtained based on the mean accelerometer readings for this interval. Considering that the modulus of the local gravity measurement is a constant value g norm Thus, the gravitational component is obtained. Subtracting the gravitational component from the average value measured by the accelerometer in this segment yields the initial accelerometer drift b. a Subsequently, the origin O of the coordinate system will be used as the starting position, the radial direction of the vehicle will be the X-axis, the direction of gravity will be the Z-axis, and the Y-axis of the vehicle will be determined by the right-hand rule to perform state estimation.

[0054] After completing the initialization step, step S102 mainly includes: adding a proportional coefficient corresponding to the wheel speed to the inertial navigation equation under the ESKF filtering model framework corresponding to the current vehicle; determining the error state measurement equation of the current vehicle based on the proportional coefficient of the wheel speed as a constraint; and determining the navigation state value of the current vehicle based on the error state measurement equation.

[0055] As an example, after completing static initialization, the system starts and begins motion state recursion. Under the ESKF filtering model framework, the system estimates the IMU inertial navigation equation and uses wheel speed to constrain the error accumulation of inertial navigation. Considering the influence of wheel radius changes caused by vehicle tire pressure changes on wheel speed output values, the proportional coefficient of wheel speed meter output values ​​is used as an additional parameter of the system state to establish the measurement error equation, thereby reducing the scale error of wheel speed measurement.

[0056] Among them, the wheel speed-integrated IMU-enhanced odometer adds a wheel speed proportionality coefficient K to the original inertial navigation state. od The system state is defined according to the following formula:

[0057] x n =(p n v n θ n b g b a k od (1.1)

[0058] The error state of ESKF filtering can be derived using the following formula:

[0059] δp k+1 =δp k +δv k Δt

[0060]

[0061]

[0062]

[0063]

[0064] System state transition matrix equation:

[0065]

[0066] Combining the vehicle's own non-holonomic motion constraints with the wheel speed gauge's velocity constraints in the vehicle's direction of travel:

[0067]

[0068]

[0069]

[0070] Where V is the true value of the vehicle's velocity along the forward direction, thus yielding the measurement equation for wheel speed constraints:

[0071]

[0072] in, It is estimated from the IMU inertial navigation equations, and then the current navigation state value is updated.

[0073] Step S104: The prior information of the current vehicle's gravity direction is associated with the vertical line element in the camera structure line element to determine the vertical principal direction of the current scene. The prior information of the current vehicle's gravity direction can be obtained in advance based on the vehicle's sensor perception. On this basis, associating with the camera structure line element can estimate the vertical principal direction of the current scene. The camera structure line element can be understood as the line feature extracted from the current frame image captured by the current vehicle.

[0074] Firstly, the geometric elements in the underground parking scene are relatively limited. The traditional Manhattan structural framework extraction process often fails to fully satisfy the constraints that the perceived geometric elements in the scene are orthogonal or parallel. However, by tracking the prior information of the IMU gravity direction, we can quickly associate the vertical line elements in the camera structural line elements, reduce the prior requirements for the constraints of geometric elements in the scene, and improve the universality of the application scenarios.

[0075] For example, the LSD line feature extraction algorithm is used to extract line features from the image in the current scene. i (x l1 ,x l2 ), here x l1 x l2 Image coordinates of the two endpoints of the characterization line, and the approximate pose of camera i at the current time k. Here, according to the following formula, L(l,k) is defined as the normal vector of the line element l on the camera's Gaussian ellipsoid projection plane at time k, as shown in the following figure. Figure 2 :

[0076]

[0077] Define the sine of the normal vector relative to the direction of gravity:

[0078]

[0079] Thus, using a difference of 10° as a threshold, we can filter out line features from the current line features that satisfy condition (1.9) and define them as line features constrained by the vertical principal direction in the scene. Therefore, by combining the vertical line features within the sliding window that satisfy the constraint conditions, the following constraint equation is constructed:

[0080]

[0081] Based on this, the optimized gravity component g is obtained.

[0082] Step S106: Cluster the vertical main direction of the current scene and the camera structure line elements projected onto the horizontal main direction to determine the horizontal main direction of the current scene.

[0083] After determining the vertical principal direction of Manhattan in the scene, we obtain the plane containing the horizontal principal direction on the Manhattan ellipsoid perpendicular to this direction. We project the straight lines of the image on the plane onto the Gaussian ellipsoid, and the corresponding vanishing points are clustered by projection on the horizontal principal direction. Considering the orthogonality of the principal directions on this plane, we further converge the model sampling space to obtain the sample set with the best consistency with the horizontal principal direction.

[0084] For example, the vertical principal direction and the horizontal principal direction are parameterized to determine the parametric equation of the horizontal principal direction; the numerical distribution range of the horizontal principal direction is determined according to the normal vector corresponding to the projection of the line features of the horizontal principal direction onto the Gaussian sphere; and the approximate value of the current scene in the horizontal principal direction is clustered based on the geometric constraint relationship between the horizontal principal direction and the coordinate axes.

[0085] After estimating the vertical principal direction (VDD) of the basement scene, we obtain the plane containing the horizontal principal direction. Now, we need to quickly determine the horizontal principal direction (HDD) of the scene within this plane. Here, we first parameterize VDD and HDD. On the Gaussian ellipsoid where the camera is located, we represent VDD as spherical coordinates of the horizontal azimuth angle α and the vertical azimuth angle β:

[0086] v(α,β)=[cosαcosβ,sinαcosβ,sinβ] T (1.11)

[0087] Define the unit vector u = [-sinα cosα 0] T The fundamental reference vector is defined as u in the horizontal principal direction, while HDD is defined as the reference vector u rotated around VDD. Furthermore, the principal plane π can be defined:

[0088] h(α,β,θ)=[[a1 b1]t[a2 b2]t[a3 b3]t] T

[0089] π(α,β,θ):[xyz]·h(α,β,θ)=0 (1.12)

[0090] For the set of consistent line features in HDD, the normal vector of the projection surface corresponding to the line features in this set is unique on the Gaussian sphere. Assuming the projection point on this Gaussian sphere is pi, and considering the error of HDD and the perceptual error of the line features, the normal vector of the projection surface of the Gaussian sphere containing the line features of this set is distributed at p. i The neighborhood, and the corresponding HDD's numerical distribution range is Furthermore, considering the geometric constraint of 90 degrees on the remaining coordinate axes of the horizontal principal direction, a consistent set of line features is obtained through cluster voting statistics, thus yielding the optimal principal direction h(αβθ). optimal ).

[0091] Step S108: Based on the inertial navigation equation and the constraint relationship between the vertical principal direction and the horizontal principal direction of the current scene on the camera attitude, the camera attitude of the current vehicle is self-calibrated.

[0092] In step S106, the camera pose of the current vehicle is optimized based on the structural line constraints of the single-camera pose and the multi-camera pose constraints to achieve self-calibration.

[0093] For example, for single-camera pose optimization under structural line constraints, firstly, a pose cost function is established based on the distance from the line features sensed in the current frame to the corresponding vanishing point, and the single-camera pose of the current vehicle is optimized. Then, a cost function is established based on the correlation between the vertical principal direction and the gravity direction, as well as the relative pose estimation of the vehicle body between adjacent time steps, and the multi-camera pose of the current vehicle is optimized.

[0094] In this process, after obtaining the approximate results of the main direction of the Manhattan frame and the model data consistent with those results, an attitude cost function for the distance from the line element perceived in the current frame to the corresponding vanishing point is established, thereby obtaining the nonlinear optimization result of the camera's attitude estimation without drift in the Manhattan structure of the underground parking garage.

[0095] The aforementioned steps yield the approximate value θ of the optimal horizontal principal direction. optimal Furthermore, considering the noise in line feature perception and the sampling interval error in voting statistics during Gaussian fan projection, further optimization is performed. Using three rotation parameters—the vertical principal axis component of VDD and the horizontal direction angle of HDD rotating along VDD—as variables, and the set of line features consistent with the Manhattan main structure perceived in the image at that moment as sample data, the following cost function is established:

[0096]

[0097] Here, M k Let d represent the parallel line corresponding to the k-th vanishing point.i,k (θ) represents the image pixel distance from the vanishing point to the corresponding straight line, and the optimized, drift-free absolute pose of the camera is obtained.

[0098] For extrinsic parameter estimation of joint multi-camera attitude constraints, drift-free attitude estimation of continuous frames of key trajectory segments in the surround view system is performed, and the sampling rate is made comparable to the output frequency of the IMU wheel speed meter fused with the odometer. The coordinate system of the fused odometer is established to correspond to the key constraint relationship in the surround view system, including the correlation constraint from the gravity direction to the vertical principal direction of the scene and the correlation constraint of the rotation component in the horizontal principal direction. This yields the nonlinear optimization result of the attitude of the surround view camera relative to the vehicle body. Specifically, the roll angle and pitch angle of the current vehicle are estimated based on the camera with vertical gravity constraints, and the yaw angle of the current vehicle is estimated based on the camera with horizontal relative attitude constraints.

[0099] For example, the vehicle attitude R(θ) is estimated based on the wheel speed enhanced odometer in the aforementioned steps. k The obtained camera pose without drift within the Manhattan main structure. Camera i to vehicle initial attitude relationship Assuming the image frame length within the sliding window is n, the pose of camera i relative to the initial O-XYZ coordinate system at time k is:

[0100]

[0101] Considering that the perpendicular line in the Manhattan main structure assumption of this basement scene is equivalent to the direction of gravity, the two are equivalent in the vertical direction. Corresponding to drift-free attitude The equivalent Euler axis-angle expression is also... correspond The equivalent Euler axis-angle expression leads to the following cost function:

[0102]

[0103] Meanwhile, for the two image frames {k} within this window that have completed attitude estimation under the Manhattan main structure framework, ... i k j The relative attitude between two adjacent moments is calculated from the odometer readings of the vehicle body attitude:

[0104]

[0105] in, For an equivalent roll-pitch-yaw Euler angle representation, the relative poses at two adjacent moments within the Manhattan main structural framework are:

[0106]

[0107] in, To correspond to the equivalent roll-pitch-yaw Euler angle expression, since both are based on a coordinate system with gravity as the Z-axis, we assume that... Therefore, the cost function is established as follows:

[0108]

[0109] Therefore, the overall cost function can be expressed as:

[0110]

[0111] The embodiments of this invention reduce the time-consuming and labor-intensive manpower and equipment investment in traditional calibration work, while avoiding the shortcomings of motion trajectory hand-eye calibration methods that cannot fully meet the requirements of high-precision trajectory and complex motion patterns in underground parking environments. It fully considers the geometric constraints of Manhattan scenes in underground parking environments, and limits the camera's extrinsic parameter (position + attitude) space to the key camera attitude space for the characteristics of camera parameter changes in actual application scenarios. At the same time, it avoids the errors introduced by scene point 3D reconstruction, and improves the effectiveness and accuracy of parameter estimation.

[0112] Specifically, an enhanced odometry system with wheel speed scale correction was constructed by using IMU-assisted wheel speed to improve the trajectory accuracy of the vehicle itself. Simultaneously, by tracking the gravity direction in the inertial device, it was quickly associated with the vertical line perceived by the Manhattan geometry scene in the underground parking garage. Furthermore, based on this prior constraint, the horizontal principal direction of the scene was quickly associated. This avoids the shortcomings of traditional methods for estimating the principal direction of the Manhattan geometry framework, such as strict geometric constraints on the scene, high computational resource consumption for RANSAC search traversal and mean-shift clustering, and difficulty in real-time solution. This ensures the reliability and real-time performance of camera attitude estimation in the underground parking garage scene. In other words, this embodiment of the invention can update the camera parameter settings corresponding to the vehicle in real time according to the real-time changes in vehicle driving parameters, achieving dynamic monitoring of the camera calibration accuracy in the underground parking garage environment.

[0113] The embodiments of the present invention also support online calibration of vehicle body parameters, which can monitor the status parameters of the vehicle body multi-sensor system in a timely and dynamic manner, avoid the impact of parameter changes on the quality of results during long-term operational use, and improve the stability and safety of the vehicle body sensor perception system.

[0114] like Figure 3 As shown, this embodiment of the invention also provides a vehicle surround view camera self-calibration device 200 suitable for underground parking environments, the device comprising:

[0115] The determination module 201 uses the current vehicle's wheel speed to constrain the inertial navigation equations under the ESKF filtering model framework to determine the current vehicle's navigation state value; wherein, the navigation state value is based on prior information about the current vehicle's gravity direction.

[0116] The first estimation module 202 associates the prior information of the gravity direction of the current vehicle with the vertical line elements in the camera structure line elements to determine the vertical principal direction of the current scene; wherein, the camera structure line elements are features extracted from the current frame image captured by the current vehicle.

[0117] The second estimation module 203 clusters the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction to determine the horizontal principal direction of the current scene.

[0118] The optimization module 204 performs self-calibration of the camera attitude of the current vehicle based on the inertial navigation equation and the constraint relationship between the vertical principal direction and the horizontal principal direction of the current scene on the camera attitude.

[0119] In some embodiments, the determining module 201 is further configured to: add a proportional coefficient corresponding to the wheel speed to the inertial navigation equation under the ESKF filtering model framework corresponding to the current vehicle; determine the error state measurement equation of the current vehicle based on the proportional coefficient of the wheel speed as a constraint; and determine the navigation state value of the current vehicle based on the error state measurement equation.

[0120] In some embodiments, the first estimation module 202 is further specifically used to: extract line features in the current scene from the acquired image of the current vehicle; determine the vertical line features constrained by the vertical principal direction in the current scene based on the angle between the direction of the current vehicle and the normal vector of the line feature on the camera Gaussian ellipsoid projection plane at a preset time; and construct constraint equations based on the vertical line features to obtain the optimized gravity component of the current scene in the vertical principal direction.

[0121] In some embodiments, the second estimation module 203 is further specifically used to: parameterize the vertical principal direction and the horizontal principal direction to determine the parametric equation of the horizontal principal direction; determine the numerical distribution range of the horizontal principal direction based on the normal vector corresponding to the projection of the line features of the camera structure line elements in the horizontal principal direction onto the Gaussian sphere; and cluster the approximate value of the current scene in the horizontal principal direction based on the geometric constraint relationship between the horizontal principal direction and the coordinate axes.

[0122] In some embodiments, the optimization module 204 is further specifically used to optimize the single-camera pose of the current vehicle by establishing a pose cost function for the distance from the line features perceived in the current frame to the corresponding vanishing point based on the vertical main direction and the horizontal main direction of the current scene.

[0123] In some embodiments, the optimization module 204 is further specifically used to optimize the multi-camera attitude of the current vehicle by establishing a cost function based on the correlation between the vertical principal direction and the gravity direction and the relative attitude estimation of the vehicle body at two adjacent time points.

[0124] In some embodiments, before the step of determining module 201 in determining the navigation state value of the current vehicle by constraining the inertial navigation equation under the ESKF filtering model framework through the wheel speed of the current vehicle, the vehicle state of the current vehicle is specifically initialized statically.

[0125] Figure 4 This is a schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. See also... Figure 4 As shown, the electronic device 300 includes a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304, and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303, and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the vehicle surround view camera self-calibration method for underground parking environments described in the above embodiments by reading and executing machine-executable instructions for vehicle surround view camera self-calibration in underground parking environments from the machine-readable storage medium 301.

[0126] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0127] Non-volatile media can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or combinations thereof.

[0128] It is understood that the specific operation methods of each functional module in this embodiment can be referred to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0129] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program. When the computer program code is executed, it can implement the self-calibration method of vehicle surround view camera applicable to underground parking environment as described in any of the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0131] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0132] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0133] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A self-calibration method for vehicle surround-view cameras suitable for underground parking environments, characterized in that, The method includes: The inertial navigation equations under the ESKF filtering model framework are constrained by the current vehicle's wheel speed to determine the current vehicle's navigation state value; wherein, the navigation state value is based on prior information about the current vehicle's gravity direction. The prior information of the current vehicle's gravity direction is associated with the vertical line elements in the camera structure line elements to determine the vertical principal direction of the current scene; wherein, the camera structure line elements are features extracted from the current frame image captured by the current vehicle. Cluster the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction to determine the horizontal principal direction of the current scene; Based on the vertical and horizontal principal directions of the current scene, an attitude cost function is established to calculate the distance from the line features perceived in the current frame to the corresponding vanishing point, thereby optimizing the single-camera attitude of the current vehicle. Based on the correlation between the vertical principal direction and the gravity direction, as well as the relative attitude estimation of the vehicle body at two adjacent time points, a cost function is established to optimize the multi-camera attitude of the current vehicle.

2. The method according to claim 1, characterized in that, The step of determining the navigation state value of the current vehicle by constraining the inertial navigation equations under the ESKF filtering model framework using the current vehicle wheel speed includes: Add a proportional coefficient corresponding to the wheel speed to the inertial navigation equation under the ESKF filtering model framework corresponding to the current vehicle; The error state measurement equation for the current vehicle is determined based on the wheel speed proportionality coefficient as a constraint condition. Based on the error state measurement equation, the current navigation state value of the vehicle is determined.

3. The method according to claim 1, characterized in that, The step of determining the vertical principal direction of the current scene by associating the prior information of the current vehicle's gravity direction with the vertical line elements in the camera's structural line elements includes: Extract line features from the captured image of the current vehicle in the current scene; Based on the angle between the current vehicle's gravity direction and the normal vector of the line element on the camera's Gaussian ellipsoid projection plane at a preset time, the vertical line element constrained by the vertical principal direction in the current scene is determined. Based on the vertical line elements, constraint equations are constructed to obtain the optimized gravity component of the current scene in the vertical principal direction.

4. The method according to claim 1, characterized in that, The step of clustering the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction to determine the horizontal principal direction of the current scene includes: The vertical principal direction and the horizontal principal direction are parameterized to determine the parametric equation of the horizontal principal direction; The numerical distribution range of the horizontal principal direction is determined based on the normal vector corresponding to the projection of the line features of the camera structure line elements in the horizontal principal direction onto the Gaussian sphere. Based on the geometric constraints between the horizontal principal direction and the coordinate axes, the approximate value of the current scene in the horizontal principal direction is clustered.

5. The method according to claim 1, characterized in that, Before the step of determining the navigation state value of the current vehicle by constraining the inertial navigation equations under the ESKF filtering model framework using the current vehicle's wheel speed, the method further includes: The current vehicle status is statically initialized.

6. A self-calibration device for a vehicle surround-view camera suitable for underground parking environments, characterized in that, The device includes: The determination module uses the current vehicle's wheel speed to constrain the inertial navigation equations under the ESKF filtering model framework, thereby determining the current vehicle's navigation state value; wherein, the navigation state value is based on prior information about the current vehicle's gravity direction; The first estimation module associates the prior information of the current vehicle's gravity direction with the vertical line elements in the camera structure line elements to determine the vertical principal direction of the current scene; wherein, the camera structure line elements are features extracted from the current frame image captured by the current vehicle. The second estimation module clusters the vertical principal direction of the current scene and the projection of the camera structure line elements onto the horizontal principal direction to determine the horizontal principal direction of the current scene. The optimization module establishes a pose cost function based on the vertical and horizontal principal directions of the current scene, which is the distance from the line features perceived in the current frame to the corresponding vanishing point, and optimizes the single-camera pose of the current vehicle. Based on the correlation between the vertical principal direction and the gravity direction and the relative pose estimation of the vehicle body at two adjacent time moments, a cost function is established to optimize the multi-camera pose of the current vehicle.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 5.

Images