A method for calibrating fish-eye lens external parameters

Abstract

The application relates to the field of machine vision, and proposes a fish-eye lens external parameter calibration method for calibrating the external parameters of multiple fish-eye lenses installed on a vehicle body, wherein the multiple fish-eye lenses are arranged at the front, rear, left and right parts of the vehicle body, and the method comprises the following steps: determining the conversion relationship among multiple fish-eye lens coordinate systems according to images of multiple checkerboard calibration boards captured by the multiple fish-eye lenses; the multiple checkerboard calibration boards are placed horizontally on the ground in the left front, left rear, right front and right rear directions of the vehicle body; determining the external parameters of each fish-eye lens in the multiple fish-eye lenses according to the conversion relationship among the multiple fish-eye lens coordinate systems and the images of first and second wheel targets in the multiple fish-eye lenses; the first and second wheel targets are placed horizontally on the left and right sides of the rear wheels of the vehicle body respectively.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision, and in particular to a method for calibrating external parameters of a fisheye lens. BACKGROUND

[0002] When an intelligent vehicle uses assisted driving or automatic driving, it needs to observe the surrounding environment through vehicle-mounted perception sensors such as cameras, ultrasonic waves, fisheye lenses, etc., so as to ensure that the subsequent assisted driving function can be accurately completed. Poor perception results will lead to inaccurate assessment of the surrounding environment by the intelligent vehicle, and in severe cases, may even cause accidents. Therefore, it is necessary to reasonably calibrate the vehicle-mounted perception sensors to obtain accurate perception results of the surrounding environment.

[0003] Calibration refers to the process of using standard instruments to calibrate sensors. After a period of use, the sensor needs to be retested to detect whether its basic performance has changed and whether it can continue to be used. If some indicators of the sensor have changed, it should be corrected or recalibrated.

[0004] The vehicle-mounted fisheye lens is a wide-angle lens installed on the vehicle body. When each vehicle is shipped, the external parameters of the fisheye lens need to be individually calibrated, and the accuracy and precision of the calibration results directly affect the subsequent intelligent assisted driving or automatic driving effect. How to calibrate the external parameters of the fisheye lens to meet the needs of convenience and independence from complex environments is a problem that needs to be solved. SUMMARY

[0005] To solve the above problems, an embodiment of the present application provides a method for calibrating external parameters of a fisheye lens, which is used to calibrate the external parameters of a plurality of fisheye lenses installed on a vehicle body, the plurality of fisheye lenses are respectively arranged at the front, rear, left and right parts of the vehicle body, and the method comprises: determining the conversion relationship between a plurality of fisheye lens coordinate systems according to images of a plurality of checkerboard calibration boards captured by the plurality of fisheye lenses; the plurality of checkerboard calibration boards are placed on the ground at the left front, left rear, right front and right rear corner points of the vehicle body; determining the external parameters of each fisheye lens in the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and images of first and second wheel targets in the plurality of fisheye lenses; the first and second wheel targets are respectively placed close to the left and right rear wheels of the vehicle body. In this way, only the calibration board and the wheel target are needed to calibrate the external parameters, and the method of fixing the vehicle position by fixing the vehicle position with the wheel fixer in the common method is abandoned to solve the problem of external parameter calibration when the fisheye lens deviates from the initial position due to installation problems. The calibration space is easy to obtain, which can meet the requirement of periodic calibration of the fisheye lens of the intelligent vehicle in subsequent use, is conducive to the popularization of calibration, and can reduce the cost, and meet the requirement of quick calibration of the external parameters of the fisheye lens of the vehicle anytime and anywhere.

[0006] In an implementation, the first fisheye lens of the plurality of fisheye lenses is arranged at one of front, rear, left and right positions of the vehicle body, and the second fisheye lens is arranged at one of the front, rear, left and right positions of the vehicle body, adjacent to the first fisheye lens. The method for determining the conversion relationship between the plurality of fisheye lens coordinate systems according to the images of the plurality of checkerboard calibration boards captured by the plurality of fisheye lenses comprises: obtaining a first image captured by the first fisheye lens, determining a first fisheye lens coordinate system according to a first checkerboard calibration board on the first image; the first checkerboard calibration board is one of the plurality of checkerboard calibration boards; obtaining a second image captured by the second fisheye lens, determining a second fisheye lens coordinate system according to the first checkerboard calibration board on the second image; and determining a first conversion relationship according to the coordinates of a plurality of pairs of same position points in the first checkerboard calibration board on the first image and the first checkerboard calibration board on the second image, the first conversion relationship being the conversion relationship between the first fisheye lens coordinate system and the second fisheye lens coordinate system. In this way, the conversion relationship between two adjacent fisheye lens coordinate systems is determined by using the same checkerboard calibration board, double target calibration is achieved, the calibration space is simple and convenient to set, and the calibration cost is low.

[0007] In an implementation, the method for determining the conversion relationship between the plurality of fisheye lens coordinate systems according to the images of the plurality of checkerboard calibration boards captured by the plurality of fisheye lenses comprises: calculating a rotation matrix and a translation transformation matrix between the first fisheye lens coordinate system and the second fisheye lens coordinate system according to the coordinates of a plurality of pairs of same position points in the first checkerboard calibration board on the first image and the first checkerboard calibration board on the second image, and determining the first conversion relationship. In this way, double target calibration is achieved, and the positional relationship between the first fisheye lens coordinate system and the second fisheye lens coordinate system is obtained.

[0008] In one implementation, a third fisheye lens of the plurality of fisheye lenses is disposed at one of front, rear, left and right positions of the vehicle body, adjacent to the first fisheye lens and opposite to the second fisheye lens. The method for determining the conversion relationship between the plurality of fisheye lens coordinate systems comprises: obtaining a third image captured by the first fisheye lens, the third image being an image including a third checkerboard calibration board; the third checkerboard calibration board being one of the plurality of checkerboard calibration boards; obtaining a fourth image captured by the third fisheye lens; determining the third fisheye lens coordinate system according to the third checkerboard calibration board in the fourth image; determining a second conversion relationship according to the coordinates of a plurality of pairs of same position points on the third checkerboard calibration board in the third image and the fourth image, the second conversion relationship being the conversion relationship between the third fisheye lens coordinate system and the first fisheye lens coordinate system; and determining a third conversion relationship according to the second conversion relationship and the first conversion relationship, the third conversion relationship being the conversion relationship between the third fisheye lens coordinate system and the second fisheye lens coordinate system. In this way, the conversion relationship between two fisheye lens coordinate systems with non-overlapping view regions can be determined by a double-target determination method according to the conversion relationship between two adjacent fisheye lens coordinate systems and the coordinates of same position points on the same checkerboard calibration board in the images of the two fisheye lenses.

[0009] In an implementable embodiment, the determining the extrinsic parameters of each of the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and the images of the first and second wheel targets in the plurality of fisheye lenses comprises: determining a fifth conversion relationship according to images of a plurality of checkerboard calibration boards captured by the plurality of fisheye lenses, the fifth conversion relationship being a conversion relationship between each of the plurality of fisheye lens coordinate systems and a corresponding checkerboard calibration board coordinate system; the checkerboard calibration board coordinate system being a coordinate system determined by the checkerboard calibration board; determining a sixth conversion relationship according to the conversion relationship between the plurality of fisheye lens coordinate systems and the images of the first and second wheel targets in the plurality of fisheye lenses, the sixth conversion relationship being a conversion relationship between a vehicle body plane coordinate system and the plurality of checkerboard calibration board coordinate systems; the vehicle body plane coordinate system being a coordinate system with the center points of the left and right rear wheels of the vehicle body as the center, the direction from the second wheel target to the first wheel target as the positive direction of the Y axis, and the front direction of the vehicle as the positive direction of the X axis; determining a seventh conversion relationship according to the fifth conversion relationship and the sixth conversion relationship, the seventh conversion relationship being a conversion relationship between each of the plurality of fisheye lens coordinate systems and the vehicle body plane coordinate system; and determining the extrinsic parameters of each of the plurality of fisheye lenses according to the seventh conversion relationship. In this way, after obtaining the conversion relationship between the fisheye lens coordinate systems, the position of the rear wheel in the fisheye lens coordinate system can be determined using the wheel target, so as to determine the conversion relationship between the vehicle body plane coordinate system and the fisheye lens coordinate system, and obtain the extrinsic parameters of the fisheye lens.

[0010] In an implementable embodiment, the determining the fifth conversion relationship according to the images of the plurality of checkerboard calibration boards captured by the plurality of fisheye lenses comprises: obtaining a fifth image captured by each of the plurality of fisheye lenses, the fifth image being an image including one of the plurality of checkerboard calibration boards; and obtaining a fifth conversion relationship according to the fifth image and the coordinates of a plurality of pairs of same-position points on the corresponding one of the checkerboard calibration boards, the fifth conversion relationship including a scale matrix, a rotation matrix, and a translation transformation matrix between the coordinate system of each of the plurality of fisheye lenses and the corresponding checkerboard calibration board coordinate system. In this way, the conversion relationship between the coordinate system of each of the plurality of fisheye lenses and the corresponding checkerboard calibration board coordinate system can be obtained.

[0011] In an implementable embodiment, the determining the sixth conversion relationship according to the conversion relationship between the plurality of fisheye lens coordinate systems and the images of the first and second wheel targets in the plurality of fisheye lenses comprises: obtaining a sixth image and a seventh image captured by the second and third fisheye lenses in the plurality of fisheye lenses; the sixth image is an image captured by the second fisheye lens and including the first wheel target, and the seventh image is an image captured by the third fisheye lens and including the second wheel target; determining coordinates of a midpoint between the first and second wheel targets in the first or third checkerboard coordinate system according to the conversion relationship between the second and third fisheye lens coordinate systems in the conversion relationship between the plurality of fisheye lens coordinate systems; and determining the sixth conversion relationship according to the coordinates of the midpoint between the first and second wheel targets in the first or third checkerboard coordinate system, the coordinates of the first and second wheel targets in the second or third checkerboard coordinate system, and the direction in which the second wheel target points to the first wheel target. In this way, the conversion relationship between the vehicle body plane coordinate system and the plurality of checkerboard coordinate systems can be obtained.

[0012] In an implementable embodiment, the determining the seventh conversion relationship according to the fifth conversion relationship and the sixth conversion relationship comprises: determining the seventh conversion relationship between the second fisheye lens coordinate system and the vehicle body plane coordinate system according to the fifth conversion relationship between the second fisheye lens coordinate system and the corresponding first checkerboard coordinate system and the sixth conversion relationship between the first checkerboard coordinate system and the vehicle body plane coordinate system. In this way, the conversion relationship between the vehicle body plane coordinate system and the second fisheye lens coordinate system can be obtained.

[0013] In an implementable embodiment, the determining the seventh conversion relationship according to the fifth conversion relationship and the sixth conversion relationship comprises: determining the seventh conversion relationship between the third fisheye lens coordinate system and the vehicle body plane coordinate system according to the fifth conversion relationship between the third fisheye lens coordinate system and the corresponding third checkerboard coordinate system and the sixth conversion relationship between the third checkerboard coordinate system and the vehicle body plane coordinate system. In this way, the conversion relationship between the vehicle body plane coordinate system and the third fisheye lens coordinate system can be obtained.

[0014] In an implementable embodiment, the determining the seventh conversion relationship according to the fifth conversion relationship and the sixth conversion relationship comprises: determining the seventh conversion relationship between the first fisheye lens coordinate system and the vehicle body plane coordinate system according to the conversion relationship between the first fisheye lens coordinate system and the third fisheye lens coordinate system, the fifth conversion relationship between the third fisheye lens coordinate system and the corresponding third chessboard calibration plate coordinate system, and the sixth conversion relationship between the third chessboard calibration plate coordinate system and the vehicle body plane coordinate system. In this way, the conversion relationship between the vehicle body plane coordinate system and the first fisheye lens coordinate system can be obtained.

[0015] In an implementable embodiment, the determining the extrinsic parameters of each fisheye lens in the plurality of fisheye lenses according to the seventh conversion relationship comprises: the seventh conversion relationship comprises a rotation matrix and a translation transformation matrix between each fisheye lens coordinate system and the vehicle body plane coordinate system; and the extrinsic parameters of the first fisheye lens coordinate system, the second fisheye lens coordinate system, the third fisheye lens coordinate system and the fourth fisheye lens coordinate system are determined according to the rotation matrix and the translation transformation matrix between the first fisheye lens coordinate system, the second fisheye lens coordinate system, the third fisheye lens coordinate system and the fourth fisheye lens coordinate system and the vehicle body plane coordinate system. In this way, the preliminary extrinsic parameters of each fisheye lens coordinate system can be calibrated.

[0016] In an implementable embodiment, an inertial measurement unit is arranged on the vehicle body chassis, and the inertial measurement unit is used to determine values of a roll angle, a pitch angle and a heading angle of the vehicle body; the method further comprises: obtaining the values of the roll angle, the pitch angle and the heading angle output by the inertial measurement unit; determining an IMU coordinate system according to the values of the roll angle, the pitch angle and the heading angle; the values of the roll angle, the pitch angle and the heading angle indicate the deflection angles of three axes of the IMU coordinate system and the Y axis, the X axis and the Z axis of the vehicle body plane coordinate system; determining the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system; and adjusting the extrinsic parameters of the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system. In this way, the vehicle body pose at the time of calibration obtained through IMU information processing can be combined with the preliminary fisheye lens extrinsic parameters, so that the influence of the attitude of the vehicle body at the time of calibration on the fisheye lens extrinsic parameters can be corrected, the relative shaking and tilting of the vehicle body to the ground can be corrected, and the fisheye lens extrinsic parameters at the standard pose can be obtained.

[0017] In an implementable embodiment, the determining the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system comprises: obtaining a rotation matrix between the IMU coordinate system and the vehicle body plane coordinate system according to the values of the pitch angle, the roll angle and the yaw angle; and determining the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system according to the rotation matrix between the IMU coordinate system and the vehicle body plane coordinate system and the conversion relationship between the vehicle body plane coordinate system and the fisheye lens coordinate system. In this way, the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system can be obtained.

[0018] In an implementable embodiment, the adjusting the extrinsic parameters of the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system comprises: dynamically adjusting the extrinsic parameters of the plurality of fisheye lenses according to IMU information collected by the inertial measurement unit in real time during the driving of the vehicle, the IMU information comprising the values of the roll angle, the pitch angle and the yaw angle of the vehicle body. In this way, the influence of the attitude between the vehicle body and the ground on the shooting angle of the fisheye lens can be corrected in real time, and even if the vehicle body is shaking, the perception image obtained according to the lens parameters changing in real time will not change.

[0019] The fisheye lens extrinsic parameter calibration method provided by the embodiments of the present application calibrates the extrinsic parameters of the vehicle-mounted fisheye lens based on the combination of vision and IMU, can meet the demand of the actual production line of the vehicle factory for the fast calibration of the extrinsic parameters of the fisheye lens, and can complete the calibration of the extrinsic parameters of the fisheye lens of a new vehicle within tens of seconds. While ensuring high efficiency in calibration, the fisheye lens extrinsic parameter calibration method provided by the embodiments of the present application can exclude the influence of the vehicle condition and ensure high precision of the calibration of the extrinsic parameters of the fisheye lens in real time. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in the specification, the drawings required in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only a part of the embodiments disclosed in the specification, and other drawings can also be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0021] The drawings required in the embodiment or prior art description will be briefly introduced as follows.

[0022] Figure 1 The calibration space scene diagram of the fisheye lens extrinsic parameter calibration method provided by the embodiments of the present application;

[0023] Figure 2 The principle diagram of the fisheye lens extrinsic parameter calibration method provided by the embodiments of the present application;

[0024] Figure 3 The principle diagram for adjusting the fish-eye lens extrinsic parameter by using the IMU sensor in the fish-eye lens extrinsic parameter calibration method provided by the embodiment of the present application;

[0025] Figure 4 The IMU information and vehicle body posture angle information schematic diagram in the fish-eye lens extrinsic parameter calibration method provided by the embodiment of the present application;

[0026] Figure 5 The system architecture diagram of the fish-eye lens extrinsic parameter calibration method provided by the embodiment of the present application;

[0027] Figure 6 The flow chart of the fish-eye lens extrinsic parameter calibration method provided by the embodiment of the present application;

[0028] Figure 7 The flow chart of the fish-eye lens extrinsic parameter calibration method provided by the embodiment 1 of the present application;

[0029] Figure 8 The calibration scene schematic diagram of the fish-eye lens extrinsic parameter calibration method provided by the embodiment 1 of the present application;

[0030] Figure 9 The flow chart of the fish-eye lens extrinsic parameter calibration method provided by the embodiment 2 of the present application. DETAILED DESCRIPTION

[0031] In the description of the embodiments of the present application, the words "exemplary", "for example", or "for instance" are used to mean serving as an example, instance or illustration. Any embodiment or design solution described as "exemplary", "for example" or "for instance" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. In fact, the words "exemplary", "for example" or "for instance" are used to present related concepts in a specific manner.

[0032] In the description of the embodiments of the present application, the term "and / or" is merely used to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the three cases of existence of A alone, existence of B alone, and existence of A and B simultaneously. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems mean two or more systems, and multiple terminals mean two or more terminals.

[0033] In addition, the terms "first", "second", "third", etc., and the terms "module A", "module B", "module C", etc., are used only for the purpose of description, and are not intended to indicate or imply relative importance or imply that the indicated technical features are required. Therefore, the features defined as "first", "second", etc., can explicitly or implicitly include one or more of the features. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0034] In the description of the embodiments of the present application, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0035] In the description of the embodiments of the present application, the terms "first", "second", "third", etc., and the terms "module A", "module B", "module C", etc., are used only for the purpose of distinguishing similar objects, and do not represent a specific order. It can be understood that the specific order or sequence can be interchanged as permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In the description of the embodiments of the present application, the steps represented by the labels such as S110, S120, etc., do not necessarily mean that the steps are executed in this order, and the order of the steps can be interchanged or executed simultaneously as permitted.

[0037] In the description of the embodiments of the present application, the inertial measurement unit (IMU) is a device for measuring three-axis attitude angles or angular rates and accelerations of a vehicle body, and the three-axis attitude angles include roll angle, pitch angle and heading angle.

[0038] The chessboard calibration plate is provided with X-axis, Y-axis and origin, and the X-axis and Y-axis are perpendicular with the intersection being the chessboard origin. M grid lines are arranged along the X-axis at a unit length interval, and N grid lines are arranged along the Y-axis at a unit length interval. The chessboard calibration plate is divided into M*N unit cells, and M and N are natural numbers. The unit length can be defined as 0.1 meters, 0.2 meters, 0.5 meters or 1 meter, without specific limitation.

[0039] The chessboard calibration plate coordinate system is a coordinate system determined by the X-axis, Y-axis, origin and unit length on the chessboard calibration plate.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0041] The technical solutions related to the embodiments of the present application will be described below.

[0042] The first solution provides a method for aligning and registering a vehicle-mounted camera, including a radar system and an optical sensor, and the specific method is as follows: a calibration space is set, a vehicle is fixed by using a wheel clamp and is placed at a determined position, markers are placed around the vehicle to calibrate the vehicle lens, a camera is set in the calibration space, a vehicle position and a structured light projection on a vehicle wheel are shot, a relationship between the vehicle and the space calibration markers is determined, a determination of the vehicle and the ground position is completed, and finally calibration parameters of the vehicle-mounted camera are obtained. This solution has special equipment requirements for the calibration space, has high cost, needs to place a fixed parking clamp, a camera, and a structured light projection device in the calibration space, and is not easy for subsequent periodic calibration of the vehicle.

[0043] The second solution provides a calibration facility for calibrating a vehicle-mounted laser and an optical camera of an intelligent vehicle by using calibration markers. The specific method is as follows: the vehicle is parked on a vehicle-mounted platform at a fixed position, and the vehicle is moved to a calibration range as required; the calibration markers are set in a detectable range of the vehicle, the position and distance of the calibration markers relative to the vehicle are determined by adjusting the vehicle-mounted platform, and calibration is performed. Meanwhile, the solution also proposes a remote calibration method, which places calibration markers on an unmanned aerial vehicle and other ways to place calibration markers in a detectable range of the vehicle, and then calibrates after the relationship between the calibration markers and the vehicle is determined. This solution needs to use a fixed vehicle-mounted platform to move the vehicle to the actual calibration position, the calibration space is limited, and there is an error in the calibration position determined after the vehicle is moved to the fixed position each time.

[0044] The first solution and the second solution set a complex calibration space to obtain the relative relationship between the vehicle and the ground, which is not conducive to popularization and empowerment, and increases the calibration cost. Moreover, the vehicle body pose angle affects the calibration effect of the external parameters of the fisheye lens during calibration, the stability is not enough, and positioning errors are easily caused.

[0045] In view of the problems in the above solutions, the embodiments of the present application provide a calibration method for external parameters of a vehicle-mounted fisheye lens, which jointly calibrates based on vision of the fisheye lens and IMU sensor perception to determine the external parameters of the vehicle-mounted fisheye lens.

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0047] First, the concept of the calibration method for external parameters of the fisheye lens provided by the embodiments of the present application is introduced.

[0048] In order to illustrate the position, speed, direction, etc. of the object observed by the vehicle, a reference frame must be selected. In the reference frame, a set of ordered data selected by a specified method for determining the position of a point in space is called "coordinates".

[0049] When the object moves in a plane, in order to quantitatively describe the position change of the object, a plane rectangular coordinate system can be constructed by two number axes perpendicular to each other and having a common origin in the same plane. The two number axes are placed in the horizontal position and the vertical position, respectively, and the origin, the positive direction and the unit length are specified in the plane rectangular coordinate system. For example, the direction to the left and the direction forward are respectively the positive directions of the two number axes. The number axis perpendicular to the x-axis is called the y-axis or the vertical axis, and the common origin O of the x-axis and the y-axis is called the origin of the plane rectangular coordinate system. The unit length can be defined as 0.1 meters, 0.2 meters, 0.5 meters or 1 meter, without specific limitation. The normal direction of the plane in which the x-y is defined as the z-axis, and the upward direction is positive.

[0050] In practice, the vehicle body plane coordinate system is usually defined as a coordinate system with the center points of the two rear wheels as the coordinate center O V , the direction from the right wheel to the left wheel as the positive direction of the Y-axis, and the forward direction of the vehicle as the positive direction of the X-axis, and the normal direction of the vehicle body plane as the Z-axis.

[0051] The fisheye lens coordinate system is used to determine the coordinates of the object photographed by the fisheye lens in the image, with the unit of pixels.

[0052] The vehicle can convert the coordinates of the object in the vehicle body plane coordinate system p V into the fisheye coordinates p cam of the object in the fisheye lens coordinate system through the internal and external parameters of the fisheye lens, and the conversion relationship is as follows:

[0053] p cam =(KD)(RT)p V (1)

[0054] In formula (1), K is the internal parameter of the camera, and D is the distortion parameter of the camera. The values of K and D are related to the properties of the camera itself, K and D are the camera internal parameters, representing the optical characteristics of the camera, and are fixed by default after the camera is manufactured and are known parameters, which can not be considered in this application; wherein R represents a rotation matrix, and T represents a translation transformation matrix, R and T are the results required for calibration in this application.

[0055] Let the extrinsic parameter of the vehicle-mounted fisheye lens be E=RT, E is used to indicate the rotation relationship and translation transformation relationship between the fisheye lens coordinate system and the vehicle body plane coordinate system, which is determined by the rotation matrix R and the translation transformation matrix T. The translation transformation matrix T indicates the position transformation relationship between the fisheye lens coordinate system and the vehicle body plane coordinate system, and the rotation matrix R indicates the rotation relationship between the fisheye lens coordinate system and the vehicle body plane coordinate system.

[0056] In order to observe the surrounding environment 360° without dead angle, four fisheye lenses are usually arranged at the front, rear, left and right parts of the vehicle body. The extrinsic parameter E of the fisheye lens needs to be calibrated for each vehicle when it is manufactured, so as to determine the position and shooting angle of the fisheye lens relative to the center point of the vehicle body, so that the vehicle can accurately perceive the surrounding environment.

[0057] According to the actual flow line production capacity requirement of the vehicle factory, the calibration of the extrinsic parameter E of the fisheye lens needs to be fast and accurate, and the extrinsic parameter calibration of the fisheye lens of the new vehicle often needs to be completed within 20 seconds. While ensuring high efficiency in completing the calibration, the accuracy of the calibration also needs to be ensured, and the accuracy directly affects the effect of the subsequent intelligent driving module.

[0058] In addition, during the use of the vehicle, the fisheye lens may deviate from the initial position due to installation problems. Therefore, the intelligent vehicle needs to be periodically calibrated during subsequent use. The calibration method of the fisheye lens needs to meet the requirements of convenience and independence from complex environment.

[0059] Based on the above situation, in order to facilitate the popularization of calibration and reduce costs, a simple and convenient calibration space needs to be set up to meet the demand of quickly calibrating the extrinsic parameter E of the fisheye lens of the vehicle anytime and anywhere.

[0060] Figure 1 The calibration space scene diagram of the fisheye lens extrinsic parameter calibration method provided by the embodiments of the application. As shown in Figure 1 In the fisheye lens extrinsic parameter calibration method provided by the embodiments of the application, a simple calibration space can be an open and horizontal ground area. The vehicle can be parked in an open and horizontal ground area during calibration.

[0061] The fisheye lens extrinsic parameter calibration method provided in this application uses checkerboard calibration plates and wheel targets as reference objects. Four identical checkerboard calibration plates U1, U2, U3, and U4 are set at the four corner points of the vehicle body (left rear, left front, right rear, and right front) as standard reference objects. Wheel targets are also set at the two rear wheels, with wheel target positions P1 and P2 representing the positions of the two rear wheels. The four fisheye lenses to be calibrated are cam1, cam2, cam3, and cam4, which can be designated as the first, second, third, and fourth fisheye lenses. The four fisheye lenses are set at the rear, left, right, and front positions of the vehicle body, which can be designated as the first, second, third, and fourth positions. The four identical checkerboard calibration plates U1, U2, U3, and U4 are designated as the first, second, third, and fourth checkerboard calibration plates, respectively.

[0062] Four identical checkerboard calibration plates U1, U2, U3, and U4 are horizontally placed at the four corner points of the vehicle body: left rear, left front, right rear, and right front. At each of these four corner points, the image of each checkerboard calibration plate can be simultaneously captured by two adjacent fisheye lenses from four different lenses. For example, as shown... Figure 1 As shown, the checkerboard calibration plate U1 is located at the left rear of the vehicle body, and fisheye lenses cam1 and cam2 can simultaneously capture images of the checkerboard calibration plate U1; the checkerboard calibration plate U2 is located at the left front of the vehicle body, and fisheye lenses cam2 and cam4 can simultaneously capture images of the checkerboard calibration plate U2; the checkerboard calibration plate U3 is located at the right rear of the vehicle body, and fisheye lenses cam1 and cam3 can simultaneously capture images of the checkerboard calibration plate U3; the checkerboard calibration plate U4 is located at the right front of the vehicle body, and fisheye lenses cam3 and cam4 can simultaneously capture images of the checkerboard calibration plate U4.

[0063] Two wheel targets are placed horizontally and symmetrically at positions P1 and P2 of the two rear wheels on the vehicle body. For example, the target points on the wheel targets can be set to the size of the wheels, so that the projections of the center points of the two target points and the center points of the two rear wheels on the ground coincide as much as possible.

[0064] In one feasible implementation, wheel clamps can be used instead of two wheel targets, with the center points of the two wheel clamps coinciding with the projections of the center points of the two rear wheels onto the ground.

[0065] Since the transformation relationship between the fisheye lens coordinate system and the vehicle body plane coordinate system cannot be directly measured, the next step is to use the coordinates of multiple reference objects in different fisheye lens images to perform coordinate transformation calculations to obtain the rotation matrix R and translation transformation matrix T between the fisheye lens coordinate system and the vehicle body plane coordinate system, thereby determining the transformation relationship between the fisheye lens coordinate system and the vehicle body plane coordinate system.

[0066] Figure 2This is a schematic diagram illustrating the principle of the fisheye lens extrinsic parameter calibration method provided in this application embodiment. The fisheye lens extrinsic parameter calibration method provided in this application obtains the fisheye coordinates of the origin position of the checkerboard calibration board on the image by using four fisheye lenses (cam1, cam2, cam3, and cam4) on the vehicle body to detect the checkerboard calibration board and the wheel target, respectively.

[0067] Checkerboard calibration plate detection refers to the process of using a fisheye lens to photograph a checkerboard calibration plate on the ground, obtaining the fisheye coordinates of multiple points on the calibration plate in the image. The vehicle determines the relative position and shooting angle between two adjacent fisheye lenses based on the coordinates of multiple identical points on the same checkerboard calibration plate in the images from both adjacent fisheye lenses; this process is called dual-target calibration.

[0068] The principle of dual-target calibration is to calculate the transformation relationship between the coordinate systems of the two fisheye lenses based on the pixel coordinates of multiple identical points on the same checkerboard calibration board in the images of the two fisheye lenses.

[0069] In one feasible implementation, two fisheye lenses capture images of the same checkerboard calibration board to obtain two images. The transformation relationship R and T between the coordinate systems of the two fisheye lenses is calculated based on the different pixel coordinates of multiple pairs of points at the same position in the two images, where R represents the rotation matrix of the two coordinate systems and T represents the translation transformation matrix.

[0070] like Figure 2 As shown, fisheye lenses cam1 and cam2 simultaneously capture images of the checkerboard calibration board U1. The pixel coordinates of a point on the checkerboard calibration board U1 in image 1 are obtained from image 1 created by fisheye lens cam1. cam1 (x 1,U1 ,y 1,U1 The pixel coordinates of the same position point of the checkerboard calibration board U1 obtained from image 2 of the fisheye lens cam2 are p. cam2 (x 2,U1 ,y 2,U1 Without considering the intrinsic parameters of cam1 and cam2 being the same, since the pixel coordinates of the same position point on U1 in images 1 and 2 have a rotational and translational relationship with the shooting angle, the pixel coordinates of a position point on U1 in image 1 and the pixel coordinates in image 2 are converted to:

[0071] p cam2 (x 2,U1 ,y 2,U1 ) = R cam1→cam2 (U1)p cam1 (x 1,U1 ,y 1,U1 )+T cam1→cam2(ΔX1, ΔY1) (2)

[0072] (ΔX1, ΔY1) = (x 2,U1 -x 1,U1 ,y 2,U1 -y 1,U1 ) (3)

[0073] wherein R cam1→cam2 (U1) is the rotation angle between the pixel coordinates of the same position point on U1 in image 1 and image 2, T cam1→cam2 (ΔX1, ΔY1) represents the translation transformation amount between the fisheye lens cam1 and cam2 coordinate systems.

[0074] In an implementable embodiment, the rotation matrix R cam1→cam2 and the translation transformation matrix T cam1→cam2 between the two fisheye lens coordinate systems are obtained according to the different pixel coordinates of multiple pairs of the same position points in image 1 and image 2.

[0075] p cam2 = R cam1→cam2 p cam1 + T cam1→cam2 (4)

[0076] Let E cam1→cam2 = R cam1→cam2 T cam1→cam2 , then the conversion relationship between the fisheye lens cam2 coordinate system and the fisheye lens cam1 coordinate system is:

[0077] p cam2 = E cam1→cam2 p cam1 (5)

[0078] wherein p cam2 is the fisheye lens cam2 coordinate system, and p cam1 is the fisheye lens cam1 coordinate system.

[0079] Similarly, the fisheye lens cam1 and cam3 simultaneously capture the checkerboard calibration plate U3, and the pixel coordinates of a position point on the top of the checkerboard calibration plate U3 in image 3 obtained through the image 3 of the fisheye lens cam1 are p cam1 (x 1,U3 ,y 1,U3 ), and the pixel coordinates of the same position point in the image 4 of the fisheye lens cam3 are p cam3 (x 3,U3 ,y 3,U3 ).

[0080] In one feasible implementation, the rotation matrix R between the two fisheye lens coordinate systems is calculated based on the different pixel coordinates of multiple pairs of points at the same location in images 3 and 4. cam1→cam3 Translation matrix T cam1→cam3 The transformation relationship between the coordinate system of fisheye lens cam3 and the coordinate system of fisheye lens cam1 is as follows:

[0081] p cam3 =R cam1→cam3 p cam1 +T cam1→cam3 (6)

[0082] Where p cam3 This is the cam3 coordinate system for the fisheye lens.

[0083] Let E cam1→cam3 =R cam1→cam3 T cam1→cam3 The transformation relationship between the coordinate system of fisheye lens cam3 and the coordinate system of fisheye lens cam1 is as follows:

[0084] p cam3 =E cam1→cam3 p cam1 (7)

[0085] Similarly, fisheye lenses cam2 and cam4 simultaneously capture images of the checkerboard calibration board U2. The pixel coordinates of a point on the checkerboard calibration board U2 are obtained from image 5 of fisheye lens cam2. cam2 (x 2,U2 ,y 2,U2 The pixel coordinates of the same point on the checkerboard calibration board U2 are obtained from image 6 of the fisheye lens cam4. cam4 (x 4,U2 ,y 4,U2 ).

[0086] In one feasible implementation, the rotation matrix R between the two fisheye lens coordinate systems is calculated based on the different pixel coordinates of multiple pairs of points at the same location in images 5 and 6. cam2→cam4 Translation matrix T cam2→cam4 The transformation relationship between the coordinate systems of fisheye lens cam2 and fisheye lens cam4 is as follows:

[0087] p cam4 =R cam2→cam4 p cam2 +T cam2→cam4 (8)

[0088] Where p cam4 This is the cam4 coordinate system for the fisheye lens.

[0089] Let E cam2→cam4 = R cam2→cam4 T cam2→cam4 Then the conversion relationship between the fisheye lens cam4 coordinate system and the fisheye lens cam2 coordinate system is:

[0090] p cam4 = E cam2→cam4 p cam2 (9)

[0091] It can be understood that the fisheye lens cam1 and cam4 are two fisheye lenses in relative positions, and the view areas photographed are completely non-overlapping. By solving the conversion relationship between the fisheye lens cam1 and cam2, cam2 and cam4 coordinate systems, the simultaneous equations (5) and (9) can be obtained, and the conversion relationship between the fisheye lens cam1 coordinate system and the fisheye lens cam4 coordinate system is

[0092] p cam4 = E cam1→cam2 E cam2→cam4 p cam1

[0093] = E cam1→cam4 p cam1 (10)

[0094] Similarly, the fisheye lens cam2 and cam3 are two fisheye lenses in relative positions, and the view areas photographed are completely non-overlapping. By solving the conversion relationship between the fisheye lens cam1 and cam2, cam1 and cam3 coordinate systems, the simultaneous equations (5) and (7) can be obtained, and the conversion relationship between the fisheye lens cam2 coordinate system and the fisheye lens cam3 coordinate system is:

[0095] p cam3 = E cam1→cam3 E cam1→cam2 p cam2

[0096] = E cam2→cam3 p cam2 (11)

[0097] In actual driving, in order to reduce errors and improve positioning accuracy when observing the surrounding environment 360°, the fisheye lens cam3 and cam4 will also be used to photograph the checkerboard calibration plate U4 at the same time. The pixel coordinates of a position point on the top of the checkerboard calibration plate U4 obtained through the image 7 of the fisheye lens cam3 are p cam3 (x 3,U4 ,y 3,U4 ), and the pixel coordinates of the same position point on the checkerboard calibration plate U4 obtained through the image 8 of the fisheye lens cam4 are p cam4 (x4,U4 ,y 4,U4 In one feasible implementation, the rotation matrix R between the two fisheye lens coordinate systems is calculated based on the different pixel coordinates of multiple pairs of points at the same location in images 7 and 8. cam3→cam4 Translation matrix T cam3→cam4 The transformation relationship between the coordinate system of fisheye lens cam3 and the coordinate system of fisheye lens cam4 is as follows:

[0098] p cam4 =R cam3→cam4 p cam3 +T cam3→cam4 (12)

[0099] Let E cam3→cam4 =R cam3→cam4 T cam3→cam4 The transformation relationship between the coordinate systems of fisheye lens cam4 and fisheye lens cam3 is as follows:

[0100] p cam4 =E cam3→cam4 p cam3 (13)

[0101] In this way, based on the data obtained by the four fisheye lenses detecting the designated position points of the four chessboard calibration plates, the two adjacent fisheye lenses are calibrated together to determine the transformation relationship between the coordinate systems of the two fisheye lenses, and further, the transformation relationship between the coordinate systems of the four fisheye lenses can be obtained.

[0102] Target detection uses wheel targets to determine the coordinates of the rear wheel center in the fisheye lens coordinate system, thereby establishing the transformation relationship between the vehicle body plane coordinate system and the fisheye lens coordinate system. (Continue to refer to...) Figure 2 The target positions P1 and P2 on the two wheels represent the positions of the left and right rear wheels, respectively, located on the ground and centered at point O of the rear wheels. V In a vehicle-plane coordinate system centered at a circle, with the direction from the right target P2 to the left target P1 as the positive Y-axis and the forward direction of the vehicle as the positive X-axis, we can set z = 0. The position coordinates of the left target point P... 1,v (0,y v The position coordinates of the right target point P 2,v (0,-y v ), 2 times y v This represents the actual distance between the two target points, in meters. For example, y v =0.7 meters, then the position of the left target point is O, which is a distance from the center point of the rear wheel of the car. V The target point is located 0.7 meters to the left and 0.7 meters to the right of the center of the rear wheel of the vehicle. The actual distance between the two targets is 1.4 meters.

[0103] The target point detection includes the following steps:

[0104] S21, an image 9 is obtained by capturing the wheel target at the left wheel through the fisheye camera cam2, and the coordinates of the left target point position P1 on the image 9 are p cam2 (x 2,P1 ,y 2,P1 ); an image 10 is obtained by capturing the wheel target at the right wheel through the fisheye camera cam3, and the coordinates of the right target point position P2 on the image 10 are p cam3 (x 3,P2 ,y 3,P2 ).

[0105] S22, a conversion relationship between the fisheye camera coordinate system and the vehicle body plane coordinate system is calculated to determine the extrinsic parameters of the fisheye camera coordinate system.

[0106] In an implementable embodiment, step 22 includes steps S221-S224.

[0107] S221, a conversion relationship between the fisheye camera cam3 coordinate system and the vehicle body coordinate system is calculated to determine the extrinsic parameters of the fisheye camera cam3 coordinate system and the vehicle body plane coordinate system, including steps S2211-S2217.

[0108] S2211, an image 11 is obtained by capturing the checkerboard calibration board U3 through the fisheye camera cam3, and a conversion relationship between the fisheye camera cam3 coordinate system and the checkerboard calibration board U3 coordinate system is obtained according to the coordinates of a plurality of pairs of same position points in the image 11 and the checkerboard calibration board U3 coordinate system, which is:

[0109] p U3 =M cam3→U3 E cam3→U3 p cam3 (14)

[0110] wherein M cam3→U2 is a magnification matrix, wherein the magnification elements are obtained by calculating the ratio of the length value of a unit cell on the checkerboard calibration board U3 and the pixel length value of the unit cell on the image, E cam3→U3 is the extrinsic parameters of the fisheye camera cam3 and the checkerboard calibration board U3 coordinate system, including a rotation matrix R cam3→U3 and a translation transformation matrix T cam3→U3 .

[0111] It can be understood that the length value of the unit cell in the image 11 is in pixels, and the length value of the unit cell in the checkerboard calibration board U3 coordinate system is in meters.

[0112] S2212, according to the conversion relationship between the coordinate systems of the left fisheye camera cam2 and the right fisheye camera cam3 and formula (11), the position P1 of the left target point in the coordinate system of the fisheye camera cam3 is obtained as:

[0113] p cam3 (x 3,P1 ,y 3,P1 )=E cam2→cam3 p cam2 (x 2,P1 ,y 2,P1 ) (15)

[0114] S2213, according to the conversion relationship between the coordinate system of the fisheye camera cam3 and the coordinate system of the chessboard calibration plate U3, formula (14) and (15) are solved to obtain the coordinates p U3 (x U3,P1 ,y U3,P1 ) of the left target point P1 in the coordinate system of the chessboard calibration plate U3:

[0115] p U3 (x U3,P1 ,y U3,P1 )=M cam3→U3 E cam3→U3 p cam3 (x 3,P1 ,y 3,P1 )

[0116] =M cam3→U3 E cam3→U3 E cam2→cam3 p cam2 (x 2,P1 ,y 2,P1 ) (16)

[0117] Let E cam2→U3 =M cam3→U3 E cam3→U3 E cam2→cam3 , then the coordinates of the left target point P1 in the coordinate system of the chessboard calibration plate U3 are:

[0118] p U3 (x U3,P1 ,y U3,P1 )=E cam2→U3 p cam2 (x 2,P1 ,y 2,P1 ) (17)

[0119] S2214, according to the conversion relationship between the coordinate system of the fisheye camera cam3 and the coordinate system of the chessboard calibration plate U3, the coordinates of the right target point P2 in the coordinate system of the chessboard calibration plate U3 are obtained as:

[0120] p U3(x U3,P2 ,y U3,P2 ) = M cam3→U3 E cam3→U3 p cam3 (x 3,P2 ,y 3,P2 (18)

[0121] S2215, determine the center O of the vehicle body plane coordinate system based on the midpoints of the left and right targets in the coordinate system of the checkerboard calibration plate U3. V The coordinates in the checkerboard U3 coordinate system are:

[0122]

[0123] S2216, based on the center point O of the vehicle body plane coordinate system V Given the coordinates of the left target point P1 and the right target point P2 in the coordinate system of the checkerboard calibration plate U3, and the positive y-axis direction of the vehicle plane coordinate system (the direction from the right target to the left target), determine the rotation matrix R in the plane between the checkerboard U3 calibration plate coordinate system and the vehicle plane coordinate system. U3→V Translation matrix T U3→V Since the two coordinate systems belong to the same plane (ground), the transformation relationship between the vehicle body plane coordinate system and the coordinate system of the checkerboard calibration plate U3 is as follows:

[0124] p V =R U3→V p U3 +T U3→V (20)

[0125] Where, p V Let E be the vehicle body plane coordinate system. U3→V =R U3→V T U3→V The transformation relationship between the vehicle body plane coordinate system and the checkerboard U3 coordinate system is as follows:

[0126] p V =E U3→V p U3 (twenty one)

[0127] S2217, based on the transformation relationship between the vehicle body plane coordinate system and the checkerboard U3 coordinate system, and the transformation relationship between the fisheye lens cam3 and the checkerboard U3 coordinate system, solve equations (14) and (21) simultaneously to determine the relationship between the fisheye lens cam3 and the vehicle body plane coordinate system as follows:

[0128] p V =E U3-V M cam3→U3 E cam3→U3 p cam3 (twenty two)

[0129] Let E cam3-V = E U3-V M cam3→U3 E cam3→U3 , the relationship between fisheye camera cam3 coordinate system and vehicle body plane coordinate system is:

[0130] p V = E cam3-V p cam3 (23)

[0131] Thus, the extrinsic parameter E cam3-V between fisheye camera cam3 and vehicle body plane coordinate system is determined.

[0132] S222, the conversion relationship between fisheye camera cam2 coordinate system and vehicle body coordinate system is calculated to determine the extrinsic parameter between fisheye camera cam2 coordinate system and vehicle body plane coordinate system.

[0133] The extrinsic parameter E cam2-V between fisheye camera cam2 and vehicle body plane coordinate system is obtained according to the above steps S2211-S2217.

[0134] It should be noted that fisheye camera cam2 captures image 12 of the checkerboard calibration board U1, and the conversion relationship (M cam2→U1 E cam2→U1 ) between fisheye camera cam2 coordinate system and checkerboard calibration board U1 coordinate system is obtained according to the coordinates of multiple pairs of same position points in image 12 and checkerboard calibration board U1 coordinate system; the pixel coordinates p cam2 (x 2,P1 ,y 2,P1 ) and p cam3 (x 3,P2 ,y 3,P2 ) of left and right target points are converted to checkerboard calibration board U1 coordinate system to determine the center O v of vehicle body plane coordinate system in the coordinates of checkerboard calibration board U1 coordinate system and the positive direction of y-axis of vehicle body plane coordinate system (the direction from right target to left target), so as to determine the rotation matrix R U1→V and translation transformation matrix T U1→V of vehicle body plane coordinate system and checkerboard calibration board U1 coordinate system in the plane, because the two coordinates belong to the same plane (ground), so the conversion relationship E U1→V between vehicle body plane coordinate system and checkerboard U1 coordinate system is obtained, according to the conversion relationship E U1→V between vehicle body plane coordinate system and checkerboard U1 coordinate system, the conversion relationship E cam2→U1 between fisheye camera cam2 and checkerboard U1 coordinate system, the relationship between fisheye camera cam2 and vehicle body plane coordinate system is:

[0135] pV = E U1-V M cam2→U1 E cam2→U1 p cam2 (24)

[0136] wherein E U1→V = R U1→V T U1→V Let E cam2-V = E U1-V M cam2→U1 E cam2→U1 , then the relationship between fisheye camera cam2 coordinate system and vehicle body plane coordinate system is:

[0137] p V = E cam2-V p cam2 (25)

[0138] Thus, the extrinsic parameter E cam2-V between fisheye camera cam2 coordinate system and vehicle body plane coordinate system is calibrated.

[0139] S223, the conversion relationship between fisheye camera cam1 coordinate system and vehicle body coordinate system is calculated to determine the extrinsic parameter between fisheye camera cam1 coordinate system and vehicle body plane coordinate system.

[0140] In an implementable embodiment, the conversion relationship between fisheye camera cam1 coordinate system and vehicle body coordinate system can be determined according to the conversion relationship E cam1→cam2 between fisheye camera cam2 coordinate system and fisheye camera cam1 coordinate system, the conversion relationship E cam2-V between fisheye camera cam2 coordinate system and vehicle body plane coordinate system, simultaneous equations (5) and (25) are solved, and the conversion relationship between fisheye camera cam1 coordinate system and vehicle body coordinate system is:

[0141] p V = E cam2-V E cam1→cam2 p cam1 (26)

[0142] Let E cam1-V = E cam2-V E cam1→cam2 , then the relationship between fisheye camera cam1 coordinate system and vehicle body plane coordinate is:

[0143] p V = E cam1-V p cam1 (27)

[0144] In an implementable embodiment, the conversion relationship between fisheye camera cam3 coordinate system and fisheye camera cam1 coordinate system can be determined according to the conversion relationship E cam1→cam3 between fisheye camera cam3 coordinate system and fisheye camera cam1 coordinate system, and the relationship Ecam3-V , and formula (23) is solved to determine the conversion relationship between fisheye camera cam1 coordinate system and vehicle body coordinate system as follows:

[0145] p V = E cam3-V E cam1→cam3 p cam1 (28)

[0146] Let E cam1-V = E cam3-V E cam1→cam3 , then the relationship between fisheye camera cam1 and vehicle body plane coordinates is as follows:

[0147] p V = E cam1-V p cam1 (29)

[0148] Thus, the extrinsic parameter E cam1-V of fisheye camera cam1 and vehicle body plane coordinates is calibrated.

[0149] In step S224, the conversion relationship between fisheye camera cam4 coordinate system and vehicle body coordinate system is calculated to determine the extrinsic parameter of fisheye camera cam4 coordinate system and vehicle body plane coordinate system.

[0150] Similarly to step S223, in an implementable embodiment, the conversion relationship E cam2→cam4 between fisheye camera cam2 coordinate system and fisheye camera cam4 coordinate system, and the conversion relationship E cam2-V between fisheye camera cam2 coordinate system and vehicle body plane coordinate system are used to solve formula (9) and formula (25) to determine the conversion relationship between fisheye camera cam4 coordinate system and vehicle body coordinate system as follows: cam4 = E cam2→cam4 p cam2

[0151] p V (x V ,y V ) = E cam2-V E cam4→cam2 p cam4 (x4,y4) (30)

[0152] Wherein, E cam4→cam2 E cam2→cam4 = 1, let E cam4-V = E cam2-V E cam4→cam2 , then the conversion relationship between fisheye camera cam4 coordinate system and vehicle body plane coordinate system is as follows:

[0153] p V = E cam4-V pcam4 (31)

[0154] In an implementable embodiment, the conversion relationship E cam3→cam4 between the fisheye lens cam3 coordinate system and the fisheye lens cam4 coordinate system can be determined according to the conversion relationship E cam3-V between the fisheye lens cam3 and the vehicle body plane coordinate system, i.e., solving formula (13) and formula (23) to determine the conversion relationship E

[0155] p V =E cam3-V E cam4→cam3 p cam4 (32)

[0156] wherein E cam4→cam3 E cam3→cam4 =1, E cam4-V =E cam3-V E cam4→cam3 , the relationship between the fisheye lens cam4 and the vehicle body plane coordinate system is:

[0157] p V =E cam4-V p cam4 (33)

[0158] Thus, the extrinsic parameter E cam4-V between the fisheye lens cam4 coordinate system and the vehicle body plane coordinate system is calibrated.

[0159] According to the fisheye lens extrinsic parameter calibration method provided in the embodiments of the present application, because the two coordinate systems belong to the same plane (ground), the conversion relationship between the vehicle body plane coordinate system and the checkerboard calibration plate coordinate system only needs to use four checkerboard calibration plates and two wheel targets (hereinafter referred to as targets), and the calibration of the fisheye lens extrinsic parameter can be completed in any open and level ground area.

[0160] In addition, the factors affecting whether the fisheye lens extrinsic parameter calibration is accurate or not also include the influence of the vehicle body pose on the fisheye lens. Specifically, during the calibration of the fisheye lens extrinsic parameter, the condition of the vehicle body itself will affect the fisheye lens extrinsic parameter, such as the number of people sitting during calibration, and the tire condition will all affect the inclination of the vehicle body during calibration, so that the position and shooting angle of the fisheye lens are inclined, indirectly causing the extrinsic parameter during calibration to be inconsistent with the actual driving condition of the vehicle body, at this time z≠0, affecting the evaluation result of the surrounding environment of the intelligent vehicle body, so the calibration of the fisheye lens extrinsic parameter also needs to exclude the influence of these vehicle body pose factors.

[0161] The fish-eye lens external parameter calibration method provided in the embodiments of the present application obtains the current state of the vehicle body and the accurate relative position relationship with the ground in combination with IMU information, fine tunes the obtained preliminary fish-eye external parameters, and removes the influence of the vehicle body posture on the external parameters.

[0162] Figure 3 The principle diagram of adjusting the fish-eye lens external parameter using the IMU sensor in the fish-eye lens external parameter calibration method provided in the embodiments of the present application is shown. The IMU sensor can be installed at the position of the center point of the vehicle body on the chassis to monitor the change of the vehicle body posture in real time and identify some relatively complex road condition information. As shown in Figure 3 , due to the relative displacement between the vehicle body and the chassis in the actual driving process, the center point of the vehicle body is not projected onto the origin o z of the vehicle body plane coordinate system parallel to the ground, but is projected onto the origin o' z of another coordinate system based on the change of the IMU information. The coordinate system in which the origin o' z is located can be referred to as the IMU coordinate system. The conversion relationship between the vehicle body plane coordinate system and the IMU coordinate system is as follows:

[0163] p imu =E imu→V p V (34)

[0164] wherein p imu is the IMU coordinate system under the current IMU information, E imu→V =R imu→V T ium→V , R imu→V is the rotation matrix of the IMU coordinate system and the vehicle body plane coordinate system under the current IMU information, and T ium→V is the translation transformation matrix of the IMU coordinate system and the vehicle body plane coordinate system under the current IMU information.

[0165] Figure 4 The IMU information and the vehicle body posture angle information schematic diagram in the fish-eye lens external parameter calibration method provided in the embodiments of the present application is shown. As shown in Figure 4As shown, the IMU coordinate system includes a pitch axis, a roll axis, and a yaw axis, and an origin o'. The IMU information is the angular acceleration information of the vehicle body output by the IMU sensor, and the inclination angle of the current vehicle body relative to the ground can be obtained according to the angular acceleration information, that is, the pose of the vehicle body during calibration. The deflection angles of the Y axis, the X axis, and the Z axis of the IMU coordinate system and the vehicle body plane coordinate system can be read from the IMU information, that is, the values of the roll angle (roll), the heading angle (heading), and the pitch angle (pich). According to the values of the pitch angle, the roll angle, and the heading angle, the rotation matrix R imu→V of the IMU coordinate system and the vehicle body plane coordinate system is obtained. In combination with the relationship between the fisheye lens and the vehicle body plane coordinate system as shown in formula (23), formula (25), formula (27), or formula (31), the conversion relationship of the fisheye lens coordinate system relative to the vehicle body plane coordinate system is also projected onto the IMU coordinate system based on the change of the IMU information, and the conversion relationship between the fisheye lens coordinate system and the IMU coordinate system is converted by the following formula:

[0166] p imu = E imu→V E cam→V p cam (35)

[0167] Let E imu→cam = E imu→V E cam→V Then:

[0168] p imu = E imu→cam p cam (36)

[0169] E imu→cam is the conversion matrix between the IMU coordinate system and the fisheye lens coordinate system under the influence of the current pose angle r imu (pich, roll, heading) of the vehicle body.

[0170] During the driving of the vehicle, the rotation matrix R imu→V of the IMU coordinate system and the vehicle body plane coordinate system under the current IMU information and the translation transformation matrix T ium→V of the IMU coordinate system and the vehicle body plane coordinate system under the current IMU information are dynamically adjusted according to the real-time attitude value between the vehicle body and the ground collected by the real-time inertial measurement unit IMU, the influence of the current attitude on the fisheye lens coordinate system is dynamically removed, and the accuracy of the fisheye lens external parameter under different vehicle body attitude values at each moment is ensured.

[0171] Based on the above principles, embodiments of this application provide a method for calibrating the extrinsic parameters of fisheye lenses, used to calibrate the extrinsic parameters of multiple fisheye lenses mounted on a vehicle. The multiple fisheye lenses are respectively positioned at the front, rear, left, and right sides of the vehicle body. The method includes: determining the transformation relationship between the coordinate systems of the multiple fisheye lenses based on images of multiple checkerboard calibration plates captured by the multiple fisheye lenses; placing the multiple checkerboard calibration plates flat on the ground at the four corners of the vehicle (left rear, left front, right rear, and right front); determining the extrinsic parameters of each fisheye lens based on the transformation relationship between the coordinate systems of the multiple fisheye lenses and the images of the first and second wheel targets in the multiple fisheye lenses; wherein the first and second wheel targets are respectively placed flat against the left and right rear wheels of the vehicle, close to the checkerboard calibration plates.

[0172] The fisheye lens extrinsic parameter calibration method provided in the embodiments of this application is applied in vehicle calibration sites and subsequent service providers, as well as in scenarios where vehicle extrinsic parameter calibration of fisheye lenses is required. The extrinsic parameters of the fisheye lens are jointly calibrated based on visual information from multiple reference objects and IMU information.

[0173] Figure 5 This is a system architecture diagram of the fisheye lens extrinsic parameter calibration method provided in an embodiment of this application. (See diagram below.) Figure 5 As shown, it includes a spatial arrangement module 51, a measurement and calculation module 52, and a dynamic parameter acquisition module 53.

[0174] The spatial arrangement module 51 is used to determine the calibration space.

[0175] In one feasible implementation, an open and level area is defined as a calibration space, and multiple reference objects are arranged at multiple locations within the calibration space.

[0176] For example, during calibration, the car can be parked in an open area with a level ground. Four identical checkerboard calibration plates U1, U2, U3, and U4 are set at the four corners of the car: left rear, left front, right rear, and right front, as standard references. At the same time, wheel targets are placed at the two rear wheels of the car, with the target points P1 and P2 representing the positions of the two rear wheels.

[0177] In one feasible implementation, the four fisheye lenses to be calibrated, cam1, cam2, cam3 and cam4, can be set at the rear, left, right and front positions of the vehicle body, respectively.

[0178] The measurement calculation module 12 is configured to collect images of the four fisheye lenses cam1, cam2, cam3 and cam4, detect the chessboard calibration plate and the wheel target according to the images respectively, and preliminarily determine the extrinsic parameters of the four fisheye lenses by calculating the conversion relationship between the four fisheye lens coordinate systems and the conversion relationship between the four fisheye lens coordinate systems and the vehicle plane coordinate system, the conversion relationship between the vehicle plane coordinate system and the chessboard calibration plate coordinate system.

[0179] In an implementable embodiment, the images of the four fisheye lenses and the data of the current IMU sensor can be collected and recorded synchronously, and the chessboard calibration plate detection and the target point detection can be performed.

[0180] The dynamic parameter obtaining module 13 is configured to collect the angular acceleration data output by the current IMU sensor, wherein the angular acceleration data includes the values of the pitch angle (pich), the roll angle (roll) and the heading angle (heading); obtain the relative position relationship between the current attitude of the vehicle body and the ground according to the angular acceleration data, so as to obtain the conversion relationship between the IMU coordinate system and the vehicle plane coordinate system under the current IMU information, and fine-tune the preliminarily determined extrinsic parameters of the four fisheye lenses to remove the influence of the vehicle body attitude on the extrinsic parameters.

[0181] When the vehicle body is normally driven, the extrinsic parameter can be dynamically updated and adjusted according to the real-time IMU information, so as to ensure high precision and stability.

[0182] Figure 6 A flowchart of the extrinsic parameter calibration method of the fisheye lens provided by the embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the embodiment of the present application calibrates the extrinsic parameters of the fisheye lens based on the vision and IMU information, which includes the following steps:

[0183] S61, determine the calibration space, and set multiple chessboard calibration plates and two wheel targets.

[0184] In an implementable embodiment, the step of determining the calibration space can include the following steps:

[0185] S611, place the vehicle in an open and level area.

[0186] S612, place multiple chessboard calibration plates on the ground at the left front, left rear, right front and right rear corner points of the vehicle body.

[0187] For example, four same chessboard calibration plates U1, U2, U3 and U4 are set as standard reference objects on the ground at the left rear, left front, right rear and right front corner points of the vehicle.

[0188] S613, place the wheel targets at the positions of the two rear wheels of the vehicle, and the positions P1 and P2 of the target points represent the positions of the two rear wheels.

[0189] The vehicle body plane coordinate system is determined according to the wheel targets, wherein the vehicle body plane coordinate system takes the center points of the left and right rear wheels of the vehicle as the centers, takes the direction from the second wheel target to the first wheel target as the positive direction of the Y axis, and takes the front direction of the vehicle as the positive direction of the X axis.

[0190] In an implementable embodiment, the plurality of fisheye lenses that need to be calibrated are arranged at the front, rear, left, and right parts of the vehicle body, respectively. Exemplarily, the four fisheye lenses that need to be calibrated are cam1, cam2, cam3, and cam4, which are arranged at the rear, left, right, and front parts of the vehicle body, respectively.

[0191] In an implementable embodiment, the IMU sensor is arranged at the center position of the vehicle body chassis.

[0192] S62, image acquisition, synchronously acquiring images of the four fisheye lenses cam1, cam2, cam3, and cam4, and obtaining coordinates of a plurality of position points of the checkerboard calibration board in the fisheye lens coordinate system.

[0193] In an implementable embodiment, the fisheye lens cam1 and the fisheye lens cam2 simultaneously capture the checkerboard calibration board U1, the fisheye lens cam2 and the fisheye lens cam4 simultaneously capture the checkerboard calibration board U2, the fisheye lens cam1 and the fisheye lens cam3 simultaneously capture the checkerboard calibration board U3, and the fisheye lens cam3 and the fisheye lens cam4 simultaneously capture the checkerboard calibration board U4, and the vehicle acquires image data of the four fisheye lenses.

[0194] S63, corner point detection of the checkerboard calibration board is performed by the four fisheye lenses cam1, cam2, cam3, and cam4, respectively, to obtain coordinates of the same position points in different fisheye lens coordinate systems.

[0195] Exemplarily, the same position points can be the origin of the checkerboard calibration board, the corner points of the unit cells, and the like. Then, step S63 is implemented by the following steps:

[0196] S631, coordinates of a position point of the checkerboard calibration board U1 in image 1 are obtained as p cam1 (x 1,U1 ,y 1,U1 ) by image 1 of the fisheye lens cam1, and coordinates of the same position point of the checkerboard calibration board U1 in image 2 are obtained as p cam2 (x 2,U1 ,y 2,U1 ) by image 2 of the fisheye lens cam2.

[0197] S632, the fisheye camera cam1 and cam3 simultaneously capture the checkerboard calibration board U3, the coordinates of the same position point of the checkerboard calibration board U3 in the image 3 obtained by the fisheye camera cam1 is p cam1 (x 1,U3 ,y 1,U3 ), the coordinates of the same position point of the checkerboard calibration board U3 in the image 4 obtained by the fisheye camera cam3 is p cam3 (x 3,U3 ,y 3,U3 ).

[0198] S633, the coordinates of the position point of the checkerboard calibration board U2 in the image 5 obtained by the fisheye camera cam2 is p cam2 (x 2,U2 ,y 2,U2 ), the coordinates of the same position point of the checkerboard calibration board U2 in the image 6 obtained by the fisheye camera cam4 is p cam4 (x 4,U2 ,y 4,U2 ).

[0199] S634, the fisheye camera cam3 and cam4 simultaneously capture the checkerboard calibration board U4, the coordinates of the same position point of the checkerboard calibration board U4 in the image 7 obtained by the fisheye camera cam3 is p cam3 (x 3,U4 ,y 3,U4 ), the coordinates of the same position point of the checkerboard calibration board U4 in the image 8 obtained by the fisheye camera cam4 is p cam4 (x 4,U4 ,y 4,U4 ).

[0200] S64, the conversion relationship between the multiple fisheye camera coordinate systems is determined according to the images of the multiple checkerboard calibration boards captured by the multiple fisheye cameras.

[0201] In an implementable embodiment, the conversion relationship between the two fisheye camera coordinate systems can be calculated according to the coordinates of multiple pairs of same position points of the same checkerboard calibration board on the images of the two fisheye cameras, and the conversion relationship between the four fisheye camera coordinate systems can be determined according to the double calibration results of the two fisheye cameras.

[0202] In an implementable embodiment, the first fisheye lens of the plurality of fisheye lenses can be arranged at one of the front, rear, left and right of the vehicle, and the second fisheye lens can be arranged at one of the front, rear, left and right of the vehicle, adjacent to the first fisheye lens. Then, step S64 is implemented by obtaining the first image captured by the first fisheye lens, determining the conversion relationship between the first fisheye lens coordinate system and the second fisheye lens coordinate system according to the coordinates of the plurality of pairs of same position points in the first chessboard calibration plate of the first image and the first chessboard calibration plate of the second image, and recording the conversion relationship between the first fisheye lens coordinate system and the second fisheye lens coordinate system as the first conversion relationship. Exemplarily, fisheye lens cam1 can be taken as the first fisheye lens, fisheye lens cam2 can be taken as the second fisheye lens, and chessboard calibration plate U1 can be taken as the first chessboard calibration plate. Then, step S64 can be implemented by the following step S641.

[0203] S641, obtaining the conversion relationship between the fisheye lens cam2 coordinate system and the coordinate system of fisheye lens cam1 according to the plurality of coordinate pairs of the plurality of position points of chessboard calibration plate U1 in image 1 and image 2 as formula (5):

[0204] p cam2 =E cam1→cam2 p cam1 .

[0205] At this time, the conversion relationship between the fisheye lens cam1 coordinate system and the second fisheye lens cam2 coordinate system can be taken as the first conversion relationship.

[0206] In an implementable embodiment, the conversion relationship between the fisheye lens cam2 coordinate system and the coordinate system of fisheye lens cam1 can also be determined according to the scaling matrix, rotation and translation of the unit cell of chessboard calibration plate U1.

[0207] Exemplarily, the first length value and the first corner point coordinate of the unit cell of chessboard calibration plate U1 on image 1 can be obtained, the second length value and the second corner point coordinate of the unit cell of chessboard calibration plate U1 on image 2 can be obtained, and the conversion relationship between the fisheye lens cam1 coordinate system and the fisheye lens cam2 coordinate system can be obtained according to the first and second length values and the first and second corner point coordinates:

[0208] p cam2 =M cam1→cam2 E cam1→cam2 p cam1 (37)

[0209] wherein M cam1→cam2 is the scaling matrix between the fisheye lens cam1 coordinate system and the fisheye lens cam2 coordinate system.

[0210] According to the previous example, the conversion relationship of the coordinate system of the fisheye lens cam1, the coordinate system of the fisheye lens cam2 and the coordinate system of the checkerboard calibration board U1 can also be determined according to the unit length value, the first length value and the second length value of the checkerboard calibration board U1, as shown in the following formula:

[0211] p U1 =M cam1→U1 E cam1→U1 p cam1 (38)

[0212] p U2 =M cam2→U1 E cam2→U1 p cam2 (39)

[0213] In an implementable embodiment, the third fisheye lens in the plurality of fisheye lenses is arranged at one of the front, rear, left and right positions of the vehicle, adjacent to the first fisheye lens and opposite to the second fisheye lens, and then the step S64 can obtain a third image including a third checkerboard calibration board, the third image being captured by the first fisheye lens; the third checkerboard calibration board is one of the plurality of checkerboard calibration boards; a fourth image captured by the third fisheye lens is obtained, and the third fisheye lens coordinate system is determined according to the third checkerboard calibration board on the fourth image; the second conversion relationship is determined according to the coordinates of the plurality of pairs of same position points on the third checkerboard calibration board in the third image and the fourth image, the second conversion relationship being the conversion relationship between the third fisheye lens coordinate system and the first fisheye lens coordinate system; the third conversion relationship is determined according to the second conversion relationship and the first conversion relationship, the third conversion relationship being the conversion relationship between the third fisheye lens coordinate system and the second fisheye lens coordinate system.

[0214] Exemplarily, the fisheye lens cam3 can be taken as the third fisheye lens, the fisheye lenses cam1 and cam2 can be taken as the first and second fisheye lenses, and the checkerboard calibration board U3 can be taken as the third checkerboard calibration board, and then the step S64 can be implemented through the following steps S642-643.

[0215] S642, the conversion relationship between the fisheye lens cam3 coordinate system and the fisheye lens cam1 coordinate system is obtained according to the coordinate pairs of the plurality of position points of the checkerboard calibration board U3 in the image 3 and the image 4, as shown in the following formula (7):

[0216] p cam3 =E cam1→cam3 p cam1 .

[0217] At this time, the conversion relationship between the fisheye lens cam3 coordinate system and the first fisheye lens cam1 coordinate system can be taken as the second conversion relationship.

[0218] S643, determining the conversion relationship between the fisheye lens cam3 coordinate system and the fisheye lens cam2 coordinate system according to the second conversion relationship and the first conversion relationship, and recording the conversion relationship between the third fisheye lens coordinate system and the second fisheye lens coordinate system as a third conversion relationship.

[0219] Exemplarily, in order to reduce errors and improve positioning accuracy when observing the surrounding environment by 360°, and further obtain the conversion relationship between the four fisheye lens coordinate systems, the step S64 can further obtain the conversion relationship between the fisheye lens cam4 coordinate system and the fisheye lens cam2 coordinate system and the conversion relationship between the fisheye lens cam4 coordinate system and the fisheye lens cam3 coordinate system through steps S644-S645.

[0220] S644, obtaining the conversion relationship between the fisheye lens cam4 coordinate system and the fisheye lens cam2 coordinate system according to the coordinate pairs of the plurality of position points of the checkerboard calibration board U2 in the image 5 and the image 6 as formula (9):

[0221] p cam4 =E cam2→cam4 p cam2

[0222] S645, obtaining the conversion relationship between the fisheye lens cam4 coordinate system and the fisheye lens cam3 coordinate system according to the coordinate pairs of the plurality of position points of the checkerboard calibration board U4 in the image 7 and the image 8 as formula (13):

[0223] p cam4 =E cam3→cam4 p cam3

[0224] In the above embodiment, the rotation matrix R and the translation transformation matrix T between the first fisheye lens coordinate system and the second fisheye lens coordinate system can also be calculated according to the coordinates of a plurality of pairs of same position points in the checkerboard calibration board of the first image and the checkerboard calibration board of the second image, and the first conversion relationship is determined.

[0225] Exemplarily, the rotation matrix R cam2→cam1 and the translation transformation matrix T cam2→cam1 between the fisheye lens cam1 coordinate system and the fisheye lens cam2 coordinate system can be calculated according to the coordinates of a plurality of pairs of same position points in the checkerboard calibration board U1 of the image 1 and the checkerboard calibration board U1 of the image 2, and the conversion relationship between the fisheye lens cam1 coordinate system and the fisheye lens cam2 coordinate system is determined.

[0226] S65, target point detection is performed according to the conversion relationship between the coordinate systems to obtain the extrinsic parameters of the fisheye lens.

[0227] In an implementable embodiment, the extrinsic parameters of each of the plurality of fisheye lenses can be determined according to the conversion relationship among the plurality of fisheye lens coordinate systems and the images of the first and second wheel targets in the plurality of fisheye lenses.

[0228] Exemplarily, the wheel targets at the left and right rear wheels are taken as the first and second wheel targets. The target point detection can be performed on the wheel targets at the left and right rear wheels, the images of the two wheel targets in the plurality of fisheye lenses are obtained, the conversion relationship of the vehicle body plane coordinate system relative to the fisheye lens coordinate system is determined according to the conversion relationship among the four fisheye lens coordinate systems and the position coordinates of the target points corresponding to the wheel targets at the left and right rear wheels in the images, so as to obtain the extrinsic parameters of the plurality of fisheye lenses. In an implementable embodiment, step S65 comprises steps S651-S654.

[0229] S651, determining a fifth conversion relationship according to the images of the plurality of checkerboard calibration boards captured by the plurality of fisheye lenses, the fifth conversion relationship being the conversion relationship between each of the plurality of fisheye lens coordinate systems and the corresponding checkerboard calibration board coordinate system.

[0230] In an implementable embodiment, step S651 is implemented by the following steps.

[0231] S6511, obtaining a fifth image captured by each of the plurality of fisheye lenses, the fifth image being an image including one of the plurality of checkerboard calibration boards.

[0232] S6512, obtaining the fifth conversion relationship according to the fifth image and the coordinates of the plurality of pairs of same-position points on the corresponding checkerboard calibration board, the fifth conversion relationship including the scale matrix, the rotation matrix and the translation transformation matrix between each of the fisheye lens coordinate systems and the corresponding checkerboard calibration board coordinate system; wherein the checkerboard calibration board coordinate system is a coordinate system determined by the X-axis, the Y-axis, the origin and the unit length on the checkerboard calibration board.

[0233] The specific implementation and examples can refer to the entire content of step S2211, which will not be repeated here.

[0234] S652, determining a sixth conversion relationship according to the conversion relationship among the plurality of fisheye lens coordinate systems and the images of the first and second wheel targets in the plurality of fisheye lenses, the sixth conversion relationship being the conversion relationship between the vehicle body plane coordinate system and the plurality of checkerboard calibration board coordinate systems.

[0235] In an implementable embodiment, step S652 is implemented by the following steps.

[0236] S6521, obtain a sixth image and a seventh image captured by the second fisheye lens and the third fisheye lens in the plurality of fisheye lenses; the sixth image is an image including the first wheel target captured by the second fisheye lens, and the seventh image is an image including the second wheel target captured by the third fisheye lens.

[0237] Exemplarily, the image 9 can be obtained by capturing the wheel target at the left wheel through the fisheye lens cam2, and the coordinates of the left target point position P1 on the image 9 are p cam2 (x 2,P1 ,y 2,P1 ); the image 10 can be obtained by capturing the wheel target at the right wheel through the fisheye lens cam3, and the coordinates of the right target point position P2 on the image 10 are p cam3 (x 3,P2 ,y 3,P2 ). Here, the image 9 can be regarded as the sixth image, the image 10 can be regarded as the seventh image, the fisheye lens cam2 can be regarded as the second fisheye lens, and the fisheye lens cam3 can be regarded as the third fisheye lens.

[0238] S6522, determine the coordinates of the midpoint between the first wheel target and the second wheel target in the first or third chessboard calibration plate coordinate system according to the conversion relationship between the second fisheye lens coordinate system and the third fisheye lens coordinate system in the conversion relationship between the plurality of fisheye lens coordinate systems.

[0239] Exemplarily, the specific implementation can refer to all or part of the contents of steps S2212-S22125, which will not be described here.

[0240] S6523, determine the sixth conversion relationship according to the coordinates of the midpoint between the first wheel target and the second wheel target in the first or third chessboard calibration plate coordinate system, the coordinates of the first wheel target and the second wheel target in the second or third chessboard calibration plate coordinate system, and the direction in which the second wheel target points to the first wheel target.

[0241] Exemplarily, the specific implementation can refer to all or part of the contents of step S2216, which will not be described here.

[0242] S653, determine a seventh conversion relationship according to the fifth conversion relationship and the sixth conversion relationship, the seventh conversion relationship being a conversion relationship between each fisheye lens coordinate system and the vehicle body plane coordinate system.

[0243] S654, determine the extrinsic parameters of each fisheye lens in the plurality of fisheye lenses according to the seventh conversion relationship.

[0244] In an implementable embodiment, the conversion relationship between the second fisheye lens coordinate system and the vehicle body plane coordinate system can be determined according to the conversion relationship between the second fisheye lens coordinate system and the corresponding first checkerboard calibration plate coordinate system, and the sixth conversion relationship between the first checkerboard calibration plate coordinate system and the vehicle body plane coordinate system.

[0245] The specific implementation can refer to all or part of the content of step S222, which will not be repeated here.

[0246] In an implementable embodiment, the conversion relationship between the third fisheye lens coordinate system and the vehicle body plane coordinate system can be determined according to the conversion relationship between the third fisheye lens coordinate system and the corresponding third checkerboard calibration plate coordinate system, and the sixth conversion relationship between the third checkerboard calibration plate coordinate system and the vehicle body plane coordinate system.

[0247] The specific implementation can refer to all or part of the content of step S2217, which will not be repeated here.

[0248] In an implementable embodiment, the conversion relationship between the first fisheye lens coordinate system and the vehicle body plane coordinate system can be determined according to the conversion relationship between the first fisheye lens coordinate system and the third fisheye lens coordinate system, the conversion relationship between the third fisheye lens coordinate system and the corresponding third checkerboard calibration plate coordinate system, and the sixth conversion relationship between the third checkerboard calibration plate coordinate system and the vehicle body plane coordinate system. The specific implementation can refer to all or part of the content of step S223, which will not be repeated here.

[0249] Similarly, the conversion relationship between the fourth fisheye lens coordinate system and the vehicle body plane coordinate system can be determined. The specific implementation can refer to all or part of the content of step S224, which will not be repeated here.

[0250] In an implementable embodiment, the seventh conversion relationship includes a rotation matrix and a translation transformation matrix between each fisheye lens coordinate system and the vehicle body plane coordinate system; and the preliminary extrinsic parameters of the first, second, third, and fourth fisheye lens coordinate systems are determined according to the rotation matrix and the translation transformation matrix between the first, second, third, and fourth fisheye lens coordinate systems and the vehicle body plane coordinate system.

[0251] S66, adjusting the fisheye lens extrinsic parameters using IMU information.

[0252] In an implementable embodiment, an inertial measurement unit (IMU) can be arranged on a chassis of a vehicle body, and the IMU is configured to determine values of a roll angle, a pitch angle and a heading angle of the vehicle body; the vehicle obtains the values of the roll angle, the pitch angle and the heading angle output by the IMU; determines an IMU coordinate system according to the values of the roll angle, the pitch angle and the heading angle; wherein the values of the roll angle, the pitch angle and the heading angle indicate angular deflection of three axes of the IMU coordinate system and a vehicle body plane coordinate system, i.e., a Y-axis, an X-axis and a Z-axis; determines a conversion relationship between a plurality of fisheye lens coordinate systems and the IMU coordinate system; and adjusts extrinsic parameters of the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system.

[0253] In an implementable embodiment, a rotation matrix of the IMU coordinate system and the vehicle body plane coordinate system can be obtained according to the values of the pitch angle, the roll angle and the heading angle; and a conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system is determined according to the rotation matrix of the IMU coordinate system and the vehicle body plane coordinate system and a conversion relationship between the vehicle body plane coordinate system and the fisheye lens coordinate system, as shown in equation (33):

[0254] p imu =E imu→V E cam→V p cam

[0255] In the equation, E imu→V is an extrinsic parameter for converting a current pose angle r imu (p, r, h) of the vehicle body to the fisheye lens coordinate system, wherein p is a value of a pitch angle (pitch), r is a value of a roll angle (roll), and h is a value of a heading angle (heading). E imu→V is an extrinsic parameter for converting a current pose angle r imu (p, r, h) of the vehicle body to an angle change parameter and a position change corresponding to each fisheye lens coordinate system, i.e., a rotation matrix R ium→V and a translation transformation matrix T ium→V .

[0256] According to the preliminary extrinsic parameter adjustment of the fisheye lens coordinate system based on the current pose angle r imu (p, r, h) of the vehicle body, an extrinsic parameter of the fisheye lens for a standard pose can be obtained, and a conversion relationship between a fisheye lens cam1 coordinate system and the IMU coordinate system is as follows:

[0257] p imu =E imu→V E cam1→V p cam1 .

[0258] A conversion relationship between a fisheye lens cam2 coordinate system and the IMU coordinate system is as follows:

[0259] pimu =E imu→V E cam2→V p cam2 .

[0260] The transformation relationship between the fisheye lens cam3 coordinate system and the vehicle body coordinate system is as follows:

[0261] p imu =E imu→V E cam3→V p cam3 .

[0262] The transformation relationship between the fisheye lens cam4 coordinate system and the IMU coordinate system is as follows:

[0263] p imu =E imu→V E cam4→V p cam4 .

[0264] Based on this, the extrinsic parameters E = Ea of the four-way fisheye lens after removing the current calibration pose can be obtained. imu→V E cam4→V This refers to the external parameters of the four-way fisheye lens when the vehicle is in a standard position.

[0265] In one feasible implementation, during vehicle operation, the extrinsic parameters of multiple fisheye lenses can be dynamically adjusted based on the IMU information collected in real time by the inertial measurement unit. The IMU information includes the values ​​of the vehicle body's roll angle, pitch angle, and yaw angle, ensuring high precision and stability.

[0266] The fisheye lens extrinsic parameter calibration method provided in the embodiments of this application collects IMU information output by IMU sensor, combines IMU information to obtain the accurate relative position relationship between the current state of the vehicle and the ground, and fine-tunes the obtained preliminary fisheye extrinsic parameters to remove the influence of extrinsic parameters corresponding to the vehicle attitude.

[0267] Example 1

[0268] Figure 7 This is a flowchart illustrating the calibration method for the extrinsic parameters of a fisheye lens provided in Embodiment 1 of this application. This embodiment uses target points to determine the vehicle-to-ground relationship, such as... Figure 7 As shown, it includes:

[0269] S71, when a new car is in the parking lot, it enters a calibration space, and the manufacturer determines the car's parking position using images or lasers.

[0270] S72 projects four checkerboard-shaped calibration boards and two target points onto the ground. For example... Figure 8As shown, two black dots are the projected circular rear wheel targets, and four identical checkerboard calibration boards U1, U2, U3 and U4 are set as standard reference objects at four positions of the left rear, left front, right rear and right front of the vehicle body.

[0271] S73, the vehicle body to be calibrated acquires four fisheye camera images, and the checkerboard calibration board is detected by the four fisheye cameras cam1, cam2, cam3 and cam4 respectively to obtain the coordinates of the specified position points in the fisheye camera coordinate system. For details, refer to step S63, which will not be repeated here.

[0272] S74, the vehicle body to be calibrated acquires the information output by the current IMU sensor.

[0273] S75, according to the coordinates of the specified position points of the checkerboard calibration board in the fisheye camera coordinate system obtained in step S73, the positional relationship between two adjacent fisheye cameras is determined, and then the conversion relationship between the four fisheye camera coordinate systems is obtained. For details, refer to step S64, which will not be repeated here.

[0274] S76, the projection of the wheel target on the ground is detected to determine the conversion relationship between the fisheye camera coordinate system and the vehicle body plane coordinate system. For details, refer to step S65, which will not be repeated here.

[0275] S77, the extrinsic parameters of the four fisheye cameras are preliminarily determined according to the conversion relationship between the fisheye camera coordinate system and the vehicle body plane coordinate system.

[0276] S78, the vehicle body attitude is calculated according to the current IMU information, and the vehicle body attitude at the time of calibration is obtained after processing the IMU information.

[0277] S79, the extrinsic parameters of the fisheye camera are obtained by combining the preliminarily determined extrinsic parameters of the fisheye camera and the vehicle body attitude at the time of calibration. For details, refer to step S66, which will not be repeated here.

[0278] After the extrinsic parameters of the fisheye camera are calibrated using the above implementation method, the extrinsic parameters of the fisheye camera can be used for subsequent assisted driving and intelligent driving of the intelligent vehicle.

[0279] After the extrinsic parameters of the fisheye camera are calibrated using the above implementation method, the extrinsic parameters of the fisheye camera can be used for subsequent assisted driving and intelligent driving of the intelligent vehicle.

[0280] Different from the traditional method of determining the spatial 3D point position coordinates using a total station and combining the 2D point coordinates in the fisheye image for fisheye lens extrinsic parameter calibration, the embodiment 1 method of the present application adopts a pure vision method, which reduces the time required for 3D dotting, reduces the error sources, and improves the precision. Compared with the prior art, the embodiment 1 method of the present application does not need to use a vehicle-mounted platform to move the vehicle to the required specified position, can simply calibrate the environment arrangement according to the current position of the vehicle, has low cost, is fast to place, and can be adjusted accordingly according to the environment, such as placing the wheel target point and the calibration indicator through projection in a relatively mature environment in a parking lot, and using a wheel clamping device to fix the vehicle position in a simple space such as a car wash, etc. can also use the method to determine the wheel and ground positions, or manually placing using printing in a space arranged by the user can also calibrate the extrinsic parameters.

[0281] Embodiment 2

[0282] Figure 9 The flowchart of the fisheye lens extrinsic parameter calibration method provided for the embodiment 2 of the present application. The embodiment uses an IMU sensor to calibrate the dynamic fisheye lens extrinsic parameter, and calibrates when the user finds that the fisheye lens extrinsic parameter of the vehicle needs to be corrected again, such as Figure 9 as shown, including:

[0283] S91, drive the vehicle into a nearby simple calibration space, such as a car wash.

[0284] S92, use a wheel clamping device to fix the position of the rear wheel of the vehicle.

[0285] S93, set four same chessboard calibration boards U1, U2, U3 and U4 as standard reference objects at four positions of the left rear, left front, right rear and right front of the vehicle.

[0286] S94, acquire images captured by four fisheye lenses, and perform point detection on the chessboard calibration boards through the four fisheye lenses cam1, cam2, cam3 and cam4 to obtain the coordinates of the specified position points in the fisheye lens coordinate system. For details, refer to step S63, which will not be repeated here.

[0287] S95, determine the positional relationship of two adjacent lenses according to the coordinates of the specified position points of the chessboard calibration boards in the fisheye lens coordinate system obtained in step S94, and then obtain the conversion relationship between the four fisheye lens coordinate systems. For details, refer to step S64, which will not be repeated here.

[0288] S96, acquire the information output by the current IMU sensor while calibrating the vehicle.

[0289] S97, according to the wheel position of the card, the position of the left and right rear wheel target points is determined, and the conversion relationship between the fisheye lens coordinate system and the vehicle body plane coordinate system is determined according to the left and right target point coordinates in the fisheye lens image, and the preliminary extrinsic parameter is obtained. For details, refer to step S65, which will not be repeated here.

[0290] S98, the vehicle body posture is calculated according to the current IMU information, and the posture of the vehicle body at the time of calibration is obtained after processing the IMU information.

[0291] In an implementable embodiment, the IMU information is angular acceleration information, and the inclination angle of the current vehicle body relative to the ground, i.e., the posture of the vehicle body at the time of calibration, can be obtained through the angular acceleration information. The current posture of the vehicle body is converted to the angle change corresponding to each fisheye lens, and the fisheye lens extrinsic parameter after removing the current calibration posture, i.e., the fisheye lens extrinsic parameter at the standard posture, can be obtained in combination with the preliminary fisheye lens extrinsic parameter.

[0292] Embodiment 2 of the present application provides a method different from embodiment 1 in that the determination of the wheel target point, the simplicity of the calibration space, and the use of different forms of target points. The method proposed in the present application can be used to determine the vehicle-ground position relationship, and the dependence of calibration on the space environment is eliminated.

[0293] The method provided by the embodiment of the present application only needs to use the calibration board and the wheel target to obtain the vehicle-ground relationship through coordinate conversion to perform extrinsic parameter calibration, improves the method of fixing the vehicle position used in the prior art, such as the wheel fixer, the method provided by the embodiment of the present application estimates the current vehicle posture through IMU information, and adjusts the fisheye lens extrinsic parameter to obtain dynamic parameters using the current vehicle posture information, thereby ensuring high precision and stability.

[0294] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0295] Moreover, various aspects or features of the embodiments disclosed herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. For example, computer-readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips, etc.), optical disks (e.g., compact disk (CD), digital versatile disk (DVD), etc.), smart cards, and flash memory devices (e.g., EPROM, card, stick, or key drive, etc.). Additionally, various storage media described herein can represent one or more devices and / or other machine-readable media for storing information. The term "machine- readable medium" can include, without being limited to, wireless channels and various other media capable of storing, containing, and / or carrying instruction(s) and / or data.

[0296] It should be understood that the sequence size of the above-mentioned procedures does not mean the execution sequence, and the execution sequence of the procedures should be determined according to the functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments.

[0297] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0298] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-mentioned device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0299] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0300] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0301] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A method for calibrating the extrinsic parameters of a fisheye lens, used to calibrate the extrinsic parameters of multiple fisheye lenses mounted on a vehicle body, wherein the multiple fisheye lenses are respectively disposed at the front, rear, left, and right parts of the vehicle body, and the multiple fisheye lenses include a first fisheye lens, a second fisheye lens, a third fisheye lens, and a fourth fisheye lens; characterized in that, The method comprises: determining a conversion relationship between a plurality of fisheye lens coordinate systems according to images of a plurality of checkerboard calibration boards captured by the plurality of fisheye lenses; the plurality of fisheye lens coordinate systems comprise fisheye lens coordinate systems of first, second, third and fourth fisheye lenses; the plurality of checkerboard calibration boards are placed on the ground at left rear, left front, right rear and right front corner points of a vehicle body; determining an extrinsic parameter of each fisheye lens in the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and images of first and second wheel targets in the plurality of fisheye lenses; the first and second wheel targets are placed respectively close to left and right rear wheels of the vehicle body.

2. The method of claim 1, wherein, The first fisheye lens in the plurality of fisheye lenses is arranged at one of front, rear, left and right parts of the vehicle body, and the second fisheye lens is arranged at one of the front, rear, left and right parts of the vehicle body and adjacent to the first fisheye lens, and the conversion relationship between the plurality of fisheye lens coordinate systems is determined according to images of a plurality of checkerboard calibration boards captured by the plurality of fisheye lenses, comprising: obtaining a first image captured by the first fisheye lens, and determining a first fisheye lens coordinate system according to a first checkerboard calibration board on the first image; the first checkerboard calibration board is one of the plurality of checkerboard calibration boards; obtaining a second image captured by the second fisheye lens, and determining a second fisheye lens coordinate system according to the first checkerboard calibration board on the second image; determining a first conversion relationship according to coordinates of a plurality of pairs of same position points in the first checkerboard calibration board of the first image and the first checkerboard calibration board of the second image; the first conversion relationship is a conversion relationship between the first fisheye lens coordinate system and the second fisheye lens coordinate system.

3. The method of claim 2, wherein, The first conversion relationship is determined according to coordinates of a plurality of pairs of same position points in the first checkerboard calibration board of the first image and the first checkerboard calibration board of the second image, comprising: calculating a rotation matrix and a translation transformation matrix between the first fisheye lens coordinate system and the second fisheye lens coordinate system according to coordinates of a plurality of pairs of same position points in the first checkerboard calibration board of the first image and the first checkerboard calibration board of the second image, and determining the first conversion relationship.

4. The method according to claim 2 or 3, characterized in that, The third fisheye lens in the plurality of fisheye lenses is arranged at one of front, rear, left and right parts of the vehicle body, adjacent to the first fisheye lens and opposite to the second fisheye lens, and the conversion relationship between the plurality of fisheye lens coordinate systems is determined according to images of a plurality of checkerboard calibration boards captured by the plurality of fisheye lenses, comprising: obtaining a third image captured by the first fisheye lens, the third image being an image comprising a third checkerboard calibration board; the third checkerboard calibration board is one of the plurality of checkerboard calibration boards; obtaining a fourth image captured by the third fisheye lens, and determining a third fisheye lens coordinate system according to the third checkerboard calibration board on the fourth image; determining a second conversion relationship according to coordinates of a plurality of pairs of same position points on the third checkerboard calibration board in the third image and the fourth image; the second conversion relationship is a conversion relationship between the third fisheye lens coordinate system and the first fisheye lens coordinate system. Determine a third conversion relationship according to the second conversion relationship and the first conversion relationship, the third conversion relationship being a conversion relationship between the third fisheye lens coordinate system and the second fisheye lens coordinate system.

5. The method of claim 1, wherein, The method further comprises: Determine a fifth conversion relationship according to images of a plurality of checkerboard calibration boards captured by the plurality of fisheye lenses, the fifth conversion relationship being a conversion relationship between each fisheye lens coordinate system of the plurality of fisheye lens coordinate systems and a corresponding checkerboard calibration board coordinate system; the checkerboard calibration board coordinate system being a coordinate system determined by the checkerboard calibration board; Determine a sixth conversion relationship according to the conversion relationships between the plurality of fisheye lens coordinate systems and the images of the first and second wheel targets in the plurality of fisheye lenses, the sixth conversion relationship being a conversion relationship between a vehicle body plane coordinate system and the plurality of checkerboard calibration board coordinate systems; the vehicle body plane coordinate system being a coordinate system with the center points of the left and right rear wheels of the vehicle body as the center, the direction from the second wheel target to the first wheel target as the positive direction of the Y axis, and the front direction of the vehicle as the positive direction of the X axis; Determine a seventh conversion relationship according to the fifth conversion relationship and the sixth conversion relationship, the seventh conversion relationship being a conversion relationship between each fisheye lens coordinate system and the vehicle body plane coordinate system; Determine the extrinsic parameters of each fisheye lens of the plurality of fisheye lenses according to the seventh conversion relationship.

6. The method of claim 5, wherein, The method further comprises: Obtain a fifth image captured by each fisheye lens of the plurality of fisheye lenses, the fifth image being an image including one checkerboard calibration board of the plurality of checkerboard calibration boards; Obtain a fifth conversion relationship according to the fifth image and the coordinates of a plurality of pairs of same-position points on the corresponding one checkerboard calibration board, the fifth conversion relationship including a scale matrix, a rotation matrix, and a translation transformation matrix between each fisheye lens coordinate system and the corresponding checkerboard calibration board coordinate system.

7. The method according to claim 5 or 6, characterized in that, The method further comprises: Obtain sixth and seventh images captured by the second and third fisheye lenses of the plurality of fisheye lenses; the sixth image being an image including the first wheel target captured by the second fisheye lens, and the seventh image being an image including the second wheel target captured by the third fisheye lens; Determine the coordinates of the midpoint between the first and second wheel targets in the first checkerboard calibration board coordinate system or the third checkerboard calibration board coordinate system according to the sixth and seventh images and the conversion relationship between the second and third fisheye lens coordinate systems in the conversion relationships between the plurality of fisheye lens coordinate systems. According to the coordinates of the midpoint between the first and second wheel targets in the first or third checkerboard calibration plate coordinate system, the coordinates of the first and second wheel targets in the second or third checkerboard calibration plate coordinate system, and the direction in which the second wheel target points to the first wheel target, a sixth conversion relationship is determined; the second checkerboard calibration plate is one of the plurality of checkerboard calibration plates.

8. The method of claim 5, wherein, According to the fifth conversion relationship and the sixth conversion relationship, a seventh conversion relationship is determined, including: according to the fifth conversion relationship between the second fisheye lens coordinate system and the corresponding first checkerboard calibration plate coordinate system, and the sixth conversion relationship between the first checkerboard calibration plate coordinate system and the vehicle body plane coordinate system, a seventh conversion relationship between the second fisheye lens coordinate system and the vehicle body plane coordinate system is determined.

9. The method of claim 5, wherein, According to the fifth conversion relationship and the sixth conversion relationship, a seventh conversion relationship is determined, including: According to the fifth conversion relationship between the third fisheye lens coordinate system and the corresponding third checkerboard calibration plate coordinate system, and the sixth conversion relationship between the third checkerboard calibration plate coordinate system and the vehicle body plane coordinate system, a seventh conversion relationship between the third fisheye lens coordinate system and the vehicle body plane coordinate system is determined.

10. The method of claim 5, wherein, According to the fifth conversion relationship and the sixth conversion relationship, a seventh conversion relationship is determined, including: According to the conversion relationship between the first fisheye lens coordinate system and the third fisheye lens coordinate system, the fifth conversion relationship between the third fisheye lens coordinate system and the corresponding third checkerboard calibration plate coordinate system, and the sixth conversion relationship between the third checkerboard calibration plate coordinate system and the vehicle body plane coordinate system, a seventh conversion relationship between the first fisheye lens coordinate system and the vehicle body plane coordinate system is determined.

11. The method according to one of claims 5-10, characterized in that, According to the seventh conversion relationship, the extrinsic parameters of each fisheye lens in the plurality of fisheye lenses are determined, including: The seventh conversion relationship includes a rotation matrix and a translation transformation matrix between each fisheye lens coordinate system and the vehicle body plane coordinate system; According to the rotation matrix and the translation transformation matrix between the first, second, third, and fourth fisheye lens coordinate systems and the vehicle body plane coordinate system, the extrinsic parameters of the first, second, third, and fourth fisheye lens coordinate systems are determined.

12. The method of claims 1-11, wherein, An inertial measurement unit is arranged on the vehicle body chassis, which is used to determine the values of the roll angle, pitch angle, and heading angle of the vehicle body; the method further includes: Obtaining the values of the roll angle, pitch angle, and heading angle output by the inertial measurement unit; Determining an IMU coordinate system according to the values of the roll angle, pitch angle, and heading angle; the values of the roll angle, pitch angle, and heading angle indicate the deflection angles of the Y-axis, X-axis, and Z-axis of the IMU coordinate system and the vehicle body plane coordinate system; Determining the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system; Adjusting the extrinsic parameters of the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system.

13. The method of claim 12, wherein, The determination of the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system includes: obtaining a rotation matrix of the IMU coordinate system and the vehicle body plane coordinate system according to the values of the pitch angle, the roll angle and the heading angle; determining a conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system according to the rotation matrix of the IMU coordinate system and the vehicle body plane coordinate system and a conversion relationship between the vehicle body plane coordinate system and the fisheye lens coordinate system.

14. The method of claim 12, wherein, The adjusting the extrinsic parameters of the plurality of fisheye lenses according to the conversion relationship between the plurality of fisheye lens coordinate systems and the IMU coordinate system comprises: dynamically adjusting the extrinsic parameters of the plurality of fisheye lenses according to IMU information collected by the inertial measurement unit in real time during vehicle driving, the IMU information comprising values of a roll angle, a pitch angle and a heading angle of the vehicle body.

Images