Vehicle-mounted camera calibration method and device, equipment, storage medium and program product

Abstract

The invention relates to a calibration method and device of a vehicle-mounted camera, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring first images respectively shot by a plurality of fixed cameras in a calibration room; a vehicle is parked in the calibration room, and vehicle body angular points exist on the surface of the vehicle; vehicle body angular points in the first images are detected, and coordinates of target angular points in a room coordinate system are determined according to detection results; the target angular point is a vehicle body angular point co-occurring in at least two first images; according to the coordinate of the target angular point in a vehicle body coordinate system and the coordinate of the target angular point in a room coordinate system, determining a conversion relation between the room coordinate system and the vehicle body coordinate system; and calibrating external parameters of the vehicle-mounted camera of the vehicle relative to a vehicle body coordinate system according to the conversion relation. By adopting the method, the calibration accuracy of the vehicle-mounted camera can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a calibration method, device, electronic device, computer-readable storage medium and computer program product for a vehicle-mounted camera. Background Art

[0002] In autonomous driving applications, on-board cameras generally need to go through the external parameter calibration stage. The calibration at the end of production line (EOL) is usually carried out in a specific calibration room, and the calibration results are the external parameters (position and attitude) of each on-board camera in the calibration room coordinate system.

[0003] In order to realize the external parameter calibration of the vehicle-mounted camera, in the traditional way, a vehicle limiting device (such as a centering platform) is usually set up in the production line calibration room to facilitate the positioning, limiting or locking of the vehicle. These devices use various mechanical pressures (air pressure or hydraulic pressure, etc.) to move the vehicle as much as possible and fix it to a uniform position, so that the calibration room coordinate system and the vehicle coordinate system can be fixedly converted.

[0004] However, due to the effects of pressure and friction, the vehicle is often not in a natural equilibrium state during the process of being fixed. It has a different posture from when the vehicle is statically parked on a flat road. Therefore, the external parameters of the on-board camera and the calibration room coordinate system calibrated by EOL are not equivalent to the external parameters of the vehicle naturally balanced on a flat road under the same load distribution. Due to the uncontrollability of pressure and friction, the posture difference itself is relatively random (regardless of whether it is between different vehicles or the same vehicle is calibrated multiple times). Therefore, the traditional calibration method has the problem of inaccurate calibration. Summary of the invention

[0005] Based on this, it is necessary to provide a calibration method, device, electronic device, computer-readable storage medium and computer program product for a vehicle-mounted camera that can improve the calibration accuracy of the vehicle-mounted camera in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for calibrating a vehicle-mounted camera, comprising:

[0007] Acquire first images taken by multiple fixed cameras in a calibration room; a vehicle is parked in the calibration room, and body corners exist on the surface of the vehicle;

[0008] Detecting the corner points of the vehicle body in each of the first images, and determining the coordinates of the target corner points in the room coordinate system according to the detection results; the target corner points are the corner points of the vehicle body that co-appear in at least two of the first images;

[0009] Determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system;

[0010] The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated according to the conversion relationship.

[0011] In a second aspect, the present application also provides a calibration device for a vehicle-mounted camera, comprising:

[0012] An acquisition module is used to acquire first images taken by multiple fixed cameras in a calibration room, wherein a vehicle is parked in the calibration room and body corners exist on the surface of the vehicle;

[0013] A first determination module, used to detect the corner points of the vehicle body in each of the first images, and determine the coordinates of the target corner point in the room coordinate system according to the detection results; the target corner point is a corner point of the vehicle body that co-occurs in at least two first images;

[0014] A second determination module, configured to determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system;

[0015] The first calibration module is used to calibrate the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the conversion relationship.

[0016] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Acquire first images taken by multiple fixed cameras in a calibration room; a vehicle is parked in the calibration room, and body corners exist on the surface of the vehicle;

[0018] Detecting the corner points of the vehicle body in each of the first images, and determining the coordinates of the target corner points in the room coordinate system according to the detection results; the target corner points are the corner points of the vehicle body that co-appear in at least two of the first images;

[0019] Determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system;

[0020] The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated according to the conversion relationship.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0022] Acquire first images taken by multiple fixed cameras in a calibration room; a vehicle is parked in the calibration room, and body corners exist on the surface of the vehicle;

[0023] Detecting the corner points of the vehicle body in each of the first images, and determining the coordinates of the target corner points in the room coordinate system according to the detection results; the target corner points are the corner points of the vehicle body that co-appear in at least two of the first images;

[0024] Determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system;

[0025] The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated according to the conversion relationship.

[0026] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0027] Acquire first images taken by multiple fixed cameras in a calibration room; a vehicle is parked in the calibration room, and body corners exist on the surface of the vehicle;

[0028] Detecting the corner points of the vehicle body in each of the first images, and determining the coordinates of the target corner points in the room coordinate system according to the detection results; the target corner points are the corner points of the vehicle body that co-appear in at least two of the first images;

[0029] Determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system;

[0030] The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated according to the conversion relationship.

[0031] The calibration method, device, electronic device, computer-readable storage medium and computer program product of the vehicle-mounted camera are provided in the calibration room with a camera with a fixed position and posture, i.e., a fixed camera. The vehicle is photographed by multiple fixed cameras to obtain corresponding first images, and the vehicle body corner points in each first image are detected, so that the three-dimensional coordinates of the target corner points in the three-dimensional space, i.e., the coordinates of the target corner points in the room coordinate system, can be accurately determined according to the pixel coordinates of the vehicle body corner points (i.e., the target corner points) co-occurring in at least two first images. In this way, the conversion relationship between the vehicle body coordinate system and the room coordinate system can be obtained according to the coordinates of the multiple target corner points in the vehicle body coordinate system and the coordinates in the room coordinate system. The conversion relationship can represent the position and posture of the vehicle in the calibration room, so the calibration of the vehicle-mounted camera relative to the vehicle body coordinate system can be realized by using the conversion relationship. This calibration method can avoid the calibration error caused by the change of the vehicle posture when the vehicle is fixed by the vehicle limit device, and improve the accuracy of the external parameter calibration of the vehicle-mounted camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] FIG1 is a schematic diagram of a calibration room in one embodiment;

[0034] FIG2 is an application scenario diagram of a calibration method for a vehicle-mounted camera in one embodiment;

[0035] FIG3 is a schematic diagram of a fixed camera set in a calibration room in one embodiment;

[0036] Figure 4 is a flow chart of a calibration method for a vehicle-mounted camera in one embodiment;

[0037] FIG5 (a) and FIG5 (b) are schematic diagrams showing the distribution of corner points of a vehicle body in one embodiment;

[0038] Figure 6 is a flow chart of a calibration method for a vehicle-mounted camera in yet another embodiment;

[0039] Figure 7 is a flow chart of a calibration method for a vehicle-mounted camera in another embodiment;

[0040] Figure 8 is a flow chart of a calibration method for a vehicle-mounted camera in yet another embodiment;

[0041] Fig. 9 is a flow chart of a calibration method for a vehicle-mounted camera in yet another embodiment;

[0042] Fig.10 is a structural block diagram of a calibration device for a vehicle-mounted camera in one embodiment;

[0043] Fig.11 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In various application scenarios of vehicles, vehicle cameras are often used, and before they are put into use, camera parameters usually need to be calibrated. For the external parameter calibration of vehicle cameras, on the production line, it is usually necessary to fix the vehicle in a fixed position in the calibration room. Some detectable targets are pre-set in the calibration room, and there are many detectable and identifiable corner points in the targets. The physical positions of these corner points (relative to the room coordinate system measured in the calibration room) are known in advance. In general, considering the different installation angles of different vehicle cameras, some targets will be set on the walls and / or floors of the calibration room captured by all vehicle cameras for calculation. For example, the ground can adopt a chessboard pattern, and can be equipped with positioning modes of different sizes or different colors or special patterns. In addition, some targets above the ground will be equipped, which can be in the form of chessboards, dots, or special QR codes. The feature points (i.e., target corner points) in all these targets are measured in advance in the room coordinate system of the calibration room, and are easy to be detected and accurately located in camera imaging. For example, an implementation example is shown in Figure 1. The colored dots on the ground represent positioning marks, and special marks of other styles can also be selected for identification. The vertical target perpendicular to the ground is a black and white checkerboard, and in practice, it can also be selected as a self-encoded mark (for example, AprilTag, a visual reference tag).

[0046] The coordinates of the targets in the calibration room are all established in the room coordinate system. In the traditional way, the vehicle must be parked at a fixed position during calibration to successfully convert these coordinates into the body coordinate system. Therefore, in addition to the target setting, the calibration room generally has a vehicle limit device (such as a centering platform), which generally uses mechanical pressure to squeeze the wheel to fix it in a certain position.

[0047] However, in actual applications, the plane of the centering platform itself is not necessarily coplanar with the ground, and there will be a fixed error after installation and locking. Although the fixed error will be debugged to meet the accuracy requirements as much as possible, it will change again after each change, maintenance, and adjustment. Every time a vehicle enters the centering platform, the synchronization link of the centering platform aligns and centers the front axle of the vehicle by applying pressure to the front wheels of the vehicle. When pressure is applied to the tires, the vehicle may be subjected to different pressures and friction, and the body of the vehicle may change its posture when the vehicle is fixed. In this way, the posture parameters of the on-board camera relative to the calibration room are no longer the posture of the on-board camera relative to the body of the vehicle, which affects the posture and geometric calculations during actual static driving.

[0048] In general, whether due to the non-parallelism between the centering platform and the ground or the random posture changes of the vehicle body caused by pressure, it will eventually lead to a deviation between the reference coordinate system of the vehicle camera calibration (the room coordinate system of the calibration room) and the vehicle body coordinate system, affecting the accuracy of the vehicle camera external parameter calibration.

[0049] The vehicle-mounted camera calibration method provided in the present application can avoid the errors caused by the above reasons and greatly improve the accuracy of the vehicle-mounted camera extrinsic parameter calibration.

[0050] In addition, in another related technology, a method for calibrating the external parameters of a vehicle-mounted camera relative to a vehicle body coordinate system is provided, which is as follows: ground markers are laid in the calibration area, an external measurement camera is set above the calibration area, and a calibration reference object is set on the roof of the vehicle to be calibrated. The camera parameters of the external measurement camera relative to the ground coordinate system are calibrated according to the ground markers, and the camera parameters of the external measurement camera relative to the vehicle body coordinate system are calibrated according to the calibration reference object. Then, the transformation relationship between the ground coordinate system and the vehicle body coordinate system is determined according to the two camera parameters. The parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated based on the transformation relationship.

[0051] In the above-mentioned related technologies, a calibration reference object is additionally set up, and the coordinates of the calibration reference object in the vehicle body coordinate system are measured, which inevitably introduces errors; in addition, in the related technologies, the transformation relationship between the two spatial coordinate systems is determined only by a single external measurement camera, which places high requirements on the placement accuracy and measurement accuracy of the calibration reference object.

[0052] The vehicle-mounted camera calibration method provided by the present application does not require the placement of additional reference objects. It directly uses the special body corner points (i.e., target corner points) on the surface of the vehicle to achieve three-dimensional measurement of the target corner points through multiple fixed cameras at different angles, that is, to determine the coordinates of the target corner points in the room coordinate system. Then, based on the vehicle design data, the coordinates of the target corner points in the body coordinate system are obtained, which does not introduce new errors. In this way, the conversion relationship between the room coordinate system and the body coordinate system can be accurately determined using the three-dimensional coordinates of the target corner points in the room coordinate system and the three-dimensional coordinates in the body coordinate system, and then when the external parameters of the vehicle-mounted camera are determined using the conversion relationship, the accuracy of the external parameter calibration can be greatly improved. The present application uses multiple fixed cameras and multiple body corner points around the vehicle to achieve external parameter calibration of the vehicle-mounted camera, and has strong robustness to measurement errors and calculation errors.

[0053] It should be noted that the "multiple" mentioned in this application refers to at least two unless otherwise specified. Before using multiple fixed cameras to calibrate the vehicle-mounted camera, this application will perform external parameter calibration on each of the multiple fixed cameras, that is, with the help of targets set in the calibration room, calibrate the external parameters of each of the multiple fixed cameras relative to the room coordinate system. The specific calibration process will be described in detail in the following embodiments. It can be understood that the calibration of multiple fixed cameras can be carried out sequentially or synchronously and in parallel, and the embodiments of this application are not limited to this.

[0054] The following is a detailed description of the vehicle camera determination method provided by this application:

[0055] The calibration method of the vehicle camera provided in the embodiment of the present application can be applied to the application environment shown in FIG2. Among them, multiple fixed cameras 202 and vehicle cameras 204 communicate with the electronic device 206 through the network. The data storage system can store the data that the electronic device 206 needs to process. The data storage system can be integrated on the electronic device 206, or it can be placed on the cloud or other network servers. The electronic device 206 obtains the first image taken by each of the multiple fixed cameras 202 in the calibration room; there is a vehicle parked in the calibration room, and there are detectable and identifiable body corners on the surface of the vehicle. These body corners can be inherent corners of the body shape, or pattern corners sprayed or posted at fixed positions on the body appearance; the body corners in each first image are detected, and the coordinates of the target corners in the room coordinate system are determined according to the detection results; the target corners are the body corners that co-occur in at least two first images; according to the coordinates of the target corners in the body coordinate system and the coordinates in the room coordinate system, the conversion relationship between the room coordinate system and the body coordinate system is determined; according to the conversion relationship, the external parameters of the vehicle-mounted camera 204 of the vehicle relative to the body coordinate system are calibrated.

[0056] Among them, the electronic device 206 can be, but is not limited to, a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0057] In some application scenarios, the electronic device may be an independent device, and in other embodiments, the electronic device may be a vehicle-mounted terminal. It is understood that the present application does not limit the external form of the electronic device, as long as it has data processing and data storage capabilities.

[0058] Before describing in detail the calibration method of the vehicle-mounted camera provided in this application, the calibration room used in this application and the setting method of the fixed camera in the calibration room are introduced:

[0059] Please refer to FIG. 3, which is an example of a calibration room with multiple fixed cameras installed in one embodiment. Among them, some targets are set on each wall and / or floor of the calibration room for calculation. It can be understood that in other settings, ground targets may not be set, or wall targets may be reduced. The key is to provide at least two targets in different planes to improve the accuracy of camera calibration.

[0060] In this application, several (at least two) fixed cameras are placed at fixed positions in the calibration room. The so-called fixed means that both the camera installation position and the camera focus state are fixed, that is, the geometric relationship between these cameras and the calibration room is guaranteed to be fixed. For the sake of convenience, the cameras installed in this way are referred to as fixed cameras in this application. There are two principles for selecting the number and position (angle) of fixed cameras: (1) Before the vehicle enters the calibration room, each fixed camera can capture at least three non-collinear detectable target corner points of various types; (2) After the vehicle stops in the calibration room, there are at least two fixed cameras that can detect at least two detectable feature points (i.e., body corner points) on the exterior of the vehicle body. Increasing the number of fixed cameras and the number of available body corner points help the system increase its robustness against measurement errors and calculation errors.

[0061] In an exemplary embodiment, Figure 4 As shown, a calibration method for a vehicle-mounted camera is provided, and the method is applied to the electronic device in FIG. 2 as an example for description, including the following steps 402 to 408. Among them:

[0062] Step 402, obtaining first images taken by multiple fixed cameras in the calibration room; there are vehicles parked in the calibration room, and there are body corners on the surface of the vehicle.

[0063] The fixed camera is a camera installed in the calibration room and having a fixed geometric relationship with the calibration room. It should be noted that before executing step 402, the electronic device may pre-calibrate the camera parameters for each of the multiple fixed cameras. The camera parameters of the fixed camera include camera intrinsic parameters and / or camera extrinsic parameters.

[0064] The body corners are key points of the body appearance, such as the reflector corners, upper and lower corners of the window, the outermost corners of the bumper, door handles, window corners, and corners of the exterior lights, etc. All significant corners of the body appearance can be selected.

[0065] In the specific implementation, the vehicle to be calibrated will be parked in the calibration room in advance. For example, the vehicle can be parked at any position in the calibration room, or the vehicle can be fixed at a fixed position by a vehicle limiting device, which is not limited in the embodiment of the present application.

[0066] Then, all the fixed cameras installed in the calibration room take a real scene picture, which is called the first image to distinguish it from the image in the following text. It can be understood that the first image is an image taken by the fixed camera for the vehicle.

[0067] Step 404 , detecting the corner points of the vehicle body in each first image, and determining the coordinates of the target corner point in the room coordinate system according to the detection result; the target corner point is a corner point of the vehicle body that co-occurs in at least two first images.

[0068] The room coordinate system is a measurement coordinate system established in the room, which can be a Euclidean coordinate system with a certain point in the calibration room as the origin. express.

[0069] Specifically, the electronic device may perform image recognition on each first image to detect whether a corner point of the vehicle body appears therein. Exemplarily, the electronic device may perform corner point detection in the following manners: corner point detection based on grayscale images, corner point detection based on binary images, or corner point detection based on contour curves. Among them, the corner point detection based on grayscale images is specifically to detect corner points by analyzing the grayscale information of the image; the corner point detection based on binary images is specifically to convert the first image into a binary image and analyze the black and white information of the image to detect the corner points; the corner point detection based on contour curves is specifically to detect corner points by analyzing the contour curve of the image. It can be understood that the detection method of the corner points of the vehicle body is not limited to the above method, and can also be other detection methods, which are not limited in the embodiments of the present application.

[0070] Furthermore, after the electronic device detects each vehicle body corner point appearing in the first image, the pixel coordinates of the vehicle body corner point in the image are obtained. In order to successfully calculate the spatial coordinates of the target corner point in the room coordinate system, the electronic device may filter out the vehicle body corner points that co-appear in at least two first images at the same time as the target corner points for subsequent use. If there are many vehicle body corner points filtered out, in order to improve the calculation efficiency, the electronic device may also perform a secondary screening, such as retaining the vehicle body corner points whose co-occurrence number exceeds a threshold as the target corner points.

[0071] Furthermore, for each target corner point, the electronic device can determine its coordinates in the room coordinate system according to the pixel coordinates of the target corner point in different first images and in combination with the camera parameters of the fixed camera.

[0072] Step 406 , determining a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system.

[0073] In some embodiments, the electronic device obtains the coordinates of the target corner point in the body coordinate system, and then determines the conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system. The conversion relationship can be represented by a rotation matrix and a translation vector.

[0074] In some embodiments, the electronic device may obtain the coordinates of the target corner point in the vehicle body coordinate system according to vehicle design data, such as CAD (Computer Aided Design) data during vehicle design.

[0075] The coordinates of the target corner point in the room coordinate system are marked as Based on the CAD data of the vehicle design, the “ideal” 3D coordinates of these target corner points can also be obtained, denoted as Next, we need to find a transformation relationship to describe the relationship between the body coordinate system and the room coordinate system. The rigid body transformation between two Euclidean coordinate systems has six degrees of freedom, which are three rotation angles and three translation components. Assume that the change from the room coordinate system to the body coordinate system is:

[0076]

[0077] in and They are a 3x3 rotation matrix and a 3x1 translation vector. The three rotation components Generate. It can be obtained by minimizing the distance between the transformed coordinates of all target corner points from the room coordinate system to the body coordinate system and the actual coordinates in the body coordinate system, or by minimizing the distance between the transformed coordinates of all target corner points from the body coordinate system to the room coordinate system and the actual coordinates in the room coordinate system. For example, the following formula can be used for minimization to obtain the rotation matrix and translation vector, that is, the conversion relationship between the room coordinate system and the body coordinate system:

[0078]

[0079] It should be noted that there are various optimization methods in actual use, for example, the least squares method can be used.

[0080] In some application scenarios, when it is not sensitive to position errors, the displacement changes of the two coordinate systems can also be ignored, that is, .

[0081] Step 408, calibrate the external parameters of the vehicle's onboard camera relative to the vehicle body coordinate system according to the conversion relationship.

[0082] Specifically, after obtaining the conversion relationship between the room coordinate system and the vehicle body coordinate system, the electronic device calibrates the external parameters of each vehicle-mounted camera of the vehicle based on the conversion relationship. The electronic device can calibrate the external parameters of each vehicle-mounted camera relative to the vehicle body coordinate system one by one, or can calibrate the external parameters of each vehicle-mounted camera relative to the vehicle body coordinate system in parallel and synchronously, which is not limited in the embodiments of the present application.

[0083] For each vehicle-mounted camera, the electronic device can use the conversion relationship between the room coordinate system and the vehicle body coordinate system to optimize the external parameters of the vehicle-mounted camera relative to the room coordinate system, and obtain the conversion relationship of the vehicle-mounted camera relative to the vehicle body coordinate system. Alternatively, the electronic device can also use the conversion relationship between the room coordinate system and the vehicle body coordinate system to convert the coordinates of the target set in the calibration room in the room coordinate system into the coordinates in the vehicle body coordinate system, and then determine the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system based on the coordinates of the target in the vehicle body coordinate system.

[0084] In some embodiments, a target is also provided in the calibration room, and the external parameters of the vehicle's on-board camera relative to the vehicle body coordinate system are calibrated according to the conversion relationship, including: obtaining a second image taken by the vehicle's on-board camera of the target provided in the calibration room; determining the external parameters of the vehicle's on-board camera relative to the vehicle body coordinate system according to the pixel coordinates of the target corner point in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

[0085] In some embodiments, when the external parameters of the vehicle's on-board camera relative to the vehicle body coordinate system are obtained, the vehicle can obtain the geometric relationship of the on-board camera relative to the road according to the geographical location of the road on which the vehicle is located. Furthermore, through this geometric relationship, distance or geometric measurement and calculation of road conditions or targets photographed by the on-board camera can be achieved, which can play a great role, especially in autonomous driving scenarios.

[0086] The calibration method of the vehicle-mounted camera is to set a camera with a fixed position and posture in the calibration room, that is, a fixed camera, and use multiple fixed cameras to respectively shoot the vehicle to obtain the corresponding first image, and detect the body corner points in each first image, so as to accurately determine the three-dimensional coordinates of the target corner points in the three-dimensional space, that is, the coordinates of the target corner points in the room coordinate system, according to the pixel coordinates of the body corner points (that is, the target corner points) co-occurring in at least two first images. In this way, the conversion relationship between the body coordinate system and the room coordinate system can be obtained according to the coordinates of the multiple target corner points in the body coordinate system and the coordinates in the room coordinate system. The conversion relationship can represent the position and posture of the vehicle in the calibration room, so the calibration of the vehicle-mounted camera relative to the body coordinate system can be realized by using the conversion relationship. This calibration method can avoid the calibration error caused by the change of the vehicle posture when the vehicle is fixed by the vehicle limit device, and improve the accuracy of the external parameter calibration of the vehicle-mounted camera.

[0087] It should be noted that in this application, except for the pixel coordinates in the image, the coordinates in other coordinate systems all refer to spatial coordinates (such as three-dimensional coordinates). For example, the coordinates in the room coordinate system are three-dimensional coordinates, the coordinates in the vehicle body coordinate system are three-dimensional coordinates, and the coordinates in the camera coordinate system are also three-dimensional coordinates.

[0088] In this application, the three-dimensional coordinates of the target corner points in the room coordinate system and the three-dimensional coordinates in the vehicle body coordinate system are utilized to accurately determine the conversion relationship between the room coordinate system and the vehicle body coordinate system. Then, when the conversion relationship is used to determine the external parameters of the vehicle-mounted camera, the accuracy of the external parameter calibration can be greatly improved.

[0089] In an exemplary embodiment, the coordinates of the target corner point in the room coordinate system are determined based on the detection results, including: screening out the body corner points that co-occur in at least two first images from the detected body corner points as the target corner points; for any target corner point, determining the coordinates of the target corner point in the room coordinate system based on the pixel coordinates of the image point of the target corner point in different first images.

[0090] The detection result includes the identifier and pixel coordinates of the corner point of the vehicle body in the first image. The identifier of the corner point of the vehicle body is used to uniquely identify a corner point of the vehicle body, such as the upper corner point of the vehicle window, the lower corner point of the vehicle window, the corner of the reflector, etc. The pixel coordinates are the coordinates of the pixel point in the image coordinate system or the pixel coordinate system. The pixel coordinates of the corner point of the vehicle body, that is, the pixel coordinates of the image point of the corner point of the vehicle body, refer to the pixel coordinates of the point corresponding to the corner point of the vehicle body in the physical world on the image.

[0091] The image coordinate system is a coordinate system with the center of the image plane as the origin, and the X-axis and Y-axis are parallel to the two vertical sides of the image plane. The pixel coordinate system is a coordinate system with the upper left corner of the image plane as the origin, and the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system, respectively, and its coordinate values ​​are represented by (u, v).

[0092] Specifically, the electronic device can perform corner point detection on each first image, detect the vehicle body corner points and the pixel coordinates of the image points of each vehicle body corner point in each first image, and then filter out the vehicle body corner points that co-appear in at least two first images as target corner points. It can be understood that different first images are taken by different fixed cameras, and different fixed cameras are located at different positions in the calibration room, so the selected target corner points are vehicle body corner points that can be photographed by fixed cameras at different angles.

[0093] It can be understood that when multiple cameras capture the same actual physical point, this physical point falls on a spatial straight line determined by each camera, that is, the intersection of these spatial straight lines. , electronic devices can determine their coordinates in the room coordinate system in the following way: A first image is obtained, a spatial straight line is determined based on a fixed camera from which the first image originates, and a target corner point can be detected. The intersection of all the fixed cameras in the space determines the corner point of the target Coordinates in room coordinates.

[0094] In the above embodiment, fixed cameras at different angles are used to shoot the target corner points, so that the three-dimensional position of the target corner points in the actual physical world, that is, the coordinates in the room coordinate system, can be accurately located according to the imaging of the target corner points at different angles.

[0095] In some embodiments, for any target corner point, the coordinates of the target corner point in the room coordinate system are determined according to the pixel coordinates of the image point of the target corner point in different first images, including: determining the coordinates of any target corner point in the room coordinate system by the following steps: taking the first image in which the target corner point currently being processed is detected as the target image; converting the pixel coordinates of the image point of the target corner point currently being processed in the target image into coordinates in the room coordinate system according to the camera parameters of the fixed camera from which the target image originates; determining the coordinates of the target corner point in the room coordinate system according to the coordinates of the fixed camera from which each target image originates and the coordinates of the image point corresponding to each target image in the room coordinate system.

[0096] Specifically, for each target corner point, the electronic device can determine its coordinates in the room coordinate system in the same way. For the convenience of description, for any target corner point, the first image in which the target corner point is detected can be called a target image.

[0097] Exemplarily, for any target image, the electronic device can determine the coordinates of the image point of the target corner point in the room coordinate system in the following manner: the electronic device can convert the pixel coordinates of the image point of the target corner point in the target image into the coordinates of the image point of the target corner point in the room coordinate system based on the camera parameters of the fixed camera that captures the target image.

[0098] In some embodiments, the pixel coordinates of the image point of the target corner point currently being processed in the target image are converted into coordinates in the room coordinate system according to the camera parameters of the fixed camera from which the target image is sourced, including: according to the intrinsic parameters of the fixed camera from which the target image is sourced, the pixel coordinates of the image point of the target corner point currently being processed in the target image are converted into coordinates in the camera coordinate system; according to the extrinsic parameters of the fixed camera from which the target image is sourced, the coordinates of the image point of the target corner point currently being processed in the camera coordinate system are converted into coordinates in the room coordinate system.

[0099] Among them, the intrinsic parameters of the camera include focal length, principal point coordinates and distortion coefficients. The extrinsic parameters of a fixed camera refer to the extrinsic parameters of a fixed camera relative to the room coordinate system. The camera coordinate system refers to a coordinate system whose origin is the optical center of the camera, whose X-axis and Y-axis are respectively parallel to the X-axis and Y-axis of the image coordinate system, and whose Z-axis is the optical axis of the camera. It can be understood that, according to the imaging principle of the camera, the electronic device can convert the pixel coordinates of the pixel points in the image captured by the camera into coordinates in the camera coordinate system. Furthermore, based on the extrinsic parameters of the fixed camera, the electronic device can convert the coordinates of the image point of the currently processed target corner point in the camera coordinate system into the coordinates of the image point in the room coordinate system.

[0100] In the above embodiment, based on the camera parameters of the fixed camera, the pixel coordinates of the image point of the target corner point can be converted into the coordinates of the image point of the target corner point in the room coordinate system, so as to be used for subsequently determining the coordinates of the target corner point in the room coordinate system.

[0101] Furthermore, the electronic device may determine the coordinates of the target corner points in the room coordinate system according to the coordinates of the fixed camera from which each target image originates in the room coordinate system and the coordinates of the image points corresponding to each target image in the room coordinate system.

[0102] In the above embodiment, according to the imaging of the target corner point in the fixed camera at different angles, the three-dimensional position of the target corner point in the actual physical space, that is, the coordinates in the room coordinate system, can be accurately determined.

[0103] In some embodiments, the coordinates of the target corner point in the room coordinate system are determined according to the coordinates of the fixed camera from which each target image originates and the coordinates of the image point corresponding to each target image in the room coordinate system, including: for each target image, determining a spatial straight line according to the coordinates of the fixed camera from which the target image originates and the coordinates of the image point corresponding to the target image in the room coordinate system; based on the spatial straight lines determined for each target image, determining the coordinates of the target corner point currently being processed in the room coordinate system.

[0104] It should be noted that for any target corner point, its pixel coordinates in the image taken by a fixed camera are , according to the internal parameters of the fixed camera, its distortion-free coordinates can be obtained in advance: The specific calculation method can be determined according to the distortion model of the fixed camera.

[0105] According to the camera imaging model, Points in the world coordinate system (specifically, the room coordinate system in this application) Projected, it satisfies the following formula:

[0106]

[0107] in, represents the internal parameters of the fixed camera, represents the external parameters of the fixed camera relative to the room coordinate system, represents the rotation matrix and T represents the translation vector.

[0108] According to the formula, the back projection process can be described as:

[0109]

[0110] It can be seen that for a point in the image, its corresponding possible actual physical point is distributed on a straight line, where λ is a variable parameter, and λ uniquely determines the coordinates of this actual physical point.

[0111] When multiple fixed cameras capture the same actual physical point, this physical point falls on a spatial straight line determined by each fixed camera, that is, the intersection of these spatial straight lines.

[0112] Therefore, for each target image in which the target corner point is detected, the electronic device can determine the spatial straight line according to the coordinates of the camera center of the fixed camera from which the target image is derived in the room coordinate system, and the coordinates of the image point in the room coordinate system determined by the pixel coordinates of the image point of the target corner point in the target image. The expression of the spatial straight line can be solved based on these two coordinates. Then, the electronic device obtains at least two spatial straight lines that theoretically pass through the target corner point. The coordinates of the target corner point in the room coordinate system can be determined from these at least two spatial straight lines.

[0113] In some embodiments, taking into account calculation errors and model errors, the actual physical spatial position of the target corner point can be defined as the distance to each related spatial straight line in space and the shortest point. That is, the coordinates of the spatial position point with the smallest distance from each spatial straight line are used as the coordinates of the target corner point currently being processed in the room coordinate system. This is a typical extreme value problem, and there are many solutions, such as the least squares method, which can solve the 3D coordinates of this point, that is, the coordinates of the target corner point in the room coordinate system. In this way, the coordinates of the target corner point in the room coordinate system can be quickly and accurately determined.

[0114] In some embodiments, the electronic device may also determine the coordinates of the target corner point in the room coordinate system in other ways. For example, the electronic device may determine the intersection of every two spatial straight lines based on multiple spatial straight lines related to the target corner point, and then cluster these intersections to obtain the cluster center point. Taking into account the existence of errors, non-parallel straight lines in three-dimensional space do not necessarily have intersections. Usually, in the sense of least squares, the point with the smallest sum of distances from two straight lines is understood as the intersection. The electronic device may use the coordinates of the cluster center point in the room coordinate system as the coordinates of the target corner point in the room coordinate system. It can be understood that the clustering method can be k-means or Gaussian mixture model clustering, etc., which is not limited to the embodiments of the present application.

[0115] In the above embodiment, based on the camera imaging principle, according to the coordinates of the fixed camera in the room coordinate system and the coordinates of the image point in the room coordinate system, at least two spatial straight lines that theoretically pass through the target corner point are determined, and then the coordinates of the target corner point in the room coordinate system can be quickly and accurately obtained based on these at least two spatial straight lines.

[0116] The following is a detailed description of how to use the conversion relationship between the room coordinate system and the vehicle body coordinate system to achieve the external parameter calibration process of the vehicle camera relative to the vehicle body coordinate system:

[0117] In some embodiments, calibrating the external parameters of a vehicle's onboard camera relative to a vehicle body coordinate system based on a conversion relationship includes: acquiring a second image taken by the vehicle's onboard camera of a target set in a calibration room; determining the external parameters of the vehicle's onboard camera relative to the vehicle body coordinate system based on the pixel coordinates of the target corner point in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

[0118] Among them, the vehicle-mounted camera is a camera installed in the vehicle, which can be installed on both sides of the vehicle exterior, or on the front or rear end of the vehicle exterior, etc. The shooting field of view of the vehicle-mounted camera can be towards the ground, or towards the front, rear or side, etc., and the embodiment of the present application does not limit this.

[0119] The calibration room is pre-set with detectable targets, which have many detectable and identifiable corner points, called target corner points. The physical positions of these target corner points (that is, the coordinates in the room coordinate system) can be measured in advance. In general, considering the different installation angles of different vehicle-mounted cameras, all the walls and floors of the calibration room photographed by all vehicle-mounted cameras will have some targets for calculation. For example, a chessboard can be set on the ground as a target, which is equipped with positioning marks of different sizes or colors or special patterns; some targets above the ground can also be equipped, usually these targets can be in the form of chessboards, dots or special QR codes. The target corner points of all these targets are measured in advance in the room coordinate system of the calibration room, and are easy to be detected and accurately located in the camera imaging.

[0120] The second image is a real scene image captured by the vehicle-mounted camera. It can be understood that the second image can be considered as an image obtained by shooting the target. In some embodiments, the number of vehicle-mounted cameras is 1 or more. In the case of multiple vehicle-mounted cameras, the same steps can be performed for the second image captured by each vehicle-mounted camera to obtain the external parameters of each vehicle-mounted camera relative to the vehicle body coordinate system.

[0121] Take any vehicle-mounted camera as an example: the vehicle-mounted camera of the vehicle is controlled to capture a second image and transmit the second image to the electronic device. The electronic device performs corner point recognition on the second image and determines the pixel coordinates of the target corner points in the second image. The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated by combining the coordinates of the target corner points in the room coordinate system, the pixel coordinates of the target corner points in the second image, and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

[0122] In this way, for each vehicle-mounted camera, the electronic device can calibrate its external parameters through the above method.

[0123] In the above embodiment, the second image obtained by the vehicle-mounted camera shooting the target in the calibration room is obtained, and the corner point recognition is performed on the second image to determine the pixel coordinates of the target corner point in the second image. Since the coordinates of the target corner point in the room coordinate system are known, combined with the conversion relationship between the room coordinate system and the vehicle body coordinate system, the external parameter calibration of the vehicle-mounted camera relative to the vehicle body coordinate system can be achieved.

[0124] In some embodiments, the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are determined based on the pixel coordinates of the target corner points in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system, including: converting the coordinates of the target corner points in the room coordinate system into coordinates in the vehicle body coordinate system according to the conversion relationship between the room coordinate system and the vehicle body coordinate system; calibrating the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system based on the pixel coordinates of the target corner points in the second image and the coordinates in the vehicle body coordinate system.

[0125] Specifically, based on the conversion relationship between the room coordinate system and the vehicle body coordinate system (rotation matrix and translation vector), the coordinates of the target corner points in the room coordinate system are converted to obtain the coordinates of the target corner points in the vehicle body coordinate system. Then, the electronic device can determine the external parameters of the vehicle camera relative to the vehicle body coordinate system based on the pixel coordinates of each target corner point in the second image and the coordinates in the vehicle body coordinate system.

[0126] Before explaining the external parameter calibration process of the vehicle-mounted camera, the external parameter calibration of any camera is explained first:

[0127] Generally, when each camera is designed and manufactured, the manufacturer can provide the camera's intrinsic parameters, including horizontal focal length. , vertical focal length , principal point coordinates , distortion coefficient The description of the distortion coefficient is related to the selected distortion model. Vehicle-mounted cameras are no exception.

[0128] Assuming camera intrinsics . A point in the 3D world coordinate system Assume that the rotation matrix of the camera extrinsic parameter is as follows: Assume that the translation vector of the camera extrinsic parameter is The rotation matrix is ​​actually composed of the rotation vector Without considering the distortion, The projection coordinates in the camera are ,have ,in By

[0129] Considering the distortion, The projection in the camera is This process simulates the camera imaging model and is called reprojection, denoted by .

[0130] Generally speaking, the camera's extrinsic calibration is achieved by optimizing the error between the reprojected position of the corner point in the world coordinate system in the camera and the coordinates of the corner point in the image obtained from the actual image.

[0131] Then, it can be understood that the external parameter calibration of the vehicle camera relative to the vehicle body coordinate system can be obtained by minimizing the error between the reprojected position of the target corner point in the vehicle body coordinate system in the vehicle camera and the pixel coordinate of the target corner point in the second image. Specifically, the external parameters of the vehicle camera can be optimized by the following formula:

[0132]

[0133] in, The coordinates of the target corner point j in the vehicle body coordinate system, The index target corner point j is the pixel coordinate in the second image, and M is the total number of target corner points appearing in the second image.

[0134] If the intrinsic parameters of the onboard camera are also unknown, then the optimization process can be performed together with the intrinsic parameters when enough non-coplanar target corner points are detected, that is,

[0135]

[0136] In some embodiments, the electronic device can optimize and solve the above formula by least square method. Many open source algorithms can solve such optimization problems, such as Ceres Solver.

[0137] In the above embodiment, according to the conversion relationship between the room coordinate system and the vehicle body coordinate system, the coordinates of the target corner points in the room coordinate system can be converted into coordinates in the vehicle body coordinate system, and then according to the pixel coordinates of multiple target corner points and the coordinates in the vehicle body coordinate system, the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system can be solved.

[0138] In some embodiments, the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are determined according to the pixel coordinates of the target corner points in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system, including: calibrating the external parameters of the vehicle-mounted camera relative to the room coordinate system according to the pixel coordinates of the target corner points in the second image and the coordinates in the room coordinate system; determining the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the external parameters of the vehicle-mounted camera relative to the room coordinate system and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

[0139] Specifically, the electronic device can calibrate the external parameters of the vehicle-mounted camera relative to the room coordinate system according to the pixel coordinates of the target corner points in the second image and the coordinates in the room coordinate system. Then, the external parameters of the vehicle-mounted camera relative to the room coordinate system are further optimized by using the conversion relationship between the room coordinate system and the vehicle body coordinate system to obtain the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system.

[0140] In some embodiments, as described in the above embodiments, the extrinsic calibration method of the camera can be understood that the extrinsic calibration of the vehicle-mounted camera relative to the room coordinate system can be obtained by minimizing the error between the reprojected position of the target corner point in the room coordinate system in the vehicle-mounted camera and the pixel coordinates of the target corner point in the second image. Specifically, the extrinsic parameters of the vehicle-mounted camera can be optimized by the following formula:

[0141]

[0142] in, The coordinates of the target corner point j in the room coordinate system, The index target corner point j is the pixel coordinate in the second image, and M is the total number of target corner points appearing in the second image.

[0143] If the intrinsic parameters of the onboard camera are also unknown, then the optimization process can be performed together with the intrinsic parameters when enough non-coplanar target corner points are detected, that is,

[0144]

[0145] In some embodiments, the electronic device may optimize and solve the above formula by using the least square method.

[0146] In some embodiments, after the external parameters of the vehicle-mounted camera relative to the room coordinate system are calibrated, the electronic device can calculate the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system in the following manner:

[0147] remember is the three-dimensional coordinate in the camera coordinate system, and the coordinate of the same physical point in the room coordinate system is , the external parameter from the room coordinate system to the camera coordinate system is the rotation matrix and translation vectors ,have .

[0148] The transformation relationship between the room coordinate system and the body coordinate system can be expressed as: rotation matrix and translation vectors , then we have the following formula: ;

[0149] Therefore, we can know that:

[0150] .

[0151] In this way, you can get , .

[0152] In the above embodiment, after calibrating the external parameters of the vehicle-mounted camera relative to the room coordinate system, the external parameters of the vehicle-mounted camera relative to the room coordinate system can be further optimized using the conversion relationship between the room coordinate system and the vehicle body coordinate system to obtain the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system.

[0153] It should be noted that before calibrating the onboard camera of each vehicle, it is necessary to calibrate the camera parameters of each fixed camera arranged in the calibration room in advance. The specific process is as follows:

[0154] In some embodiments, before obtaining the first images taken by each of the multiple fixed cameras in the calibration room, the method further includes: obtaining a room image of the calibration room taken by the multiple fixed cameras; providing a target in the calibration room; detecting the target corner points in the room image; and calibrating the camera parameters of each fixed camera according to the pixel coordinates of the target corner points in the room image and the coordinates of the target corner points in the room coordinate system.

[0155] First, establish a three-dimensional Euclidean coordinate system in the calibration room. In this coordinate system, the coordinates of each target corner point are precisely measured, assuming that target corner points, denoted as .

[0156] The calibration room is installed Fixed cameras For any of the fixed cameras, the room image captured by a single fixed camera can detect target corner points, counted as .

[0157] In some embodiments, considering that various target corner points arranged in the calibration have high similarity in a local range, it is necessary to detect some reference objects in the calibration room while detecting each valid corner point, such as using colored marker points in a chessboard, or using self-coded corner points (such as AprilTag) for a single erected target, etc., to assist in determining which point in the room each detected target corner point is, so as to select matching points from the measured target corner points, and generate a correspondence between the target corner points in the room image and the 3D coordinates of the target corner points arranged in the calibration room. The coordinates of the corresponding target corner points in the room coordinate system are .

[0158] According to the method for calculating the external parameters of the camera mentioned in the above embodiment, the optimization of the external parameters of the fixed camera can be obtained by minimizing the error between the reprojected position of the target corner point in the room coordinate system in the fixed camera and the pixel coordinates of the target corner point in the room image. Specifically, the external parameters of the camera can be optimized by the following formula:

[0159]

[0160] in, The coordinates of the target corner point i in the room coordinate system, The pixel coordinates of the target corner point i in the room image.

[0161] If the intrinsic parameters of the fixed camera are also unknown, then the optimization process can be performed together with the intrinsic parameters if enough non-coplanar target corner points are detected, that is,

[0162]

[0163] In some embodiments, the electronic device may optimize and solve the above formula by using the least square method.

[0164] It should be noted that the fixed camera installed in the calibration room in the present application can not only be used to compensate for errors caused by changes in the vehicle body posture, but also be used to monitor various geometric changes in the calibration room. For example, the posture changes of the targets placed vertically on the surrounding walls may occur under temperature, humidity, and external forces. This can effectively avoid the problem of inaccurate calibration results due to changes in the coordinates of the target corner points in the room coordinate system when calibrating the vehicle-mounted camera.

[0165] In the above embodiment, the targets set in the calibration room are used to calibrate the camera parameters of each fixed camera in the calibration room, so that the extrinsic parameter calibration of the vehicle-mounted camera can be assisted by the calibrated fixed cameras.

[0166] In an exemplary embodiment, Figure 6As shown, the calibration method of the vehicle-mounted camera includes the following steps: installing a fixed camera, shooting room images through the fixed camera; detecting target corner points in each room image; measuring the coordinates of all target corner points in the calibration room in the room coordinate system; calibrating the camera parameters of each fixed camera relative to the room coordinate system; the vehicle enters the centering platform and is fixed; each fixed camera shoots; detects the vehicle body corner points shot by each fixed camera; determines the coordinates of the common corner points with the same name (target corner points) in the room coordinate system; determines the conversion relationship from the room coordinate system to the vehicle body coordinate system based on the vehicle model CAD drawing information; converts the coordinates of the target corner points in the calibration room in the room coordinate system into the coordinates in the vehicle body coordinate system; the vehicle-mounted camera shoots the targets in the calibration room; detects the pixel coordinates of the target corner points based on the room image; determines the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the pixel coordinates of the target corner points and the coordinates in the vehicle body coordinate system.

[0167] In an exemplary embodiment, Figure 7 As shown, the calibration method of the vehicle-mounted camera includes the following steps: installing a fixed camera and shooting room images through the fixed camera; detecting target corner points in each room image; measuring the coordinates of all target corner points in the calibration room in the room coordinate system; calibrating the camera parameters of each fixed camera relative to the room coordinate system; the vehicle enters the centering platform and is fixed; each fixed camera shoots; detects the vehicle body corner points shot by each fixed camera; determines the coordinates of the common corner points with the same name (target corner points) in the room coordinate system; determines the conversion relationship from the room coordinate system to the vehicle body coordinate system based on the vehicle model CAD drawing information; the vehicle-mounted camera shoots the target in the calibration room; detects the pixel coordinates of the target corner points based on the room image; calculates the external parameters of the vehicle-mounted camera relative to the room coordinate system; converts the external parameters of the vehicle-mounted camera relative to the room coordinate system into the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system.

[0168] In addition, considering that the present application can compensate for vehicle body posture differences, the vehicle limiting devices (such as centering platforms) in the original EOL calibration room can be reduced in actual use, or at least the accuracy requirements for the vehicle limiting devices can be reduced.

[0169] Therefore, further, in one embodiment, if Figure 8As shown, the calibration method of the vehicle-mounted camera includes the following steps: installing a fixed camera and shooting room images through the fixed camera; detecting the target corner points in each room image; measuring the coordinates of all target corner points in the calibration room in the room coordinate system; calibrating the camera parameters of each fixed camera relative to the room coordinate system; the vehicle enters the calibration room (without a centering platform); each fixed camera shoots; detects the vehicle body corner points shot by each fixed camera; determines the coordinates of the common corner points with the same name (target corner points) in the room coordinate system; determines the conversion relationship from the room coordinate system to the vehicle body coordinate system based on the vehicle model CAD drawing information; the vehicle-mounted camera shoots the target in the calibration room; detects the pixel coordinates of the target corner points based on the room image; calculates the external parameters of the vehicle-mounted camera relative to the room coordinate system; converts the external parameters of the vehicle-mounted camera relative to the room coordinate system into the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system.

[0170] In some embodiments, Fig. 9 As shown, the calibration method of the vehicle-mounted camera includes the following steps: installing a fixed camera and shooting room images through the fixed camera; detecting the target corner points in each room image; measuring the coordinates of all target corner points in the calibration room in the room coordinate system; calibrating the camera parameters of each fixed camera relative to the room coordinate system; the vehicle enters the calibration room (without a centering platform); each fixed camera shoots; detects the vehicle body corner points shot by each fixed camera; determines the coordinates of the common corner points with the same name (target corner points) in the room coordinate system; determines the conversion relationship from the room coordinate system to the vehicle body coordinate system based on the vehicle model CAD drawing information; the vehicle-mounted camera shoots the target in the calibration room; detects the pixel coordinates of the target corner points based on the room image; calculates the external parameters of the vehicle-mounted camera relative to the room coordinate system; converts the external parameters of the vehicle-mounted camera relative to the room coordinate system into the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system.

[0171] It should be noted that the above Figure 6-Figure 9 The part within the dotted box can be completed offline, and the same batch of vehicles can be done only once in advance.

[0172] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides a vehicle-mounted camera calibration device for implementing the vehicle-mounted camera calibration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the calibration device for one or more vehicle-mounted cameras provided below can refer to the limitations of the vehicle-mounted camera calibration method above, and will not be repeated here.

[0174] In an exemplary embodiment, Fig.10 As shown, a calibration device 1000 for a vehicle-mounted camera is provided, comprising: an acquisition module 1001, a first determination module 1002, a second determination module 1003 and a calibration module 1004, wherein:

[0175] The acquisition module 1001 is used to acquire first images taken by multiple fixed cameras in the calibration room; there are vehicles parked in the calibration room, and there are body corner points on the surface of the vehicle.

[0176] The first determination module 1002 is used to detect the corner points of the vehicle body in each first image, and determine the coordinates of the target corner point in the room coordinate system according to the detection result; the target corner point is the corner point of the vehicle body that co-occurs in at least two first images.

[0177] The second determining module 1003 is used to determine the conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system.

[0178] The calibration module 1004 is used to calibrate the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the conversion relationship.

[0179] In some embodiments, the first determination module 1002 is also used to: screen out the body corner points that co-occur in at least two first images from the detected body corner points as target corner points; for any target corner point, determine the coordinates of the target corner point in the room coordinate system based on the pixel coordinates of the image point of the target corner point in different first images.

[0180] In some embodiments, the first determination module 1002 is further used to: use the first image in which the target corner point currently being processed is detected as the target image; convert the pixel coordinates of the image point of the target corner point currently being processed in the target image into coordinates in the room coordinate system according to the camera parameters of the fixed camera from which the target image originates; determine the coordinates of the target corner point in the room coordinate system according to the coordinates of the fixed camera from which each target image originates in the room coordinate system, and the coordinates of the image point corresponding to each target image in the room coordinate system.

[0181] In some embodiments, the first determination module 1002 is also used to: convert the pixel coordinates of the image point of the target corner point currently being processed in the target image into coordinates in the camera coordinate system according to the intrinsic parameters of the fixed camera from which the target image originates; and convert the coordinates of the image point of the target corner point currently being processed in the camera coordinate system into coordinates in the room coordinate system according to the extrinsic parameters of the fixed camera from which the target image originates.

[0182] In some embodiments, the first determination module 1002 is also used to: for each target image, determine a spatial straight line according to the coordinates of the fixed camera from which the target image originates in the room coordinate system, and the coordinates of the image points corresponding to the target image in the room coordinate system; based on the spatial straight lines determined for each target image, determine the coordinates of the target corner point currently being processed in the room coordinate system.

[0183] In some embodiments, the first determining module 1002 is further configured to: use the distances from each spatial straight line and the coordinates of the minimum spatial position point as the coordinates of the target corner point currently being processed in the room coordinate system.

[0184] In some embodiments, the first calibration module 1004 is also used to: obtain a second image taken by the vehicle's on-board camera of a target set in a calibration room; determine the external parameters of the on-board camera relative to the vehicle body coordinate system based on the pixel coordinates of the target corner point in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

[0185] In some embodiments, the first calibration module 1004 is also used to: convert the coordinates of the target corner point in the room coordinate system into the coordinates in the vehicle body coordinate system according to the conversion relationship between the room coordinate system and the vehicle body coordinate system; calibrate the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the pixel coordinates of the target corner point in the second image and the coordinates in the vehicle body coordinate system.

[0186] In some embodiments, the first calibration module 1004 is also used to: calibrate the external parameters of the vehicle-mounted camera relative to the room coordinate system based on the pixel coordinates of the target corner points in the second image and their coordinates in the room coordinate system; determine the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system based on the external parameters of the vehicle-mounted camera relative to the room coordinate system and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

[0187] In some embodiments, the device also includes a second calibration module, which is used to obtain a room image of a calibration room taken by multiple fixed cameras; a target is set in the calibration room; the target corner points in the room image are detected; and the camera parameters of each fixed camera are calibrated according to the pixel coordinates of the target corner points in the room image and the coordinates of the target corner points in the room coordinate system.

[0188] Each module in the above-mentioned vehicle-mounted camera calibration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0189] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal or a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a calibration method for a vehicle-mounted camera is implemented.

[0190] Those skilled in the art will understand that Fig.11 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0191] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0192] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0193] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0195] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0196] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0197] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for calibrating a vehicle-mounted camera, characterized in that: The method comprises: Acquire first images taken by multiple fixed cameras in a calibration room; a vehicle is parked in the calibration room, and body corners exist on the surface of the vehicle; Detecting the corner points of the vehicle body in each of the first images, and determining the coordinates of the target corner points in the room coordinate system according to the detection results; the target corner points are the corner points of the vehicle body that co-appear in at least two of the first images; Determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system; The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated according to the conversion relationship.

2. The method according to claim 1, characterized in that The step of determining the coordinates of the target corner point in the room coordinate system according to the detection result includes: From the detected vehicle body corner points, select the vehicle body corner points that co-appear in at least two first images as target corner points; For any target corner point, the coordinates of the target corner point in the room coordinate system are determined according to the pixel coordinates of the image points of the target corner point in different first images.

3. The method according to claim 2, characterized in that The step of determining, for any target corner point, the coordinates of the target corner point in the room coordinate system according to the pixel coordinates of the image point of the target corner point in different first images includes: Determine the coordinates of any target corner point in the room coordinate system by following these steps: The first image in which the target corner points currently being processed are detected is used as the target image; According to the camera parameters of the fixed camera from which the target image is derived, the pixel coordinates of the image point of the target corner point currently being processed in the target image are converted into coordinates in the room coordinate system; The coordinates of the target corner point in the room coordinate system are determined according to the coordinates of the fixed camera from which each of the target images originates in the room coordinate system and the coordinates of the image point corresponding to each of the target images in the room coordinate system.

4. The method according to claim 3, characterized in that The step of converting the pixel coordinates of the image point of the target corner point currently being processed in the target image into coordinates in the room coordinate system according to the camera parameters of the fixed camera from which the target image is obtained includes: According to the intrinsic parameters of the fixed camera from which the target image is derived, the pixel coordinates of the image point of the target corner point currently being processed in the target image are converted into coordinates in the camera coordinate system; According to the external parameters of the fixed camera from which the target image comes, the coordinates of the image point of the target corner point currently being processed in the camera coordinate system are converted into coordinates in the room coordinate system.

5. The method according to claim 3, characterized in that: Determining the coordinates of the target corner point in the room coordinate system according to the coordinates of the fixed camera from which each of the target images originates in the room coordinate system and the coordinates of the image point corresponding to each of the target images in the room coordinate system includes: For each target image, determine a spatial straight line according to the coordinates of a fixed camera from which the target image originates in a room coordinate system and the coordinates of the image point corresponding to the target image in the room coordinate system; Based on the spatial straight lines respectively determined for each target image, the coordinates of the target corner point currently being processed in the room coordinate system are determined.

6. The method according to claim 5, characterized in that The determining the coordinates of the target corner point currently being processed in the room coordinate system based on the spatial straight lines respectively determined for each target image comprises: The distances from each spatial straight line and the coordinates of the minimum spatial position point are used as the coordinates of the target corner point currently being processed in the room coordinate system.

7. The method according to claim 1, characterized in that The step of calibrating the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the conversion relationship includes: Acquire a second image captured by a vehicle-mounted camera of the vehicle with respect to a target disposed in a calibration room; According to the pixel coordinates of the target corner point in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system, the extrinsic parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are determined.

8. The method according to claim 7, characterized in that The step of determining the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the pixel coordinates of the target corner point in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system includes: According to the conversion relationship between the room coordinate system and the vehicle body coordinate system, the coordinates of the target corner point in the room coordinate system are converted into coordinates in the vehicle body coordinate system; The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are calibrated according to the pixel coordinates of the target corner points in the second image and the coordinates of the target corner points in the vehicle body coordinate system.

9. The method according to claim 7, characterized in that: The step of determining the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the pixel coordinates of the target corner point in the second image and the conversion relationship between the room coordinate system and the vehicle body coordinate system includes: Calibrate the extrinsic parameters of the vehicle-mounted camera relative to the room coordinate system according to the pixel coordinates of the target corner point in the second image and the coordinates in the room coordinate system; The external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system are determined according to the external parameters of the vehicle-mounted camera relative to the room coordinate system and the conversion relationship between the room coordinate system and the vehicle body coordinate system.

10. The method according to any one of claims 1 to 9, characterized in that Before acquiring first images respectively taken by a plurality of fixed cameras in the calibration room, the method further includes: Acquire room images of a calibration room captured by multiple fixed cameras; a target is provided in the calibration room; Detecting target corner points in the room image; The camera parameters of each of the fixed cameras are calibrated according to the pixel coordinates of the target corner points in the room image and the coordinates of the target corner points in the room coordinate system.

11. A calibration device for a vehicle-mounted camera, characterized in that: The device comprises: An acquisition module is used to acquire first images taken by multiple fixed cameras in a calibration room, wherein a vehicle is parked in the calibration room and body corners exist on the surface of the vehicle; A first determination module, used to detect the corner points of the vehicle body in each of the first images, and determine the coordinates of the target corner point in the room coordinate system according to the detection results; the target corner point is a corner point of the vehicle body that co-occurs in at least two first images; A second determination module, configured to determine a conversion relationship between the room coordinate system and the body coordinate system according to the coordinates of the target corner point in the body coordinate system and the coordinates of the target corner point in the room coordinate system; The first calibration module is used to calibrate the external parameters of the vehicle-mounted camera relative to the vehicle body coordinate system according to the conversion relationship.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images