A semi-automatic calibration method for camera and millimeter wave radar external parameters and related equipment
By selecting the inner corner point of the target library as the target point in the driving school site, placing the angle reflector, obtaining the data information in multiple position states, and calculating the external parameter matrix of the camera and millimeter wave radar, the calibration of the existing technology is solved The problem of low accuracy and efficiency achieves high precision and high efficiency external parameter calibration.
Patent Information
- Application Number
- CN202410772597.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-17
AI Technical Summary
In the existing external parameter calibration methods of camera and millimeter-wave radar, the joint calibration method relies on the main sensor lidar, which is costly and lacks high-precision positioning and map information. The calibration method between calibration is complex and low efficiency, making it difficult to ensure calibration accuracy while ensuring calibration accuracy. Improve calibration efficiency.
Using the semi-automatic calibration method, by obtaining the target library information of the driving school site where the target vehicle is located, randomly selecting the inner corner point as the target point, placing the angle reflector to obtain the vehicle position information, camera image coordinate information and millimeters in different positions. The coordinate information of the wave radar target point is calculated, the external parameter matrix of the camera and millimeter wave radar is reduced, and the calculation steps are avoided.
It improves calibration accuracy and efficiency, avoids the limitations of lidar dependence and fixed closed environment, and realizes high-precision external parameter calibration in driving school site environment.
Smart Images

Figure CN118351176B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of sensor extrinsic parameter calibration, and in particular relates to a camera and millimeter wave radar extrinsic parameter semi-automatic calibration method and related equipment. Background Art
[0002] The current autonomous driving perception system usually relies on the coupling of multiple sensors such as millimeter-wave radar, camera, and lidar to obtain data detected by multiple sensors, thereby improving the accuracy of target positioning and recognition, and improving the safety redundancy and stability of the autonomous driving system. External parameter calibration is the prerequisite for realizing multi-sensor data fusion. Through accurate external parameter calibration, information from different sensors can be converted into a unified coordinate system, providing a consistent data foundation for comprehensive perception. Secondly, external parameter calibration is crucial for the precise positioning of the vehicle. By calibrating the external parameters of the camera and millimeter-wave radar, a more accurate perception of the vehicle's surrounding environment can be achieved, and the positioning accuracy can be improved, thereby providing reliable data support for key tasks such as navigation and path planning.
[0003] At present, the external parameter calibration of cameras and millimeter-wave radars usually adopts the following two methods: one is to use a joint calibration method to determine the external parameters of the camera or millimeter wave, and the other is to use a calibration room to calibrate the external parameters of the sensor. The joint calibration method relies on the main sensor lidar to realize the external parameter calibration of the camera and millimeter-wave sensor, while the calibration room calibration method lacks high-precision positioning and high-precision map information. Determining the direction and position of the vehicle requires the help of a straightening mechanism, which is costly.
[0004] Therefore, how to provide a semi-automatic calibration method for camera and millimeter-wave radar external parameters to replace the joint calibration and inter-calibration calibration methods used in the prior art, so as to achieve the technical effect of improving calibration efficiency while ensuring calibration accuracy is a problem that technical personnel in this field urgently need to solve. Summary of the invention
[0005] The embodiment of the present invention provides a method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters and related equipment to solve at least one of the above technical problems.
[0006] In a first aspect, the present invention provides a method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters, which is used to calibrate the extrinsic parameters of a camera and a millimeter-wave radar of a target vehicle. The calibration method comprises:
[0007] Obtaining target storage location information of the driving school site where the target vehicle is located, the target storage location information including coordinates of several inner corner points constituting the target storage location based on the UTM coordinate system;
[0008] Randomly select one of the inner corner points as the target point;
[0009] After placing a corner reflector at the target point, the target vehicle is directed toward the target storage location in different posture states, and the vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter wave radar are obtained;
[0010] Calculating a camera extrinsic parameter matrix from the camera to the vehicle coordinate system according to the vehicle posture information and the storage location image coordinate information;
[0011] Based on the calculation of the camera extrinsic parameter matrix, the millimeter wave radar extrinsic parameter matrix in the vehicle coordinate system is calculated according to the target point coordinate information.
[0012] Preferably, the step of obtaining the vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter wave radar includes:
[0013] Obtain N different posture states of the target vehicle, where N is a positive integer greater than or equal to 3;
[0014] Collecting vehicle posture information of the target vehicle in each posture state, wherein the vehicle posture information includes the position coordinates and orientation angle of the target vehicle based on the UTM coordinate system;
[0015] Capturing a storage location image in each posture state by means of a camera, wherein the storage location image is an image captured by the camera containing the target storage location;
[0016] Processing the storage location image to obtain storage location image coordinate information, wherein the storage location image coordinate information includes corner point pixel coordinates of a plurality of inner corner points of the target storage location in the storage location image based on a pixel coordinate system;
[0017] The corner reflector is detected by a millimeter-wave radar to obtain target point coordinate information of the corner reflector in each posture state, wherein the target point coordinate information is the target point coordinate of the target point detected by the millimeter-wave radar based on the millimeter-wave radar coordinate system.
[0018] Preferably, the processing of the storage location image to obtain storage location image coordinate information includes:
[0019] Detecting and marking each of the storage location images according to the inner corner points of the target storage location to obtain marked pixel corner points;
[0020] Calculate the pixel coordinates of each pixel corner point in the storage location image according to the pixel coordinate system;
[0021] The pixel coordinates of a plurality of corner points constitute the storage location image coordinate information.
[0022] Preferably, the acquisition time corresponding to the vehicle posture information, the storage location image coordinate information and the target point coordinate information in the i-th posture state satisfies the following preset relationship:
[0023] ;
[0024] Wherein, i is a positive integer greater than or equal to 1 and less than or equal to N, is the acquisition time of the storage location image in the i-th posture state, is the collection time of the vehicle posture information in the i-th posture state, is the time when the coordinate information of the target point is collected in the i-th posture state.
[0025] Preferably, the step of calculating the camera extrinsic parameter matrix from the camera to the vehicle coordinate system based on the vehicle posture information and the storage location image coordinate information includes:
[0026] Calculate the relative coordinates of all the inner corner points of the target storage location to the corner points in the vehicle coordinate system in each posture state according to the vehicle posture information and the storage location image coordinate information;
[0027] Constructing a pixel coordinate matrix formed by the pixel coordinates of the corner points in all posture states in a pixel coordinate system according to a first preset order, and constructing a first vehicle coordinate matrix formed by the relative coordinates of the corner points in all posture states in a vehicle coordinate system according to the first preset order;
[0028] The camera rotation and translation matrix of the pixel coordinate matrix relative to the first vehicle coordinate matrix is calculated by a pnp algorithm, and the camera rotation and translation matrix forms a camera extrinsic parameter matrix in the camera-to-vehicle coordinate system.
[0029] Preferably, the step of calculating the relative coordinates of all inner corner points of the target storage location in each posture state to the corner points in the vehicle coordinate system based on the vehicle posture information and the storage location image coordinate information includes:
[0030] The relative coordinates of one of the inner corners of the target location in the i-th position are calculated according to the following formula: :
[0031]
[0032]
[0033]
[0034] in, is the horizontal coordinate of one of the inner corner points in the i-th posture state based on the UTM coordinate system, is the ordinate of one of the inner corner points in the i-th posture state based on the UTM coordinate system, is the horizontal coordinate of the target vehicle based on the UTM coordinate system in the i-th posture state, is the ordinate of the target vehicle based on the UTM coordinate system in the i-th posture state, is the orientation angle of the target vehicle based on the UTM coordinate system in the i-th posture state.
[0035] Preferably, the step of calculating the millimeter wave radar extrinsic parameter matrix of the millimeter wave radar in the vehicle coordinate system based on the target point coordinate information based on the calculation of the camera extrinsic parameter matrix includes:
[0036] Obtaining the relative coordinates of the target point to the target point in the vehicle coordinate system;
[0037] Obtain the number of millimeter-wave radars, where the number of millimeter-wave radars is M, and M is a positive integer;
[0038] Obtaining the coordinate information of the target point detected by each millimeter-wave radar in each posture state;
[0039] Constructing M millimeter-wave coordinate matrices corresponding to the M millimeter-wave radars in a millimeter-wave radar coordinate system, which are formed by the coordinate information of the target points in all posture states detected by each millimeter-wave radar in a millimeter-wave radar coordinate system in a second preset order, and constructing a second vehicle coordinate matrix formed by the relative coordinates of the target points in all posture states in a vehicle coordinate system according to the second preset order;
[0040] Calculating the millimeter wave rotation and translation matrix of the millimeter wave coordinate matrix relative to each of the second vehicle coordinate matrices by the least square method, the M millimeter wave rotation and translation matrices being the millimeter wave radar extrinsic parameter matrices corresponding to the M millimeter wave radars in the vehicle coordinate system;
[0041] Each of the millimeter wave rotation and translation matrices includes a millimeter wave rotation matrix and a millimeter wave translation matrix.
[0042] Preferably, the first preset order includes an arrangement order of posture states and an arrangement order of all inner corner points in each posture state, and the second preset order corresponds to the arrangement order of posture states in the first preset order.
[0043] In a second aspect, the present application also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the camera and millimeter-wave radar extrinsic parameter semi-automatic calibration method as described in any one of the first aspects when executing the computer program stored in the memory.
[0044] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon: when the computer program is executed by a processor, the method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters as described in any one of the first aspects is implemented.
[0045] Beneficial effect: The present invention proposes a semi-automatic calibration method for camera and millimeter-wave radar extrinsic parameters, which obtains the target storage location information of the driving school site where the target vehicle is located, the target storage location information includes the UTM coordinates of several inner corner points constituting the target storage location, randomly selects an inner corner point as the target point, places a corner reflector at the target point, and then directs the target vehicle toward the target storage location in different posture states, obtains the vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter-wave radar, and then calculates the camera extrinsic parameter matrix in the camera-to-vehicle coordinate system based on the vehicle posture information and the storage location image coordinate information, and calculates the millimeter-wave radar extrinsic parameter matrix in the millimeter-wave radar-to-vehicle coordinate system based on the target point coordinate information on the basis of the calculation of the camera extrinsic parameter matrix. The present invention makes full use of the autonomous driving environment of a driving school to calibrate the camera and millimeter-wave radar external parameters. By randomly selecting one of all the inner corner points of the target location as the target point, and placing a corner reflector at the target point to limit the position of the corner reflector, no additional complex relationship is required to associate the calculation process of the camera external parameters and the millimeter-wave radar external parameters, which reduces the calculation steps. By collecting data information in multiple different posture states to achieve external parameter calibration, it replaces the joint calibration and calibration room calibration methods used in the prior art, avoids dependence on the main sensor such as lidar and is limited by the fixed and closed environment of the calibration room, and can achieve the technical effect of improving calibration accuracy while improving calibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 Schematic diagram of the process of semi-automatic calibration method of camera and millimeter wave radar extrinsic parameters in an embodiment of the present invention;
[0048] Figure 2 It is a schematic diagram of performing target storage location detection on a storage location image in an embodiment of the present invention;
[0049] Figure 3 Schematic diagram of the inner corner point position of a rectangular target storage location in an embodiment of the present invention;
[0050] Figure 4 A schematic diagram of the structure of an electronic device provided by this application;
[0051] Figure 5 A schematic diagram of the structure of a computer-readable storage medium provided in this application. DETAILED DESCRIPTION
[0052] The present application provides a method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters and related equipment, which makes full use of the autonomous driving environment of a driving school to calibrate the camera and millimeter-wave radar extrinsic parameters. One of all the inner corner points of the target location is randomly selected as the target point, and a corner reflector is placed at the target point to limit the position of the corner reflector. Then, the extrinsic parameter calibration of the camera and millimeter-wave radar is realized by collecting data information of the inner corner points in multiple different postures.
[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0054] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.
[0055] The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.
[0056] Before introducing the camera and millimeter-wave radar extrinsic semi-automatic calibration method provided in this application, the background content involved in this application is first introduced.
[0057] The camera and millimeter-wave radar extrinsic semi-automatic calibration method provided in the present application can be applied to processing equipment, and is used to obtain the target storage location information of the driving school site where the target vehicle is located, randomly select an inner corner point as the target point, and place a corner reflector at the target point. Then, the target vehicle is oriented towards the target storage location in different postures to achieve the technical effect of improving the calibration accuracy while improving the calibration efficiency.
[0058] The camera and millimeter-wave radar external parameter semi-automatic calibration method mentioned in this application can be executed by a camera and millimeter-wave radar external parameter semi-automatic calibration device, or a network node, server, physical host or user equipment (UE) and other different types of processing equipment that integrate the camera and millimeter-wave radar external parameter semi-automatic calibration device. Among them, the camera and millimeter-wave radar external parameter semi-automatic calibration device can be implemented in hardware or software, and the UE can be a terminal device such as a smart phone, tablet computer, laptop computer, desktop computer or personal digital assistant (PDA), and the processing device can be set in the form of a device cluster.
[0059] It is understandable that if the present application involves a gateway product, it can be an application service configured in the form of a software service, which needs to be installed on the corresponding device in the network architecture to serve as a gateway node in the actual sense, or it can be a gateway node directly existing in the form of a device in the network architecture, which can be flexibly adjusted according to flexible deployment requirements / application requirements.
[0060] In this way, the processing equipment that performs the semi-automatic calibration of the camera and millimeter-wave radar external parameters provided in this application can specifically be different types of equipment such as network nodes, servers, physical hosts or UEs, which can be flexibly configured according to actual needs, and this application does not make specific limitations.
[0061] like Figure 1 As shown, the first embodiment provides a camera and millimeter-wave radar extrinsic semi-automatic calibration method, which is used to calibrate the camera and millimeter-wave radar of the target vehicle. The camera and millimeter-wave radar extrinsic semi-automatic calibration method specifically includes the following steps S110 to S150:
[0062] Step S110, obtaining target storage location information of the driving school where the target vehicle is located, wherein the target storage location information includes coordinates of several inner corner points constituting the target storage location based on the UTM coordinate system;
[0063] Specifically, the target vehicle can obtain high-precision map data, and the high-precision map data includes a variety of parking location information of the driving school where the target vehicle is located. The various parking location information at least includes vehicle parking spaces within the driving school, teaching parking spaces corresponding to the driving test subject type, etc. The present application can randomly or according to preset selection conditions select one from the various parking location information as the target parking location. Therefore, the target parking location can be a vehicle parking space within the driving school, or it can be a teaching parking space corresponding to the driving test subject type. The target parking location is drawn by lane lines with a certain width, and the inner corner points that constitute the target parking location refer to the corner points of the inner edge of the lane lines. For the convenience of calculation, in general, vehicle parking spaces with flat and open ground and clear lane lines are selected as target parking locations. Appendix Figure 3 That is, the target storage location is a schematic diagram of a vehicle parking space, which has four inner corner points, namely, the inner corner points corresponding to left_top, right_top, right_down, and left_down.
[0064] As a feasible method, before obtaining the target storage location information of the driving school site where the target vehicle is located in step S110, it is necessary to monitor the self-positioning status of the target vehicle in real time. When the self-positioning status of the vehicle is normal, the external parameter calibration step involved in the embodiment of the present application is entered; when the self-positioning status of the vehicle is abnormal, it is prompted that the external parameter calibration conditions are not met, and the self-positioning status of the vehicle is continuously monitored until the self-positioning status of the vehicle is normal and the external parameter calibration step is entered. Abnormal self-positioning status refers to a positioning status in which the positioning error of the target vehicle is greater than or equal to 0.2 cm or the vehicle heading angle is greater than or equal to 0.3°, and normal self-positioning status refers to a positioning status in which the positioning error of the target vehicle is less than 0.2 cm and the vehicle heading angle is less than 0.3°.
[0065] UTM refers to the Universal Transverse Mercator Grid System, which is a three-dimensional rectangular coordinate system (the Z coordinate is usually ignored) and is commonly used as a global coordinate in intelligent driving systems.
[0066] Step S120, randomly selecting one of the inner corner points as a target point;
[0067] Step S130: After placing a corner reflector at the target point, the target vehicle is directed toward the target storage location in different posture states, and vehicle posture information of the target vehicle in each posture state, storage location image coordinate information collected by the camera, and target point coordinate information detected by the millimeter wave radar are obtained;
[0068] Specifically, a corner reflector is placed at the target point so that the millimeter wave radar can detect the target point by detecting the corner reflector. Different posture states refer to the position change of the target vehicle relative to the target storage location, but it can be understood that the target vehicle should always face the target storage location so that the target storage location can be captured by the camera and the corner reflector can be detected by the millimeter wave radar.
[0069] As a feasible method, the number N of different posture states of the target vehicle obtained should be greater than or equal to 3. When N is less than 3, the vehicle is prompted to continue adjusting the posture state of the target vehicle and obtain the vehicle posture information in the corresponding posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter wave radar.
[0070] As an achievable method, the step S130 of obtaining the vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter wave radar specifically includes the following sub-steps:
[0071] Obtain N different posture states of the target vehicle, where N is a positive integer greater than or equal to 3;
[0072] Collecting vehicle posture information of the target vehicle in each posture state, wherein the vehicle posture information includes the position coordinates and orientation angle of the target vehicle based on the UTM coordinate system;
[0073] Capturing a storage location image in each posture state by means of a camera, wherein the storage location image is an image captured by the camera containing the target storage location;
[0074] Processing the storage location image to obtain storage location image coordinate information, wherein the storage location image coordinate information includes corner point pixel coordinates of a plurality of inner corner points of the target storage location in the storage location image based on a pixel coordinate system;
[0075] The corner reflector is detected by a millimeter-wave radar to obtain target point coordinate information of the corner reflector in each posture state, wherein the target point coordinate information is the target point coordinate of the target point detected by the millimeter-wave radar based on the millimeter-wave radar coordinate system.
[0076] The position coordinates of the target vehicle based on the UTM coordinate system refer to the coordinates of the center of mass of the target vehicle based on the UTM coordinate system, and the Z-axis coordinates are usually ignored.
[0077] As an achievable manner, the sub-step of the above step S130, in which the storage location image is processed to obtain the storage location image coordinate information, specifically includes the following contents:
[0078] Detecting and marking each of the storage location images according to the inner corner points of the target storage location to obtain marked pixel corner points;
[0079] Calculate the pixel coordinates of each pixel corner point in the storage location image according to the pixel coordinate system;
[0080] The pixel coordinates of a plurality of corner points constitute the storage location image coordinate information.
[0081] As an achievable manner, the acquisition time corresponding to the vehicle posture information, the storage location image coordinate information and the target point coordinate information in the i-th posture state satisfies the following preset relationship:
[0082] ;
[0083] Wherein, i is a positive integer greater than or equal to 1 and less than or equal to N, is the acquisition time of the storage location image in the i-th posture state, is the collection time of the vehicle posture information in the i-th posture state, is the acquisition time of the target point coordinate information in the i-th posture state, where the unit corresponding to the acquisition time is second.
[0084] The above preset relationship is used to achieve time synchronization among the vehicle posture information, the storage location image coordinate information and the target point coordinate information.
[0085] Step S140, calculating a camera extrinsic parameter matrix in a camera-to-vehicle coordinate system according to the vehicle posture information and the storage location image coordinate information;
[0086] As an implementable manner, the step S140 of calculating the camera extrinsic parameter matrix from the camera to the vehicle coordinate system according to the vehicle posture information and the storage location image coordinate information specifically includes the following sub-steps:
[0087] Calculate the relative coordinates of all the inner corner points of the target storage location to the corner points in the vehicle coordinate system in each posture state according to the vehicle posture information and the storage location image coordinate information;
[0088] Constructing a pixel coordinate matrix formed by the pixel coordinates of the corner points in all posture states in a pixel coordinate system according to a first preset order, and constructing a first vehicle coordinate matrix formed by the relative coordinates of the corner points in all posture states in a vehicle coordinate system according to the first preset order;
[0089] The camera rotation and translation matrix of the pixel coordinate matrix relative to the first vehicle coordinate matrix is calculated by a pnp algorithm, and the camera rotation and translation matrix forms a camera extrinsic parameter matrix in the camera-to-vehicle coordinate system.
[0090] As an achievable method, the sub-step of the above step S140 calculates the relative coordinates of all the inner corner points of the target storage location in each posture state to the corner points in the vehicle coordinate system according to the vehicle posture information and the storage location image coordinate information, specifically including the following contents:
[0091] The relative coordinates of one of the inner corners of the target location in the i-th position are calculated according to the following formula: :
[0092]
[0093]
[0094]
[0095] in, is the horizontal coordinate of one of the inner corner points in the i-th posture state based on the UTM coordinate system, is the ordinate of one of the inner corner points in the i-th posture state based on the UTM coordinate system, is the horizontal coordinate of the target vehicle based on the UTM coordinate system in the i-th posture state, is the ordinate of the target vehicle based on the UTM coordinate system in the i-th posture state, is the orientation angle of the target vehicle based on the UTM coordinate system in the i-th posture state.
[0096] Specifically, for the case where the target parking space is a vehicle parking space, the coordinates of the four inner corner points of the target parking space in the i-th posture state based on the UTM coordinate system are: left_top: ( )、right_top:( )、right_down:( )、left_down:( );
[0097] The relative coordinates of the four inner corner points of the target location based on the vehicle coordinate system in the i-th posture state are: left_top: 、right_top: 、right_down: 、left_down:
[0098] but:
[0099]
[0100]
[0101]
[0102]
[0103]
[0104]
[0105]
[0106]
[0107]
[0108]
[0109]
[0110]
[0111] The vehicle coordinate system is a vehicle coordinate system with the center of the rear axle of the target vehicle as the origin, the right as the positive x direction, and the forward as the positive y direction.
[0112] Furthermore, the principle of the pnp algorithm is as follows:
[0113]
[0114] Among them, u is the horizontal coordinate based on the pixel coordinate system, and v is the vertical coordinate based on the pixel coordinate system. , They are the horizontal focal length and the vertical focal length in the internal parameter matrix, , are the pixel coordinates at the intersection of the optical axis and the imaging plane, is the rotation and translation matrix, , , is the translation amount in the X, Y, and Z axis directions of the vehicle coordinate system. The camera rotation and translation matrix includes the camera rotation matrix and the camera translation matrix, where: is the camera rotation matrix, is the camera translation matrix.
[0115] Step S150: calculating the millimeter wave radar extrinsic parameter matrix of the millimeter wave radar in the vehicle coordinate system based on the calculation of the camera extrinsic parameter matrix and according to the target point coordinate information.
[0116] As an implementable manner, the step S150 of calculating the millimeter wave radar extrinsic parameter matrix of the millimeter wave radar in the vehicle coordinate system based on the target point coordinate information based on the calculation of the camera extrinsic parameter matrix specifically includes the following sub-steps:
[0117] Obtaining the relative coordinates of the target point to the target point in the vehicle coordinate system;
[0118] Obtain the number of millimeter-wave radars, where the number of millimeter-wave radars is M, and M is a positive integer;
[0119] Obtaining the coordinate information of the target point detected by each millimeter-wave radar in each posture state;
[0120] Constructing M millimeter-wave coordinate matrices corresponding to the M millimeter-wave radars in a millimeter-wave radar coordinate system, which are formed by the coordinate information of the target points in all posture states detected by each millimeter-wave radar in a millimeter-wave radar coordinate system in a second preset order, and constructing a second vehicle coordinate matrix formed by the relative coordinates of the target points in all posture states in a vehicle coordinate system according to the second preset order;
[0121] Calculating the millimeter wave rotation and translation matrix of the millimeter wave coordinate matrix relative to each of the second vehicle coordinate matrices by the least square method, the M millimeter wave rotation and translation matrices being the millimeter wave radar extrinsic parameter matrices corresponding to the M millimeter wave radars in the vehicle coordinate system;
[0122] Each of the millimeter wave rotation and translation matrices includes a millimeter wave rotation matrix and a millimeter wave translation matrix.
[0123] Specifically, the target vehicle may be equipped with one millimeter-wave radar, or may be configured with two or more. The coordinates of the target point detected by the millimeter-wave radar based on the millimeter-wave radar coordinate system can be projected into the rectangular coordinate system to display the target detected by the millimeter-wave radar in a plane. The corner reflector target point corresponding to each millimeter-wave radar can be selected by manual box selection, or it can be screened according to the scattering cross-sectional area RCS, coordinate position and other characteristics of the corner reflector to distinguish the displayed corner reflector target points.
[0124] As an implementable manner, the first preset order includes an arrangement order of posture states and an arrangement order of all inner corner points in each posture state, and the second preset order corresponds to the arrangement order of posture states in the first preset order.
[0125] The first preset order is set to make the elements in the pixel coordinate matrix and the first vehicle coordinate matrix correspond one-to-one according to the posture state order and the point position order, and the second preset order is set to make the elements in the millimeter wave coordinate matrix and the second vehicle coordinate matrix correspond one-to-one according to the posture state order.
[0126] The present invention calculates the correlation between the relative coordinates of all inner corner points of the target storage location to the corner points in the vehicle coordinate system and the pixel coordinates of all inner corner points to the corner points in the pixel coordinate system to obtain a camera extrinsic parameter matrix, and then randomly selects an inner corner point from the inner corner points as a target point, and places a corner reflector at the target point to limit the position of the corner reflector, detects the corner reflector by a millimeter-wave radar to obtain the target point coordinates of the target point where the corner reflector is located to the millimeter-wave radar coordinate system, thereby calculating the correlation between the target point coordinates of the corner reflector in the millimeter-wave radar coordinate system and the relative coordinates of the target point to the vehicle coordinate system (which have been obtained in the camera extrinsic parameter matrix calculation and can be directly obtained) to obtain the millimeter-wave radar extrinsic parameter matrix.
[0127] An embodiment of the present invention proposes a semi-automatic calibration method for camera and millimeter-wave radar extrinsic parameters. The method obtains target storage location information of a driving school site where a target vehicle is located, wherein the target storage location information includes the UTM coordinates of several inner corner points constituting the target storage location. An inner corner point is randomly selected as a target point. After a corner reflector is placed at the target point, the target vehicle is directed toward the target storage location in different posture states. The vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter-wave radar are obtained. Then, the camera extrinsic parameter matrix in the camera-to-vehicle coordinate system is calculated based on the vehicle posture information and the storage location image coordinate information. The millimeter-wave radar extrinsic parameter matrix in the millimeter-wave radar-to-vehicle coordinate system is calculated based on the target point coordinate information on the basis of the calculation of the camera extrinsic parameter matrix. The embodiment of the present invention makes full use of the autonomous driving environment of a driving school to calibrate the camera and millimeter-wave radar external parameters. By randomly selecting one of all the inner corner points of the target location as the target point, and placing a corner reflector at the target point to limit the position of the corner reflector, there is no need for additional complex relationships to associate the calculation process of the camera external parameters and the millimeter-wave radar external parameters, which reduces the calculation steps. By collecting data information in multiple different posture states to achieve external parameter calibration, it replaces the joint calibration and calibration room calibration methods used in the prior art, avoids dependence on the main sensor such as lidar and is limited by the fixed and closed environment of the calibration room, and can achieve the technical effect of improving calibration accuracy while improving calibration efficiency.
[0128] Embodiment 2
[0129] See also Figure 4 , Figure 4 The electronic device of the present invention is a schematic diagram of an embodiment of the electronic device, including:
[0130] A memory 401, a processor 402, and a computer program 403 stored in the memory and executable on the processor, wherein the processor implements the above-mentioned camera and millimeter-wave radar extrinsic parameter semi-automatic calibration method when executing the computer program 403 stored in the memory.
[0131] For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the part of the semi-automatic calibration method of camera and millimeter-wave radar extrinsic parameters in the embodiment of the present invention. The memory 401 can be used to store a computer program 403, which includes software programs, modules and data. The processor 402 executes the computer program 403 stored in the memory 401 to perform various functional applications and data processing of the electronic device.
[0132] Embodiment 3
[0133] The present invention also provides a computer-readable storage medium. Figure 5 , Figure 5 The present invention is a schematic diagram of an embodiment of a computer-readable storage medium in an embodiment of the present invention, wherein the computer-readable storage medium may store a computer program, which, when executed, includes some or all of the steps of the method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters recorded in the above method embodiment.
[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, electronic devices, and computer-readable storage media can refer to the corresponding processes of the semi-automatic calibration method of camera and millimeter-wave radar extrinsic parameters in the aforementioned method embodiment, and will not be repeated here.
[0135] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.
[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the semi-automatic calibration method of the camera and millimeter-wave radar external parameters in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0139] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters, used for calibrating the camera and millimeter-wave radar of a target vehicle, characterized in that: The calibration method comprises: Obtaining target storage location information of the driving school site where the target vehicle is located, the target storage location information including coordinates of several inner corner points constituting the target storage location based on the UTM coordinate system; Randomly select one of the inner corner points as the target point; After placing a corner reflector at the target point, the target vehicle is directed toward the target storage location in different posture states, and the vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter wave radar are obtained; Calculating a camera extrinsic parameter matrix from the camera to the vehicle coordinate system according to the vehicle posture information and the storage location image coordinate information; Based on the calculation of the camera extrinsic matrix, the millimeter wave radar extrinsic matrix in the vehicle coordinate system is calculated according to the target point coordinate information; The method of obtaining the vehicle posture information of the target vehicle in each posture state, the storage location image coordinate information collected by the camera, and the target point coordinate information detected by the millimeter wave radar includes: obtaining N different posture states of the target vehicle, where N is a positive integer greater than or equal to 3; collecting the vehicle posture information of the target vehicle in each posture state, and the vehicle posture information includes the position coordinates and orientation angle of the target vehicle based on the UTM coordinate system; collecting the storage location image in each posture state by the camera, and the storage location image is an image containing the target storage location collected by the camera; processing the storage location image to obtain the storage location image coordinate information, and the storage location image coordinate information includes the corner point pixel coordinates of several inner corner points of the target storage location in the storage location image based on the pixel coordinate system; detecting the corner reflector by the millimeter wave radar to obtain the target point coordinate information of the corner reflector in each posture state, and the target point coordinate information is the target point coordinates of the target point detected by the millimeter wave radar based on the millimeter wave radar coordinate system; The method of calculating the camera extrinsic parameter matrix from the camera to the vehicle coordinate system based on the vehicle posture information and the storage location image coordinate information includes: calculating the relative coordinates of all the inner corner points of the target storage location in each posture state to the corner points in the vehicle coordinate system based on the vehicle posture information and the storage location image coordinate information; constructing a pixel coordinate matrix in the pixel coordinate system formed by the pixel coordinates of the corner points in all posture states in a first preset order, and constructing a first vehicle coordinate matrix in the vehicle coordinate system formed by the relative coordinates of the corner points in all posture states according to the first preset order; calculating the camera rotation and translation matrix of the pixel coordinate matrix relative to the first vehicle coordinate matrix by a pnp algorithm, and the camera rotation and translation matrix is the camera extrinsic parameter matrix from the camera to the vehicle coordinate system; The method of calculating the millimeter-wave radar external parameter matrix from the millimeter-wave radar to the vehicle coordinate system according to the target point coordinate information based on the calculation of the camera external parameter matrix includes: obtaining the relative coordinates of the target point to the vehicle coordinate system; obtaining the number of millimeter-wave radars, the number of millimeter-wave radars is M, and M is a positive integer; obtaining the coordinate information of the target point detected by each millimeter-wave radar in each posture state; constructing M millimeter-wave coordinate matrices corresponding to the M millimeter-wave radars formed in a second preset order by the target point coordinate information in all posture states detected by each millimeter-wave radar in the millimeter-wave radar coordinate system, and constructing a second vehicle coordinate matrix formed by the relative coordinates of the target points in all posture states in the vehicle coordinate system according to the second preset order; calculating the millimeter-wave rotation and translation matrix of the millimeter-wave coordinate matrix relative to each of the second vehicle coordinate matrices by the least squares method, and the M millimeter-wave rotation and translation matrices are the corresponding M millimeter-wave radar external parameter matrices from the millimeter-wave radar to the vehicle coordinate system.
2. The method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters according to claim 1, characterized in that: The step of processing the storage location image to obtain storage location image coordinate information includes: Detecting and marking each of the storage location images according to the inner corner points of the target storage location to obtain marked pixel corner points; Calculate the pixel coordinates of each pixel corner point in the storage location image according to the pixel coordinate system; The pixel coordinates of a plurality of corner points constitute the storage location image coordinate information.
3. The method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters according to claim 2, characterized in that: The acquisition time corresponding to the vehicle posture information, the storage location image coordinate information and the target point coordinate information in the i-th posture state satisfies the following preset relationship: |T imagei -T locationi |<0.05&&|T imagei -T radari |<0.05; Where i is a positive integer greater than or equal to 1 and less than or equal to N, T imagei is the acquisition time of the storage location image in the i-th posture state, T locationi is the collection time of the vehicle posture information in the i-th posture state, T radari is the time when the coordinate information of the target point is collected in the i-th posture state.
4. The method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters according to claim 3, characterized in that: The step of calculating the relative coordinates of all inner corner points of the target storage location to the corner points in the vehicle coordinate system in each posture state based on the vehicle posture information and the storage location image coordinate information includes: The relative coordinates (X) of one of the inner corners of the target location in the i-th position are calculated according to the following formula: rel_i0 , Y rel_i0 , Z rel_i0 ): X rel_i0 =(X i0 -X cari )*sinθ i -(Y i0 -Y cari )*cosθ i AND rel_i0 (And i0 -AND cari )*sinθ i +(X i0 -X cari )*cosθ i Z rel_i0 =-0.4 Among them, X i0 is the horizontal coordinate of one of the inner corner points in the i-th posture state based on the UTM coordinate system, Y i0 is the ordinate of one of the inner corner points in the i-th posture state based on the UTM coordinate system, X cari is the horizontal coordinate of the target vehicle based on the UTM coordinate system in the i-th posture state, Y cari is the ordinate of the target vehicle based on the UTM coordinate system in the i-th posture state, θ i is the orientation angle of the target vehicle based on the UTM coordinate system in the i-th posture state.
5. The method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters according to claim 4, characterized in that: The first preset order includes an arrangement order of posture states and an arrangement order of all inner corner points in each posture state, and the second preset order corresponds to the arrangement order of posture states in the first preset order.
6. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the camera and millimeter-wave radar extrinsic parameter semi-automatic calibration method as described in any one of claims 1 to 5 when executing the computer program stored in the memory.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for semi-automatic calibration of camera and millimeter-wave radar extrinsic parameters as claimed in any one of claims 1 to 5 is implemented.
