Calibration Method and Device for Joint Calibration of Radar and Camera

Abstract

The embodiments of this specification disclose a calibration method and device for joint calibration of a radar and a camera. The first position information of a first target object at a first moment is obtained through the camera; at the same time, the second position information of the target object is obtained through the radar; if the spatial position distance between the first position information and the second position information is greater than a first threshold, then according to the first position information and the position information of the camera, it is determined whether the distance between the first target object and the position where the camera is located is greater than a second threshold. If so, the device parameters of the camera are adjusted to make the spatial position distance less than the first threshold; if not, the device parameters of the radar are adjusted to make the spatial position distance less than the first threshold, thereby solving the technical problem in the prior art that the joint calibration of the radar and the camera is easily affected by the environment, resulting in difficulty in fusing the position information collected by the radar and the camera and inability to accurately perform joint calibration.

Description

Technical Field

[0001] This application relates to the technical field of intelligent transportation systems, and particularly to a calibration method and device for joint calibration of radar and camera. Background Art

[0002] Roadside devices in intelligent transportation systems can scan and locate vehicles on the road through millimeter-wave radar and cameras. The existing methods for joint calibration of radar and camera are as follows:

[0003] CN202010625105.1 "A Joint Calibration Device and Calibration Method for Multi-line LiDAR and Infrared Camera" introduces a calibration method using a light-emitting regular hexagon joint calibration device. The joint calibration of multi-line LiDAR and infrared camera is achieved by solving the transformation equation constructed by laser point cloud corner points and visual corner points.

[0004] CN201910624873.2 "A Method for Joint Calibration of Millimeter-wave Radar and Camera Based on LM Algorithm" proposes a method for joint calibration of millimeter-wave radar and camera based on LM algorithm. By simultaneously turning on the millimeter-wave radar and camera and placing a calibration object for data acquisition, and then fusing the acquired data and using the LM algorithm to achieve the joint calibration of the coordinate transformation matrix.

[0005] The above existing technologies achieve the purpose of joint calibration by calculating the parameters of the radar and camera. However, due to the complex road environment, the radar and camera are vulnerable to environmental influences. As a result, during actual use, with the increase of usage time, due to installation errors, performance errors, parameter errors, etc., there are large deviations in the information such as the longitude, latitude, speed, and heading angle of the vehicle sensed by the radar and camera, resulting in large deviations in the information such as the longitude and latitude of the vehicle sensed by the radar and camera, making it difficult to fuse the position information of the vehicle sensed by the radar and camera and unable to perform joint calibration. Summary of the Invention

[0006] The calibration method and device for joint calibration of radar and camera provided in the embodiments of this specification can improve the accuracy of the position information of the target object collected by the radar and the position information of the target object collected by the camera, so as to solve the technical problem that the existing joint calibration of radar and camera is affected by the environment, resulting in large deviations between the position information results of the target object collected by the radar and the position information results of the target object collected by the camera during the joint calibration of radar and camera, making it difficult to fuse and unable to accurately perform joint calibration.

[0007] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0008] A calibration method for joint calibration of radar and camera provided in an embodiment of this specification includes:

[0009] Obtain the first position information of the first target object at the first moment through the camera;

[0010] Obtain the second position information of the first target object at the first moment through the radar;

[0011] If the spatial position distance between the first position information and the second position information is greater than a first threshold, then based on the first position information and the position information of the camera, determine whether the distance between the first target object and the position where the camera is located is greater than a second threshold, and obtain a judgment result;

[0012] If the judgment result indicates that the distance between the first target object and the position where the camera is located is greater than the second threshold, then adjust the device parameters of the camera to make the spatial position distance less than the first threshold;

[0013] If the judgment result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold, then adjust the device parameters of the radar to make the spatial position distance less than the first threshold.

[0014] Optionally, the method further includes,

[0015] Store the adjusted device parameters of the radar and the device parameters of the camera corresponding to the adjusted device parameters of the radar in the server.

[0016] Optionally, the method further includes,

[0017] Obtain the third position information of the second target object collected after the radar is adjusted;

[0018] Obtain the fourth position information of the second target object collected after the camera is adjusted;

[0019] Based on the third position information and the fourth position information, obtain the joint calibration result of the position information of the second target object.

[0020] Optionally, the method further includes:

[0021] If the spatial position distance between the first position information at the first moment and the second position information at the second moment is less than the first threshold, then perform joint calibration on the position information of the first target object based on the first position information and the second position information to obtain the position of the first target object.

[0022] Optionally, obtaining the first position information of the first target object at the first moment through the camera specifically includes:

[0023] Establish the correspondence between the pixel coordinate system of the image collected by the camera and the GPS coordinate system;

[0024] Determine the pixel coordinates of the first target object in the pixel coordinate system;

[0025] Determine the first position information of the first target object according to the pixel coordinates of the first target object and the correspondence.

[0026] Optionally, determining the pixel coordinates of the first target object in the pixel coordinate system specifically includes:

[0027] Establish the spatial three-dimensional figure corresponding to the first target object;

[0028] Determine the pixel coordinates corresponding to the geometric center of the spatial three-dimensional figure.

[0029] Optionally, establishing the correspondence between the pixel coordinate system of the image collected by the camera and the GPS coordinate system specifically includes:

[0030] Obtain the image frame collected by the camera;

[0031] Select multiple position calibration points in the image frame;

[0032] Obtain the pixel coordinates corresponding to the multiple position calibration points;

[0033] Obtain the GPS coordinates corresponding to the multiple position calibration points;

[0034] Establish the correspondence between the pixel coordinate system of the image collected by the camera and the GPS coordinate system according to the pixel coordinates of the multiple position calibration points and the GPS coordinates of the multiple position calibration points.

[0035] Optionally, adjusting the device parameters of the camera specifically includes:

[0036] Adjust the pixel coordinates of the position calibration points so that the spatial position distance is less than the first threshold;

[0037] Establish the correspondence between the adjusted pixel coordinate system and the GPS coordinate system according to the adjusted pixel coordinates of the position calibration points and the GPS position information of the position calibration points.

[0038] Optionally, adjusting the device parameters of the radar specifically includes: adjusting the heading angle of the radar so that the spatial position distance is less than the first threshold.

[0039] A real-time calibration device for radar and camera joint calibration provided by an embodiment of this specification includes:

[0040] A first acquisition module, configured to acquire first position information of a first target object at a first moment through a camera;

[0041] A second acquisition module, configured to acquire second position information of the first target object at the first moment through a radar;

[0042] A judgment module, if the spatial position distance between the first position information and the second position information is greater than a first threshold, then according to the first position information and the position information of the camera, it is used to judge whether the distance between the first target object and the position where the camera is located is greater than a second threshold, and a judgment result is obtained;

[0043] A first processing module, if the judgment result indicates that the distance between the first target object and the position where the camera is located is greater than the second threshold, is used to adjust the device parameters of the camera to make the spatial position distance less than the first threshold;

[0044] A second processing module, if the judgment result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold, is used to adjust the device parameters of the radar to make the spatial position distance less than the first threshold.

[0045] An embodiment of this specification achieves the following beneficial effects:

[0046] During the joint calibration process, when the spatial position distance between the first position information of the first target object collected by the camera at the first moment and the second position information of the first target object collected by the radar at the first moment is greater than the first threshold, it is judged whether the distance between the first target object and the position where the camera is located is greater than the second threshold. If the distance between the first target object and the position where the camera is located is greater than the second threshold, it indicates that the first target object is far from the camera. At this time, the second position information collected by the radar is more accurate than the first position information collected by the camera. Therefore, during the joint calibration process, taking the second position information collected by the radar as the standard, the relevant parameters of the camera are adjusted to make the spatial position distance between the adjusted first position information collected by the camera and the second position information less than the first threshold. The adjusted first position information collected by the camera and the second position information can be used for joint calibration of target objects with the same position information as the first target object, thereby solving the technical problem in the prior art that the radar and the camera are easily affected by the environment, resulting in a large deviation in the vehicle position information collected by the radar and the camera during the actual joint calibration process, making it difficult to fuse the position information of the vehicle collected by the radar and the camera and unable to perform accurate joint calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0048] Figure 1 It is a schematic flow chart of a calibration method for joint calibration of a radar and a camera provided by an embodiment of this specification;

[0049] Figure 2 It is a schematic structural diagram of a real-time calibration device for joint calibration of a radar and a camera provided by an embodiment of this specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] To make the objectives, technical solutions, and advantages of one or more embodiments of this specification clearer, the following will clearly and completely describe the technical solutions of one or more embodiments of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of this specification, rather than all of them. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope protected by one or more embodiments of this specification.

[0051] The following will, in conjunction with the drawings, detail the technical solutions provided by each embodiment of this specification.

[0052] The existing methods for joint calibration of a radar and a camera are as follows:

[0053] CN202010625105.1, "A Joint Calibration Device and Calibration Method for a Multi-line Lidar and an Infrared Camera", introduces a calibration method using a light-emitting regular hexagon joint calibration device. The joint calibration of the multi-line lidar and the infrared camera is achieved by solving the transformation equation constructed by the laser point cloud corner points and the visual corner points.

[0054] CN201910624873.2, "A Method for Joint Calibration of a Millimeter-wave Radar and a Camera Based on the LM Algorithm", proposes a method for joint calibration of a millimeter-wave radar and a camera based on the LM algorithm. By simultaneously turning on the millimeter-wave radar and the camera and placing a calibration object for data acquisition, and then fusing the acquired data and using the LM algorithm to achieve the joint calibration of the coordinate transformation matrix.

[0055] The above-mentioned prior art calculates the parameters of the radar and the camera to achieve the purpose of joint calibration. However, due to the complex road environment, the radar and the camera are vulnerable to environmental influences. As a result, during the actual use process, with the increase in the usage time, due to installation errors, performance errors, parameter errors, etc., there are significant deviations in the information such as the longitude, latitude, speed, and heading angle of the vehicle collected by the radar and the camera. This leads to significant deviations in the information such as the longitude and latitude of the vehicle sensed by the radar and the camera, making it difficult to fuse the position information of the vehicle collected by the radar and the camera.

[0056] To solve the defects in the prior art, the following embodiments are given in the embodiments of this specification:

[0057] Figure 1 It is a schematic flowchart of a calibration method for joint calibration of a radar and a camera provided by the embodiments of this specification. From the perspective of the device, the execution subject of this process can be a server. As Figure 1 shown, this process may include the following steps:

[0058] Step 110: Obtain the first position information of the first target object at the first moment through the camera;

[0059] In the embodiments of this specification, the first target object may be an object such as a vehicle or a pedestrian within a specified area; the position information of the first target object within the specified area can be collected in real time through the camera; obtaining the first position information of the first target object at the first moment through the camera specifically includes collecting the image frame corresponding to the first moment, establishing the correspondence between the pixel coordinate system where the image frame is located and the GPS coordinate system, determining the pixel coordinates of the first target object in the pixel coordinate system, and determining the GPS position information of the first target object according to the pixel coordinates of the first target object and the above correspondence, so as to obtain the first position information of the first target object at the first moment.

[0060] Step 120: Obtain the second position information of the first target object at the first moment through the radar.

[0061] In the embodiments of this specification, while the camera collects the position information of the first target object at the first moment, the radar is used to collect the position information of the first target object, and this position information is the second position information of the first target object.

[0062] Step 130: If the spatial position distance between the first position information and the second position information is greater than the first threshold, then judge whether the distance between the first target object and the position where the camera is located is greater than the second threshold according to the first position information and the position information of the camera, and obtain a judgment result.

[0063] In the embodiments of this specification, the first threshold is the spatial position distance between the position information of the first target object collected by the camera and the position information of the first target object collected by the radar. Specifically, this spatial position distance can be represented by the Mahalanobis distance. If the spatial position distance between the first position information and the second position information is greater than the first threshold, it means that the deviation between the first position information and the second position information is relatively large, and it is difficult to fuse them to obtain the result of joint calibration. That the distance between the first target object and the camera is within the second threshold range indicates that the accuracy of the first position information collected by the camera is higher than that of the second position information collected by the radar. Therefore, in joint calibration, the first position information should be used as the standard, and the relevant device parameters of the radar should be adjusted to adjust the second position information so that the first position information and the adjusted second position information can be fused, and then the joint calibration result of the first target object and the second target object can be obtained. Specifically, the value range of the second threshold is related to the actual usage scenarios of the radar and the camera used for joint calibration, and no specific limitation is made.

[0064] Step 140: If the judgment result indicates that the distance between the first target object and the position where the camera is located is greater than the second threshold, adjust the device parameters of the camera to make the spatial position distance less than the first threshold.

[0065] In the embodiments of this specification, step 140 may specifically include: If the distance between the first target object and the position where the camera is located is greater than the second threshold, it indicates that the first target object is far from the camera, and the first target object is outside the range where the camera can accurately determine the position information of the target object. At this time, the position information of the first target object collected by the radar is relatively accurate. Therefore, using the second position information of the first target object collected by the radar as the standard, adjust the relevant parameters of the camera to make the spatial position distance between the first position information collected by the adjusted camera and the second position information collected by the radar less than the first threshold, and record the corresponding radar parameters and camera parameters of the first target object in the server at this time, so that when jointly calibrating a target object with the same position information as the first target object later, the corresponding radar and camera parameters can be directly called, thereby improving the efficiency of joint calibration.

[0066] Step 150: If the judgment result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold, adjust the device parameters of the camera to make the spatial position distance less than the first threshold.

[0067] In the embodiments of this specification, step 150 may specifically include: If the judgment result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold, it means that the first target object is relatively close to the camera. Within the range where the camera can accurately determine the position information of the target object, at this time, the position information of the first target object collected by the camera is relatively accurate. Therefore, based on the first position information of the first target object collected by the camera, the relevant parameters of the radar are adjusted so that the spatial position distance between the second position information collected by the adjusted radar and the first position information collected by the camera is less than the first threshold. And record the radar parameters and camera parameters corresponding to the first target object at this time in the server, so that when jointly calibrating a target object with the same position information as the first target object later, the parameters corresponding to the radar and the camera can be directly called, thereby improving the efficiency of joint calibration.

[0068] Figure 2 In the method, during the joint calibration process, when the spatial position distance between the first position information of the first target object collected by the camera at the first moment and the second position information of the first target object collected by the radar at the first moment is greater than the first threshold, it is judged whether the distance between the first target object and the position where the camera is located is greater than the second threshold. If the distance between the first target object and the position where the camera is located is greater than the second threshold, it means that the first target object is relatively far from the camera. At this time, the second position information collected by the radar is more accurate than the first position information collected by the camera. Therefore, during the joint calibration process, based on the second position information collected by the radar, the relevant parameters of the camera are adjusted so that the spatial position distance between the first position information collected by the adjusted camera and the second position information is less than the first threshold. The first position information and the second position information collected by the adjusted camera can be used to jointly calibrate a target object with the same position information as the first target object, thereby solving the technical problem in the prior art that the radar and the camera are easily affected by the environment, resulting in large deviations in the information such as the longitude, latitude, speed, and heading angle of the vehicle collected by the radar and the camera due to installation errors, performance errors, parameter errors, etc. during the actual joint calibration process, resulting in large deviations in the information such as the longitude and latitude of the vehicle sensed by the radar and the camera, and making it difficult to fuse the position information of the vehicle collected by the radar and the camera.

[0069] Based on Figure 2 In the method, the embodiments of this specification also provide some specific implementation schemes of this method, which are described below.

[0070] In the embodiments of this specification, the adjusted device parameters of the radar and the device parameters of the camera corresponding to the adjusted device parameters of the radar are stored in the server correspondingly, which specifically includes: when the spatial position distance between the second position information of the first target object collected by the adjusted radar and the first position information is less than the first threshold, the adjusted radar parameters and the corresponding camera parameters at this time are recorded in the server, so that when subsequently jointly calibrating other target objects with the same position information as the first target object, the device parameters corresponding to the relevant devices can be directly extracted, and the efficiency of joint calibration can be improved.

[0071] In the embodiments of this specification, after the radar and the camera are adjusted, if the spatial position distance between the third position information collected by the radar and the fourth position information collected by the camera is less than the first threshold, then the third position information of the second target object collected after the radar is adjusted is obtained; the fourth position information of the second target object collected after the camera is adjusted is obtained; based on the third position information and the fourth position information, the joint calibration result of the position information of the second target object is obtained.

[0072] In the embodiments of this specification, if the spatial position distance between the first position information at the first moment and the second position information at the second moment is less than the first threshold, it indicates that the first position information collected by the camera and the second position information collected by the radar at this time can be used for joint calibration. Therefore, the position information of the first target object can be jointly calibrated based on the first position information and the second position information to obtain the position of the first target object.

[0073] In the embodiments of this specification, obtaining the first position information of the first target object at the first moment through the camera specifically includes: establishing the corresponding relationship between the pixel coordinate system of the image collected by the camera and the GPS coordinate system; determining the pixel coordinates of the first target object in the pixel coordinate system; according to the pixel coordinates of the first target object and the corresponding relationship, determining the first position information of the first target object.

[0074] In the embodiments of this specification, determining the pixel coordinates of the first target object in the pixel coordinate system specifically includes: establishing a spatial three-dimensional figure corresponding to the first target object, and usually a cuboid is used to represent the vehicle in the above pixel coordinate system; determining the pixel coordinates corresponding to the geometric center of the spatial three-dimensional figure as the pixel coordinates of the first target object.

[0075] In the embodiments of this specification, establishing the correspondence between the pixel coordinate system of the image captured by the camera and the GPS coordinate system specifically includes: obtaining the image frame captured by the camera; selecting multiple position calibration points in the image frame; respectively obtaining the pixel coordinates corresponding to the multiple position calibration points; respectively obtaining the GPS coordinates corresponding to the multiple position calibration points; and establishing the correspondence between the pixel coordinate system of the image captured by the camera and the GPS coordinate system according to the pixel coordinates of the multiple position calibration points and the GPS coordinates of the multiple position calibration points.

[0076] In the embodiments of this specification, adjusting the device parameters of the camera, in essence, is to adjust the correspondence between the pixel coordinate system where the image frame captured by the camera is located and the GPS coordinate system, so as to make the GPS coordinates of the pixel points determined according to the adjusted correspondence more accurate; the adjustment process specifically includes: adjusting the pixel coordinates of the position calibration points in the image frame, where the number of position calibration points is usually 4 to 6. Specifically, when the spatial position distance between the first position information determined by using the adjusted correspondence and the second position information collected by the radar is less than the first threshold by adjusting the pixel coordinates of one or more of the position calibration points, there is no need to adjust the coordinates of the position calibration points anymore. Establish the correspondence between the adjusted pixel coordinate system and the GPS coordinate system according to the adjusted pixel coordinates of the position calibration points and the GPS position information of the position calibration points.

[0077] In the embodiments of this specification, adjusting the device parameters of the radar specifically includes: adjusting the heading angle of the radar to make the spatial position distance less than the first threshold, and subsequently, the position of the target object can be jointly calibrated by using the position information of the target object collected by the adjusted radar and the position information of the target object captured by the camera.

[0078] In the embodiments of the present specification, during the joint calibration process, when the spatial position distance between the first position information of the first target object collected by the camera at the first moment and the second position information of the first target object collected by the radar at the first moment is greater than the first threshold, it is determined whether the distance between the first target object and the position where the camera is located is greater than the second threshold. If the distance between the first target object and the position where the camera is located is greater than the second threshold, it indicates that the first target object is far from the camera. At this time, the second position information collected by the radar is more accurate than the first position information collected by the camera. Therefore, during the joint calibration process, the relevant parameters of the camera are adjusted based on the second position information collected by the radar, so that the spatial position distance between the first position information collected by the adjusted camera and the second position information is less than the first threshold. The first position information and the second position information collected by the adjusted camera can be used to perform joint calibration on the target object with the same position information as the first target object, thereby solving the technical problem in the prior art that the radar and the camera are easily affected by the environment, resulting in large deviations in the information such as the longitude, latitude, speed, and heading angle of the vehicle collected by the radar and the camera due to installation errors, performance errors, parameter errors, etc. during the actual joint calibration process, resulting in large deviations in the information such as the longitude and latitude of the vehicle sensed by the radar and the camera, and making it difficult to fuse the position information of the vehicle collected by the radar and the camera and unable to accurately perform joint calibration.

[0079] Based on the same idea, the embodiments of the present specification also provide a device corresponding to the above method. Figure 2 provided by the embodiments of the present specification corresponding to Figure 1 a structural schematic diagram of a real-time calibration device for joint calibration of a radar and a camera corresponding to the method in Figure 2 As shown in

[0080] The first acquisition module 210 is configured to acquire the first position information of the first target object at the first moment through the camera; the second acquisition module 220 is configured to acquire the second position information of the first target object at the first moment through the radar; the judgment module 230 is configured to judge whether the distance between the first target object and the position where the camera is located is greater than the second threshold according to the first position information and the position information of the camera if the spatial position distance between the first position information and the second position information is greater than the first threshold, and obtain a judgment result; the first processing module 240 is configured to adjust the device parameters of the camera to make the spatial position distance less than the first threshold if the judgment result indicates that the distance between the first target object and the position where the camera is located is greater than the second threshold; the second processing module 250 is configured to adjust the device parameters of the radar to make the spatial position distance less than the first threshold if the judgment result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold.

[0081] Based on Figure 2For the device in [description], the embodiments of this specification also provide some specific implementation solutions of the device, which will be described below.

[0082] Optionally, the device is further configured to store the adjusted device parameters of the radar and the device parameters of the camera corresponding to the adjusted device parameters of the radar in the server.

[0083] Optionally, the first acquisition module 210 is configured to acquire the third position information of the second target object collected after the radar is adjusted; optionally, the second acquisition module 220 is configured to acquire the fourth position information of the second target object collected after the camera is adjusted; based on the third position information and the fourth position information, obtain the joint calibration result of the position information of the second target object.

[0084] Optionally, if the spatial position distance between the first position information at the first moment and the second position information at the second moment is less than the first threshold, the device is configured to perform joint calibration on the position information of the first target object based on the first position information and the second position information to obtain the position of the first target object.

[0085] Optionally, the first acquisition module 210 is configured to acquire the first position information of the first target object at the first moment through the camera, specifically including: establishing the correspondence between the pixel coordinate system of the image collected by the camera and the GPS coordinate system; determining the pixel coordinates of the first target object in the pixel coordinate system; according to the pixel coordinates of the first target object and the correspondence, determining the first position information of the first target object.

[0086] Optionally, the first acquisition module 210 is configured to determine the pixel coordinates of the first target object in the pixel coordinate system, specifically including: establishing the spatial three-dimensional graph corresponding to the first target object; determining the pixel coordinates corresponding to the geometric center of the spatial three-dimensional graph.

[0087] Optionally, the first acquisition module 210 is configured to establish the correspondence between the pixel coordinate system of the image collected by the camera and the GPS coordinate system, specifically including: acquiring the image frame collected by the camera; selecting multiple position calibration points in the image frame; acquiring the pixel coordinates corresponding to the multiple position calibration points; acquiring the GPS coordinates corresponding to the multiple position calibration points; according to the pixel coordinates of the multiple position calibration points and the GPS coordinates of the multiple position calibration points, establishing the correspondence between the pixel coordinate system of the image collected by the camera and the GPS coordinate system.

[0088] Optionally, the first processing module 240 is configured to adjust the device parameters of the camera, specifically including: adjusting the pixel coordinates of the position calibration points to make the spatial position distance less than the first threshold; according to the adjusted pixel coordinates of the position calibration points and the GPS position information of the position calibration points, establishing the correspondence between the adjusted pixel coordinate system and the GPS coordinate system.

[0089] Optionally, the second processing module 250 is configured to adjust the device parameters of the radar, specifically including: adjusting the heading angle of the radar to make the spatial position distance less than the first threshold.

[0090] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method.

[0091] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiment.

[0092] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logical function is determined by the user programming the device. Designers can program themselves to "integrate" a digital character system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL), and there is not just one type of HDL, but many types, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into an integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0093] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0094] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0095] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0096] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block of the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more boxes or blocks.

[0098] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more boxes or blocks.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more boxes or blocks.

[0100] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0101] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0102] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0105] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0106] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A calibration method for joint calibration of radar and camera, characterized in that, Including: Obtaining first position information of a first target object at a first moment through a camera; Obtaining second position information of the first target object at the first moment through a radar; If the spatial position distance between the first position information and the second position information is greater than a first threshold, then based on the first position information and the position information of the camera, determining whether the distance between the first target object and the position where the camera is located is greater than a second threshold to obtain a determination result; If the determination result indicates that the distance between the first target object and the position where the camera is located is greater than the second threshold, then adjusting device parameters of the camera to make the spatial position distance less than the first threshold; If the determination result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold, then adjusting device parameters of the radar to make the spatial position distance less than the first threshold.

2. The method according to claim 1, characterized in that The method further includes storing the adjusted device parameters of the radar and the device parameters of the camera corresponding to the adjusted device parameters of the radar in a server.

3. The method according to claim 2, wherein The method further includes Obtaining third position information of a second target object collected after the radar is adjusted; Obtaining fourth position information of the second target object collected after the camera is adjusted; Based on the third position information and the fourth position information, obtaining a joint calibration result of the position information of the second target object.

4. The method according to claim 1, characterized in that The method further includes: If the spatial position distance between the first position information at the first moment and the second position information at a second moment is less than the first threshold, then jointly calibrating the position information of the first target object based on the first position information and the second position information to obtain the position of the first target object.

5. The method according to claim 1, characterized in that, The method according to claim 1, wherein the obtaining first position information of a first target object at a first moment through a camera specifically includes: Establishing a correspondence relationship between a pixel coordinate system of an image collected by the camera and a GPS coordinate system; Determining pixel coordinates of the first target object within the pixel coordinate system; Based on the pixel coordinates of the first target object and the correspondence relationship, determining the first position information of the first target object.

6. The method according to claim 5, characterized in that, The determining pixel coordinates of the first target object within the pixel coordinate system specifically includes: Establishing a spatial three-dimensional figure corresponding to the first target object; Determining pixel coordinates corresponding to the geometric center of the spatial three-dimensional figure.

7. The method according to claim 5, wherein The establishing a correspondence relationship between a pixel coordinate system of an image collected by the camera and a GPS coordinate system specifically includes: Obtaining an image frame collected by the camera; Selecting a plurality of position calibration points in the image frame; Obtaining pixel coordinates corresponding to the plurality of position calibration points; Obtaining GPS coordinates corresponding to the plurality of position calibration points; Based on the pixel coordinates of the plurality of position calibration points and the GPS coordinates of the plurality of position calibration points, establishing a correspondence relationship between a pixel coordinate system of an image collected by the camera and a GPS coordinate system.

8. The method according to claim 7, wherein Adjusting the device parameters of the camera specifically includes: Adjusting the pixel coordinates of the position calibration point to make the spatial position distance less than the first threshold; According to the adjusted pixel coordinates of the position calibration point and the GPS position information of the position calibration point, establishing the corresponding relationship between the adjusted pixel coordinate system and the GPS coordinate system.

9. The method according to claim 1, characterized in that Adjusting the device parameters of the radar specifically includes: adjusting the heading angle of the radar to make the spatial position distance less than the first threshold.

10. A real-time calibration device for joint calibration of a radar and a camera, comprising: A first acquisition module, configured to acquire the first position information of the first target object at the first moment through the camera; A second acquisition module, configured to acquire the second position information of the first target object at the first moment through the radar; A judgment module, if the spatial position distance between the first position information and the second position information is greater than the first threshold, then according to the first position information and the position information of the camera, it is used to judge whether the distance between the first target object and the position where the camera is located is greater than the second threshold, and a judgment result is obtained; A first processing module, if the judgment result indicates that the distance between the first target object and the position where the camera is located is greater than the second threshold, is used to adjust the device parameters of the camera to make the spatial position distance less than the first threshold; A second processing module, if the judgment result indicates that the distance between the first target object and the position where the camera is located is not greater than the second threshold, is used to adjust the device parameters of the radar to make the spatial position distance less than the first threshold.

Images