A method and system for spatial calibration of a radar device

Abstract

The application provides a method and system for spatial calibration of a radar device. The method comprises: obtaining an image and radar measurement data at the same time, the radar measurement data comprising radar coordinates of a plurality of landmark points in a geographic region; obtaining geographic coordinates of the landmark points recorded in the image according to pixel coordinates of the landmark points recorded in the image; determining at least two landmark point matching pairs, each landmark point matching pair comprising a geographic coordinate of a landmark point and a radar coordinate of the landmark point; and obtaining a calibration relationship between the radar measurement data and the geographic region according to the at least two landmark point matching pairs. The method can improve the accuracy and efficiency of spatial calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a space calibration method and system of a radar device. BACKGROUND

[0002] At present, a large number of video monitoring cameras are deployed in each region of the city for real-time monitoring. For example, video monitoring cameras are deployed at urban traffic intersections for real-time monitoring of traffic information such as traffic flow and pedestrian flow at the traffic intersection, and with the development of artificial intelligence technology, face recognition, license plate recognition, and vehicle type recognition technologies have been widely applied to automatic monitoring of traffic intersections, greatly improving traffic monitoring efficiency and ensuring the safe and orderly operation of urban traffic. Video cameras have the advantages of high information density and rich color information, but the imaging of video cameras is not clear for distant objects, and cannot image when there is an obstruction. For example, for an electric police camera installed at a traffic intersection, there is a downward viewing angle of about 10° when it is installed. For vehicles that are far away from the camera, the imaging area of the electric police camera is small, resulting in low clarity of the photographed vehicles. In addition, for vehicles of the same type, there will be obvious obstruction in the images captured by the electric police camera. Therefore, in order to solve the shortcomings of video monitoring cameras in target positioning, radar devices (such as millimeter wave radars) are introduced in the field of intelligent traffic to locate targets. Radar devices can directly return the coordinates and speed of targets, and can effectively solve the problem of target obstruction.

[0003] Before being used for target positioning, a radar device needs to be spatially calibrated for the scene it is applied to. For example, when a radar device is used to locate targets in a traffic intersection scene, the traffic intersection needs to be spatially calibrated to determine the correspondence between the radar coordinates of the target in the radar coordinate system and the geographical coordinates of the target in the geographical area. The method of determining the correspondence between the radar coordinates of the target in the radar coordinate system and the geographical coordinates of the target in the geographical area is called a spatial calibration technology of a radar device.

[0004] Existing spatial calibration technology solutions need to determine a landmark point in the space of a geographical area, record the geographical coordinates of the landmark point, and then one-to-one correspond the geographical coordinates of the landmark point with the radar coordinates returned by the radar device to complete spatial calibration. Since the measurement results of a radar device for stationary targets are not accurate and manual recording of the geographical coordinates of the landmark point cannot strictly ensure that the collection time of the radar device is strictly consistent, the calibration is not accurate and the error is large. How to accurately and efficiently calibrate a geographical area is a problem that needs to be solved at present. SUMMARY

[0005] The application provides a space calibration method and system of a radar device, which can ensure that the geographical coordinates and radar coordinates of the same time are obtained, and can improve the accuracy and efficiency of space calibration.

[0006] In a first aspect, a space calibration method of a radar device is provided, which includes: a space calibration system obtaining image and radar measurement data at the same time, the image being obtained by a camera arranged in a geographical area, and the radar measurement data being obtained by a radar device arranged in the geographical area, the radar measurement data including radar coordinates of a plurality of landmark points in the geographical area; the space calibration system obtaining geographical coordinates of the landmark points recorded in the image according to pixel coordinates of the landmark points recorded in the image; the space calibration system determining at least two landmark point matching pairs, each landmark point matching pair including geographical coordinates of a landmark point and radar coordinates of the landmark point; and the space calibration system obtaining a calibration relationship between the radar measurement data and the geographical area according to the at least two landmark point matching pairs.

[0007] In the scheme provided in the application, the space calibration system indirectly calculates the geographical coordinates corresponding to the radar coordinates of the landmark points by using the image captured by the camera as an intermediary. In addition, the image capturing time and the radar measurement time are the same time, which can avoid manual measurement of the geographical coordinates of the landmark points, reduce the workload, and effectively avoid inaccurate measurement of the radar device on the static target by using the image as an intermediary, improve the accuracy of the obtained radar coordinates and geographical coordinates of the landmark points, and further improve the accuracy and efficiency of the space calibration of the radar device.

[0008] In combination with the first aspect, in a possible implementation manner of the first aspect, the space calibration system determines the pixel coordinates of the landmark points in the image according to a landmark point detection algorithm; and the space calibration system obtains the geographical coordinates of the landmark points recorded in the image according to the pixel coordinates of the landmark points and a calibration relationship between the image captured by the camera and the geographical area.

[0009] In the scheme provided in the application, the space calibration system indirectly calculates the geographical coordinates of the landmark points by using the pixel coordinates of the landmark points recorded in the image and the calibration relationship between the image and the geographical area, which can avoid manual measurement of the geographical coordinates of the landmark points and effectively avoid the problem of inaccurate measurement of the radar device on the static target.

[0010] In combination with the first aspect, in a possible implementation manner of the first aspect, the camera and the radar device are independently arranged in the geographical area, the measurement area of the radar device and the capturing area of the camera have an intersection area, and the landmark points corresponding to the at least two landmark point matching pairs are landmark points in the intersection area.

[0011] In the scheme provided in the application, the intersection region of the measurement region of the radar device and the shooting region of the camera can ensure that the camera and the radar device can simultaneously effectively record and measure the same marker point in the intersection region, thereby improving the accuracy of spatial calibration.

[0012] With reference to the first aspect, in a possible implementation manner of the first aspect, the spatial calibration system performs straight line fitting on the geographic coordinates or radar coordinates of all the marker points in the matching pair of marker points, to obtain a fitting straight line; the spatial calibration system calculates the distance from the geographic coordinates or radar coordinates of each marker point to the fitting straight line; and if the number of marker points whose distance to the fitting straight line is less than a first threshold is not greater than a second threshold, it is determined that the matching pair of marker points is valid.

[0013] In the scheme provided in the application, the spatial calibration system determines whether the matching pair of marker points is valid by performing straight line fitting on the geographic coordinates or radar coordinates of all the marker points and judging whether the straight line distribution is satisfied, and the matching pair of marker points is used to calculate the calibration relationship only in the case that the matching pair of marker points does not satisfy the straight line distribution, i.e., the matching pair of marker points is valid, which can ensure the accuracy of the calculated calibration relationship.

[0014] With reference to the first aspect, in a possible implementation manner of the first aspect, the spatial calibration system inputs the image into a marker detection model, and obtains the pixel coordinates of the marker in the image according to the marker detection model; and the spatial calibration system determines the pixel coordinates of the marker point according to the pixel coordinates of the marker in the image.

[0015] In the scheme provided in the application, the spatial calibration system can first determine the pixel coordinates of the marker in the image by using the marker detection model, and then further determine the pixel coordinates of the marker point according to the marker point detection algorithm, without human intervention, thereby improving the efficiency and accuracy of obtaining the pixel coordinates of the marker point.

[0016] With reference to the first aspect, in a possible implementation manner of the first aspect, the spatial calibration system obtains the geographic coordinates of the projection point of the marker point on the ground under the shooting angle of the camera in the image according to the pixel coordinates of the marker point and the calibration relationship between the image shot by the camera and the geographic region; the spatial calibration system obtains the parameters of the camera, the parameters of the camera including the vertical height of the position of the camera in the geographic region from the ground and the geographic coordinates of the vertical projection point of the camera on the ground; and the spatial calibration system determines the geographic coordinates of the marker point according to the geographic coordinates of the projection point of the marker point on the ground under the shooting angle of the camera and the parameters of the camera.

[0017] In the scheme provided in the application, the spatial calibration system cannot directly obtain the geographic coordinates of the marker points, but the geographic coordinates of the marker points have a fixed positional relationship with the geographic coordinates of the marker points recorded in the image. The spatial calibration system can indirectly determine the geographic coordinates of the marker points by obtaining the geographic coordinates of the marker points in the image and the parameters of the camera.

[0018] In combination with the first aspect, in a possible implementation manner of the first aspect, the spatial calibration system sends the calibration relationship between the radar measurement data and the geographic region to the processing device, so that the processing device, after obtaining the radar coordinates of the detected target measured by the radar device, determines the geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target; or the spatial calibration system stores the calibration relationship; when the radar coordinates of the detected target measured by the radar device are obtained, the calibration relationship is obtained, and the geographic coordinates of the detected target in the geographic region are determined according to the calibration relationship and the radar coordinates of the detected target.

[0019] In combination with the first aspect, in a possible implementation manner of the first aspect, the geographic region is a traffic intersection, and the detected target is a vehicle of the traffic intersection measured by the radar device.

[0020] In combination with the first aspect, in a possible implementation manner of the first aspect, the vehicle of the traffic intersection measured by the radar device is a suspicious vehicle, and the spatial calibration system sends the geographic coordinates of the suspicious vehicle to a police system.

[0021] In combination with the first aspect, in a possible implementation manner of the first aspect, the spatial calibration system sends the geographic coordinates of the vehicle to a traffic management system, so that the traffic management system determines a driving trajectory of the vehicle at the traffic intersection according to the geographic coordinates of the vehicle, and further determines a violation behavior of the vehicle according to the driving trajectory.

[0022] In combination with the first aspect, in a possible implementation manner of the first aspect, the detected target is a suspicious person; the spatial calibration system sends the geographic coordinates of the suspicious person to a security management system, so that a security management personnel timely searches for the suspicious person according to the geographic coordinates of the suspicious person.

[0023] In combination with the first aspect, in a possible implementation manner of the first aspect, the detected target is a dangerous target, and the spatial calibration system sends the geographic coordinates of the dangerous target to a dangerous investigation management system, so that a dangerous investigation personnel timely arrives at the region to perform dangerous investigation according to the geographic coordinates of the dangerous target.

[0024] In a second aspect, a spatial calibration system is provided, comprising: an acquisition unit configured to acquire an image and radar measurement data at a same time, the image being captured by a camera arranged in a geographical area, the radar measurement data being measured by a radar device arranged in the geographical area, the radar measurement data comprising radar coordinates of a plurality of landmark points in the geographical area; a coordinate conversion unit configured to obtain geographical coordinates of the landmark points recorded in the image according to pixel coordinates of the landmark points recorded in the image; and a calculation unit configured to determine at least two landmark point matching pairs, each landmark point matching pair comprising geographical coordinates of a landmark point and radar coordinates of the landmark point, and obtain a calibration relationship between the radar measurement data and the geographical area according to the at least two landmark point matching pairs.

[0025] With reference to the second aspect, in a possible implementation manner of the second aspect, the coordinate conversion unit is specifically configured to: determine the pixel coordinates of the landmark points in the image according to a landmark point detection algorithm; and obtain the geographical coordinates of the landmark points recorded in the image according to the pixel coordinates of the landmark points and a calibration relationship between the image captured by the camera and the geographical area.

[0026] With reference to the second aspect, in a possible implementation manner of the second aspect, the camera and the radar device are independently arranged in the geographical area, a measurement area of the radar device and a capturing area of the camera have an intersection area, and the landmark points corresponding to the at least two landmark point matching pairs are landmark points in the intersection area.

[0027] With reference to the second aspect, in a possible implementation manner of the second aspect, the at least two landmark point matching pairs are valid landmark point matching pairs, and the calculation unit is further configured to: perform linear fitting on the geographical coordinates or the radar coordinates of all the landmark points in the at least two landmark point matching pairs to obtain a fitting straight line; calculate a distance from the geographical coordinates or the radar coordinates of each landmark point to the fitting straight line; and determine that the at least two landmark point matching pairs are valid if a number of the geographical coordinates or the radar coordinates of the landmark points having a distance less than a first threshold to the fitting straight line is not greater than a second threshold.

[0028] With reference to the second aspect, in a possible implementation manner of the second aspect, the coordinate conversion unit is specifically configured to: input the image into a landmark point detection model to obtain pixel coordinates of the landmark points in the image according to the landmark point detection model; and determine the pixel coordinates of the landmark points according to the pixel coordinates of the landmark points in the image.

[0029] In a possible implementation manner of the second aspect, the acquisition unit is further configured to acquire the parameters of the camera, the parameters of the camera including a vertical height of a position of the camera in the geographic region from the ground and geographic coordinates of a vertical projection point of the camera on the ground; the coordinate conversion unit is further configured to acquire, according to a calibration relationship between the camera and the geographic region, geographic coordinates of a projection point of the mark point on the ground at a shooting angle of the camera in the image, according to the geographic coordinates of the projection point of the mark point on the ground at the shooting angle of the camera and the parameters of the camera, determine the geographic coordinates of the mark point.

[0030] In a possible implementation manner of the second aspect, the calculation unit is further configured to send the calibration relationship between the radar measurement data and the geographic region to a processing device, so that the processing device, after acquiring radar coordinates of a detected target measured by the radar device, determines geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target; or the calculation unit is further configured to store the calibration relationship, and when the radar coordinates of the detected target measured by the radar device are acquired, acquire the calibration relationship, and determine the geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target.

[0031] In a possible implementation manner of the second aspect, the calculation unit is further configured to send the calibration relationship between the radar measurement data and the geographic region to a processing device, so that the processing device, after acquiring radar coordinates of a detected target measured by the radar device, determines geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target; or the calculation unit is further configured to store the calibration relationship, and when the radar coordinates of the detected target measured by the radar device are acquired, acquire the calibration relationship, and determine the geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target.

[0032] In a possible implementation manner of the second aspect, the calculation unit is further configured to send the calibration relationship between the radar measurement data and the geographic region to a processing device, so that the processing device, after acquiring radar coordinates of a detected target measured by the radar device, determines geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target; or the calculation unit is further configured to store the calibration relationship, and when the radar coordinates of the detected target measured by the radar device are acquired, acquire the calibration relationship, and determine the geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target.

[0033] In a possible implementation manner of the second aspect, the calculation unit is further configured to send the calibration relationship between the radar measurement data and the geographic region to a processing device, so that the processing device, after acquiring radar coordinates of a detected target measured by the radar device, determines geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target; or the calculation unit is further configured to store the calibration relationship, and when the radar coordinates of the detected target measured by the radar device are acquired, acquire the calibration relationship, and determine the geographic coordinates of the detected target in the geographic region according to the calibration relationship and the radar coordinates of the detected target. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A schematic diagram of a system architecture provided by an embodiment of the present application;

[0035] Figure 2A structural schematic diagram of a space calibration system provided by an embodiment of the present application;

[0036] Figure 3 A flowchart of a space calibration method provided by an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a space measurement device provided by an embodiment of the present application;

[0038] Figure 5A A schematic diagram of a calibration device provided by an embodiment of the present application;

[0039] Figure 5B A schematic diagram of another calibration device provided by an embodiment of the present application;

[0040] Figure 6 A perspective transformation schematic diagram provided by an embodiment of the present application;

[0041] Figure 7 A flowchart of a violation detection method provided by an embodiment of the present application;

[0042] Figure 8 A structural schematic diagram of a computing device provided by an embodiment of the present application;

[0043] Figure 9 A structural schematic diagram of a computing device system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings, obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments of the present application.

[0045] The geographic region is a specific region in the physical world, such as a traffic intersection, a traffic road, a cell gate, and the like. The present application provides a spatial calibration method of a radar device, which is performed by a spatial calibration system, and through which the spatial calibration of the radar device can be achieved. The spatial calibration of the radar device specifically refers to calculating the correspondence between the radar coordinates of a point in a geographic region and the geographic coordinates corresponding to the point, which is also referred to as a calibration relationship. The radar measurement data corresponding to the geographic region can be measurement data obtained by a radar device, such as a millimeter wave device, which is independently installed, or can be measurement data obtained by a radar device installed together with a camera, such as a radar-camera integrated machine. It should be understood that each calibration relationship obtained according to the spatial calibration method is the correspondence between the radar measurement data obtained by measuring a geographic region from a fixed position and the space of the geographic region. When the measurement position changes, the calibration relationship also changes. According to the calculated calibration relationship, the radar coordinates of any target in the geographic region can be converted into the geographic coordinates of the target in the geographic region. For example, in a traffic intersection region, a radar device is arranged in the region, and the traffic intersection is spatially calibrated by the method provided in the present application, so that the radar coordinates of a vehicle and other traffic targets measured by the radar device can be converted into the geographic coordinates in the traffic intersection, and the geographic position of the vehicle can be accurately monitored.

[0046] The pixel coordinates in the present application are the coordinates of the pixel points where the targets in the image are located, and the pixel coordinates are two-dimensional coordinates.

[0047] The radar coordinates in the present application are the coordinates of the targets directly measured by the radar device installed at a fixed position, and the radar coordinates are two-dimensional coordinates. In addition, the radar device also returns the speed of the target.

[0048] The geographic coordinates in the present application are three-dimensional coordinate values representing a point in a geographic region. It should be noted that in the physical world, the corresponding coordinate values of the same point in different coordinate systems are different. The geographic coordinates of the point in the present application can be coordinate values in any coordinate system. For example, the geographic coordinates of the target in the present application can be three-dimensional coordinates composed of the longitude, latitude, and altitude corresponding to the target, can be three-dimensional coordinates composed of the X coordinate, Y coordinate, and Z coordinate in the natural coordinate system corresponding to the target, or can be other forms of coordinates, as long as the coordinates can uniquely determine the position of a point in a geographic region. The present application does not limit the specific form of the coordinates.

[0049] As Figure 1As shown, the space calibration system can be deployed in a cloud environment, specifically on one or more computing devices (e.g., a central server) in the cloud environment. The system can also be deployed in an edge environment, specifically on one or more computing devices (edge computing devices) in the edge environment, which can be servers. The cloud environment refers to a cluster of central computing devices owned by a cloud service provider, which provides computing, storage, and communication resources. The edge environment refers to a cluster of edge computing devices that are geographically close to the original data acquisition devices, which provide computing, storage, and communication resources. The original data acquisition devices refer to devices that acquire original data required by the space calibration system, including but not limited to cameras, radars, infrared cameras, magnetic coils, etc. The original data acquisition devices include devices placed at fixed positions on a traffic road to acquire original data (e.g., video data, radar data, infrared data, etc.) in their own perspective, as well as devices that manually acquire data (e.g., real-time kinematic (RTK) spatial measurement devices).

[0050] The space calibration system is used to calculate the calibration relationship based on the original data acquired by the original data acquisition devices, to obtain the correspondence between the images and the geographic area, and the correspondence between the data measured by the radar devices and the geographic area. The units within the space calibration system can be divided in various ways, which are not limited by the present application. Figure 2 For an exemplary division, as shown in Figure 2 As shown, the space calibration system 200 includes an acquisition unit 210, a coordinate conversion unit 220, and a calculation unit 230. The functions of each functional unit are described below.

[0051] The acquisition unit 210 is configured to acquire original data acquired by the original data acquisition devices at the same time, mainly including image data captured by cameras arranged in a geographic area, and radar measurement data measured by radar devices arranged in the geographic area, which includes radar coordinates of multiple points in the geographic area. It should be understood that the image data includes multiple video frames, each frame being an image. Optionally, the acquisition unit 210 outputs the image data acquired by the cameras to the coordinate conversion unit 220.

[0052] The coordinate conversion unit 220 is configured to obtain geographic coordinates of the landmark points recorded in the image based on pixel coordinates of the landmark points recorded in the image. Optionally, the landmark points can be the geometric centers of the landmarks in the image.

[0053] The computing unit 230 is configured to determine at least two landmark point matching pairs, each of which comprises geographical coordinates of a landmark point and radar coordinates of the landmark point, and obtain a calibration relationship between radar measurement data and the geographical region according to the at least two landmark point matching pairs. Through the calibration relationship obtained by calculation, the radar coordinates of a traffic target such as a vehicle can be converted into geographical coordinates.

[0054] In the present application, the spatial calibration system 200 can be a software system, and the deployment forms of the functional units contained therein on hardware devices are flexible, for example, the entire system can be deployed in one or more computing devices in an environment, or can be distributed and deployed in multiple computing devices in different environments.

[0055] It should be understood that the calibration methods of related radar devices all use manual measurement of the geographical coordinates of a target, and then align the radar coordinates of the target returned by the radar device with the manually measured geographical coordinates of the target, so as to calculate the calibration relationship. However, the measurement results obtained by the radar device when measuring a stationary target are not accurate, and it is impossible to ensure that the collection times of the radar coordinates and the geographical coordinates of the target are strictly consistent. Therefore, a large error will be generated when the existing calibration method is used for calibration. Therefore, in order to improve the accuracy and applicability of spatial calibration, the present application provides a spatial calibration method and system for a radar device. The method can determine the pixel coordinates of a corresponding point in an image when or after the radar device measures the radar coordinates of any point in a geographical region, further determine the geographical coordinates of the point by using the calibration relationship between the image and the geographical region, and finally determine the calibration relationship between the radar measurement data and the geographical region by using the radar coordinates and the geographical coordinates of multiple points. The method improves the calibration accuracy and expands the applicable scenarios.

[0056] Please refer to Figure 3 , Figure 3 A flowchart of a spatial calibration method for a radar device provided by an embodiment of the present application. As shown in Figure 3 , the method comprises but is not limited to the following steps:

[0057] Before implementing the method provided by the present application, the calibration relationship between the image captured by the camera deployed in the geographical region and the geographical region needs to be obtained in advance. For example, the geographical coordinates of multiple measurement points are measured in advance by a spatial measurement device such as an RTK device, and then the calibration relationship between the image captured by the camera and the geographical region is calculated according to the pixel coordinates of the measurement points in the image data. The spatial measurement device can select any position in the geographical region to be calibrated as a measurement point.

[0058] In an embodiment of the present application, in order to make the spatial measurement device effectively record the pixel position of the corresponding point of the spatial measurement device in the picture taken by the camera when the camera takes the geographical area, the spatial measurement device can be improved. For example, a marker can be added to the spatial measurement device, so that the geometric center of the marker (i.e. the marker point) and the measurement point (i.e. a point on the ground of the geographical area actually measured by the measurement device) are located in the same vertical direction, that is, the marker point and the measurement point are the same in other measurement values except the height. Or a light source can be added to the spatial measurement device as a marker point.

[0059] In particular, the marker can be specially designed or additionally processed to improve the detection accuracy when detecting the marker point in the image taken by the camera. For example, a spherical marker is selected, a specific color is added to the marker, the marker is made infrared sensitive, etc. to improve the distinction between the marker and other objects, so that the marker can be more easily detected in the image. A point of the marker (for example, the geometric center of the spherical marker) can be further used as a marker point in the image.

[0060] For example, referring to Figure 4 , Figure 4 is a schematic diagram of a spatial measurement device provided by an embodiment of the present application. As shown in Figure 4 , the spatial measurement device 400 includes a connecting rod 401, a measurement device 402 fixed on the connecting rod 401, and a marker 403 fixed on the connecting rod 401, and the connecting rod 401 passes through the geometric center of the marker 403. The marker 403 can be a spherical marker, the axis of the connecting rod 401 passes through the center of the spherical marker, and the measurement device 402 is internally provided with a high-performance processor and a memory, etc. The geographical coordinates of the contact point (i.e. the measurement point) on the ground vertically intersected by the connecting rod 401 can be directly measured, and the geographical coordinates of the measurement point are three-dimensional coordinates, which can be geographical coordinates composed of longitude, latitude and altitude, or three-dimensional coordinates in a natural coordinate system formed according to a geographical area. Different spatial measurement devices can obtain different types of three-dimensional coordinates or the same type of spatial measurement device can also obtain different types of three-dimensional coordinates, and different types of three-dimensional coordinates can be converted into each other. The type of geographical coordinates obtained by the spatial measurement device is not limited in the present application.

[0061] Since the two-dimensional imaging of the spherical marker in the physical space is a circle at any orientation or angle, and the sphere center and the circle center are consistent, the circle center of the circle corresponding to the spherical marker in the image captured by the camera is the sphere center of the spherical marker. By using the sphere center of the spherical marker as the marker point, after the camera captures the image, the circular image corresponding to the spherical marker can be found by using a mature circle detection algorithm, and then the pixel coordinates of the circle center (i.e., the marker point) can be accurately found. Therefore, the selection of the spherical marker can ensure the accuracy of the pixel coordinates of the marker point. Of course, other markers such as ellipsoidal markers can also be selected, and the present application does not limit this.

[0062] In an embodiment, multiple people (or devices) can use multiple space measurement devices carrying markers to perform measurement at the same time, and each image captured by the camera can contain multiple marker points at the same time, and the pixel coordinates of the marker points corresponding to multiple different spatial positions in a geographic region can be obtained at the same time. Of course, one space measurement device carrying a marker can also be used for measurement, and multiple images can be obtained by continuously moving the space measurement device in the camera shooting area, each image containing one marker point. In combination with multiple images, the pixel coordinates of the marker points corresponding to multiple different spatial positions can also be obtained. It should also be understood that multiple cameras can be used for shooting at the same time, and each camera can obtain the pixel coordinates of the marker points corresponding to multiple different spatial positions, thereby being used for the spatial calibration between the images captured by the respective cameras and the geographic region.

[0063] According to the geographic coordinates of the measurement points measured by the space measurement device and the camera parameters, the geographic coordinates of the marker points recorded in the image can be obtained, and according to the image captured by the camera, the pixel coordinates of the marker points can be obtained. After obtaining the pixel coordinates of the marker points and the corresponding geographic coordinates, multiple methods can be used to establish the calibration relationship between the image under the view angle of each camera and the physical world. For example, a homographic transformation matrix H for converting the pixel coordinates to the geographic coordinates can be calculated according to the homographic transformation principle. The homographic transformation formula is (m, n, h) = H*(s, k), (m, n, h) is the geographic coordinates of the marker points in the image, and (s, k) is the pixel coordinates of the marker points in the image. The H matrix corresponding to each camera captured image is different.

[0064] The specific steps of the space calibration method of the radar device provided in the embodiment of the present application are as follows:

[0065] S301: Obtain the image and radar measurement data at the same time.

[0066] Specifically, the space calibration system can obtain a video data captured by a camera arranged in a geographical area, the video data is composed of video frames at different time, the video frames in the video data are arranged in time sequence, each video frame is an image, and one or more markers (or marker points) are recorded in each image. The pixel coordinates of the marker points can be obtained by detecting the markers in each image and then detecting the marker points corresponding to the markers (or only detecting the marker points).

[0067] In addition, in order to ensure that the image and the radar measurement data at the same time can be obtained, it is necessary to ensure that the measurement area of the radar device and the shooting area of the camera have an intersection area, and the above-mentioned markers or marker points are the markers or marker points in the intersection area. For the camera, the effective sensing area is the geographical area close to the camera, and the radar device has a short-range radar and a long-range radar. If the short-range radar is used, the short-range radar can be installed near the camera, and then the geographical area close to the camera is the intersection area. If the long-range radar is used, the long-range radar cannot be installed near the camera, otherwise it will not be able to effectively determine the intersection area, and the long-range radar and the camera need to be installed in different fixed positions. The installation position of the long-range radar and the installation position of the camera have a certain distance, and then the geographical area close to the camera is the intersection area.

[0068] It should be understood that by determining the intersection area, the accuracy of the data collected by the camera and the radar device can be ensured, and the calibration accuracy can be further improved. After the intersection area is determined, the calibration device is moved in the intersection area perpendicular to the road surface or any selected plane in the intersection area, so that the camera and the radar device can synchronously collect sufficient data for space calibration.

[0069] It should be noted that in order to enable the camera and the radar device to accurately and effectively record and measure the marker points, the calibration device can be improved. For example, a marker can be added to the calibration device, so that the geometric center (marker point) of the marker and the contact point of the calibration device and the ground are located in the same vertical direction, or a light source is added to the calibration device as a marker point.

[0070] As Figure 5AAs shown, the calibration device 510 includes a connecting rod 511, a marker 512 fixed on the connecting rod 511, and a corner reflector 513 fixed on the connecting rod 511, the connecting rod 511 passes through the geometric center of the marker 512, the marker 512 is a spherical marker, and the geometric center is the center of the spherical marker. Because the radar device is sensitive to metal, the corner reflector 513 is made of metal. When the radar electromagnetic wave emitted by the radar device scans the corner reflector 513, the electromagnetic wave will be refracted and amplified on the metal corner, generating a very strong echo signal. At the same time, in order to reduce the interference of the radar echo on the corner reflector 513, the connecting rod 511 and the spherical marker 512 are preferably made of non-metal materials, such as plastic and the like. Alternatively, as shown in Figure 5B As shown, the calibration device 520 includes a connecting rod 521 and a metal marker 522 fixed on the connecting rod 521, the connecting rod 521 passes through the geometric center of the marker 522, and the marker 522 is a metal ball. The marker 522 has the function of the corner reflector 513 and reflects the electromagnetic wave signal emitted by the millimeter wave radar, and the connecting rod 521 is made of non-metal material. Figure 5A

[0071] It should be understood that the marker and the corner reflector can also not be connected to the same connecting rod, as long as their relative positions are fixed or their relative positions change but the change is known, and the marker and the corner reflector can also be converted to be processed on the same connecting rod through related processing, the principle is similar and will not be repeated here.

[0072] Optionally, the camera and the radar device can be integrated into one device (such as a radar-camera integrated machine), and then installed at the same fixed position of the geographic area, or the camera and the radar device can be installed at different fixed positions of the geographic area.

[0073] In a possible implementation, the camera carries a timestamp information for each image containing the marker point obtained by photographing the geographic area, and the radar device also carries a timestamp information for each radar coordinate of the marker point returned by measuring the marker point. The space calibration system compares the timestamp information carried by the image with the timestamp information carried by the radar coordinate. If the absolute value of the difference between the timestamp information carried by the image and the timestamp information carried by the radar coordinate is less than a preset threshold, it can be determined that the image acquisition time and the radar coordinate acquisition time are the same time, otherwise, it is considered that the current data is incomplete and needs to be reacquired. The preset threshold can be set according to needs, for example, it can be set to 0.001 seconds.

[0074] ​It can be understood that the space calibration system can correspond the image captured by the camera with the radar coordinate measured by the millimeter wave radar by comparing the timestamp information carried by the image with the timestamp information carried by the radar coordinate, so as to ensure that the image capturing time and the radar coordinate measuring time are the same time.

[0075] S302: Obtain the geographical coordinates of the mark point recorded in the image according to the pixel coordinates of the mark point recorded in the image.

[0076] Specifically, the pixel coordinates of the mark point can be the pixel coordinates of the center point of a two-dimensional object formed by a marker in the image captured by the camera, or the pixel coordinates of an easily identifiable point formed by a light source (such as a laser source) on the marker in the image. The mark point can be found by analyzing and processing the image, so that the pixel coordinates of the mark point can be obtained.

[0077] Optionally, the pixel coordinates of the mark point in the image are determined according to a mark point detection algorithm.

[0078] For example, when the pixel coordinates of the mark point are the pixel coordinates of the center point of a two-dimensional object formed by a marker in the image captured by the camera, the method for finding the pixel coordinates of the mark point by analyzing and processing the image can be specifically as follows:

[0079] The image is detected by a corresponding marker detection algorithm to determine the position of the marker in the image. At present, the marker detection algorithm in the field of computer vision is very mature, and various detection algorithms have been formed, which can be selected according to actual needs. For example, yolo, faster-rcnn and other deep learning models can be used for marker detection.

[0080] It should be understood that for the case where the marker detection algorithm is a deep learning model, the deep learning model needs to be trained. The trained deep learning model has the ability to detect markers. When any image containing a marker is input to the deep learning model, the deep learning model can detect the position of the marker, and mark the detected marker with a bounding box (such as a rectangular box, a circular box, an elliptical box, etc.).

[0081] The following describes the marker detection process by taking the yolo model as an example. The yolo model includes a plurality of network layers, wherein the convolutional layer is used to extract the features of the marker in the image, and the fully connected layer is used to detect and identify the features of the marker extracted by the convolutional layer.

[0082] First, the yolo model needs to be trained to enable the yolo model to have the function of marker detection. When training, first, a plurality of training sets are obtained, the training sets including a plurality of sample images, each sample image being an image containing a marker, and each image including label information of a marker in the image, the label information being used to mark the position of the marker in the image and indicate the type of the marker. Second, the parameters of the yolo model are initialized, and then the sample images of the training set are input to the yolo model. The convolutional layer in the yolo model extracts features of the marker in each sample, and the fully connected layer detects and recognizes the features of the marker output by the convolutional layer to predict the position of the marker in the image. The predicted position of the marker in the image is compared with the label information of the marker in the image, and a loss function is calculated. The parameters in the yolo model are adjusted using the calculated loss function. The above calculation process is iteratively performed until the loss function value converges and the calculated loss function value is less than a preset threshold, then the iteration is stopped, at this time, the yolo model has been trained and has the function of marker detection, and can be used for marker detection.

[0083] After obtaining the trained yolo model, the yolo model is used to detect markers in images containing markers captured by a camera. The convolutional layer extracts features of the marker in the image, and the fully connected layer detects and recognizes the features of the marker. After recognition, the position of the marker in the image is marked with a bounding box, and the class information of the marker is also marked. The bounding box can be a rectangular box, a circular box, an elliptical box, etc.

[0084] After detecting the marker in the image by the marker detection algorithm, the pixel coordinates of the marker point need to be further determined, for example, the pixel coordinates of the center point of the bounding box are selected as the pixel coordinates of the marker point. Alternatively, the pixel coordinates of the marker point can be obtained by a corresponding marker point detection algorithm. For example, for a spherical marker, its imaging is a circle at any angle, and the center of the sphere corresponds to the center of the circle, that is, the pixel coordinates of the marker point are the pixel coordinates of the center of the circle corresponding to the spherical marker in the image. After detecting the circular marker in the image, the center of the bounding box is the center of the circle, and the pixel coordinates of the center of the circle can be obtained by obtaining the pixel coordinates of the center of the bounding box. The pixel coordinates of the center of the circle can also be stably obtained by using a circle detection algorithm based on the least square method, and other circle detection algorithms can also be used. The present application does not limit this. It should be understood that the circle detection algorithm is very mature in the field of computer vision and is widely used. Therefore, the present application will not be described here.

[0085] When the pixel coordinates of the landmark point are the pixel coordinates of an easily identifiable point formed by a light source (such as a laser source) in the image, the pixel coordinates of the landmark point can be detected by a sudden point detection algorithm, for example, the RGB distribution in the image can be analyzed to determine the pixel coordinates of the light point in the image.

[0086] It should be understood that the landmark point detection can be performed on one or more images by the above method, thereby obtaining the pixel coordinates of multiple landmark points.

[0087] In a possible implementation, the geographic coordinates of the landmark point recorded in the image are obtained according to a calibration relationship between the pixel coordinates of the landmark point and the image captured by the camera and the geographic region.

[0088] Specifically, the calibration relationship between the image captured by the camera and the geographic region can be established in various ways, for example, the calibration relationship between the image captured by the camera and the geographic region can be established according to the principle of homographic transformation, and the formula of homographic transformation is (m, n, h) = H*(s, k), where (m, n, h) is the geographic coordinates of the landmark point in the image, (s, k) is the pixel coordinates of the landmark point in the image, and H is a conversion matrix. Therefore, according to the pixel coordinates of the landmark point in the image obtained above and the conversion matrix, the geographic coordinates of the landmark point in the image can be calculated.

[0089] It is worth noting that in the embodiments of the present application, the selected landmark point is not on the ground, but a point of a marker (for example, the center of a spherical marker on the calibration device) provided on the calibration device. Since the marker is easier to be detected in the image, the pixel coordinates of the landmark point of the marker obtained in the image are also more accurate.

[0090] Since the imaging model of the camera is pinhole imaging, pinhole imaging is to map a three-dimensional space to a two-dimensional space according to the principle of perspective, and the projection ray passes through the same projection center. Therefore, the geographic coordinates corresponding to the landmark point in the image captured by the camera are actually the geographic coordinates of the point on the ground where the landmark point on the calibration device is projected, that is, the geographic coordinates of a diagonal point (i.e., a vertex on the diagonal of the triangle) on the ground in a triangle formed by the angle between the center of the camera at a fixed position and the landmark point of the marker on the calibration device and the vertical height of the point on the calibration device and the ground.

[0091] For example, Figure 6As shown, the mark point of the calibration device is the center of the spherical mark R, and the point P is located in the same vertical direction as the contact point of the calibration device and the ground and is higher than the ground by a certain height. According to the imaging principle of the camera, the geographical coordinates of the projection point (i.e., the point Q) of the mark point on the ground at the camera shooting angle can be obtained according to the calibration relationship between the image captured by the camera and the geographical area. In order to calculate the matching geographical coordinates corresponding to the radar coordinates of the mark point, the geographical coordinates of the vertical projection point of the mark point, i.e., the geographical coordinates of the point P, need to be calculated. The geographical coordinates of the point P can be calculated by using the parameters of the camera (mainly the height of the camera from the ground and the geographical coordinates of the vertical projection point of the camera on the ground) and the geographical coordinates of the projection point, according to the principle that the height of the mark point on the calibration device and the triangle formed by the projection point at the camera shooting angle and the height of the camera from the ground form two similar triangles, to determine the geographical coordinates of the vertical projection point corresponding to the mark point.

[0092] A spatial coordinate system is constructed in the space of the geographical area, in which the x and y represent the first and second dimensions on the ground respectively, and the h is the third dimension of the height perpendicular to the ground. Since the points P and Q are both on the ground, the third dimension can be directly determined as 0.

[0093] In order to obtain the first and second dimension coordinates (x1, y1) of the point P, the height value (i.e., the third dimension coordinate value of the center R) h1 of the center R from the point P, the first and second dimension coordinates (x2, y2) of the point Q, the height h2 of the camera perpendicular to the ground, and the geographical coordinates (x3, y3) of the projection point M of the camera vertically projected onto the ground are required. h1 can be obtained by a spatial measurement device or by other measurement devices, such as a ruler, laser ranging, or a graduated measuring rod to directly read h1. (x2, y2) can be calculated according to the calibration relationship between the image captured by the camera and the geographical area, h2 and (x3, y3) can be obtained when the camera is set in advance, and h2 and (x3, y3) are the parameters of the camera. After obtaining the above data, the geographical coordinates of the point P can be calculated. Specifically, the first and second dimension coordinates (x1, y1) of the point P can be derived according to the basic geometric mathematical principles, such as the principle of similar triangles. For example, Figure 7As shown, since triangle RPQ and triangle AMQ are similar triangles, and RP = hi, AM = h2, MQ = (x2, y2) - (x3, y3), PQ = (x2, y2) - (x1, y1), according to the similarity principle, RP / AM = PQ / MQ, i.e. hi / h2 = (x2, y2) - (x1, y1) / (x2, y2) - (x3, y3), thus, hi / h2 = x2 - x1 / x2 - x3, hi / h2 = y2 - y1 / y2 - y3.

[0094] Therefore, x1 can be calculated using the following formula 1:

[0095] x1 = (h2 * x2 - hi * (x2 - x3)) / h2 formula 1

[0096] y1 can be calculated using the following formula 2:

[0097] y1 = (h2 * y2 - hi * (y2 - y3)) / h2 formula 2

[0098] It can be understood that by the above method, the geographical coordinates of the vertical projection points of the landmark points in the geographical area can be obtained through the image shot by the camera as an intermediary, that is, the matching geographical coordinates corresponding to the radar coordinates of the landmark points can be obtained, without directly measuring the geographical coordinates corresponding to the radar coordinates of the landmark points, which can effectively reduce the calibration workload and cost, shorten the calibration time, and improve the calibration accuracy.

[0099] S303: Determine at least two landmark point matching pairs.

[0100] Specifically, since the shooting time of the image and the measurement time of the radar coordinates are the same time, the geographical coordinates of the vertical projection points of the landmark points in the geographical area calculated and the radar coordinates of the landmark points measured by the radar device can be one-to-one corresponding according to the corresponding relationship of the time, and a plurality of landmark point matching pairs are obtained, and one landmark point matching pair represents a coordinate pair composed of the radar coordinates of one landmark point and its corresponding geographical coordinates in the geographical area. In order to calculate the calibration relationship between the radar coordinates of the landmark points and the corresponding geographical coordinates in the geographical area, and accurately calibrate the space of the geographical area, a plurality of landmark point matching pairs are usually required, for example, in an algorithm for calculating the space calibration relationship, at least two landmark point matching pairs are required to calculate the calibration relationship, and in order to improve the accuracy, a plurality of landmark point matching pairs are usually required for calculation.

[0101] Further, in order to ensure the accuracy of the calculated calibration relationship, it is necessary to determine the effectiveness of the landmark point matching pairs participating in the calculation. Only when the acquired landmark point matching pairs meet the conditions and are effective, the calculated calibration relationship will be accurate, that is, the spatial calibration will be accurate. There are three methods to judge the effectiveness of the landmark point matching pairs: 1. Straight line distribution judgment method; 2. Geographical area coverage judgment method; 3. Landmark point matching pair distribution quantity judgment method. In this application, method 1 is the necessary judgment method for judging the effectiveness of the landmark point matching pairs. In the case where the conclusion of method 1 is that the landmark point matching pairs are effective, these landmark point matching pairs can be used for the calculation of the spatial calibration relationship. It is worth noting that in actual application, the above three methods can also be combined to judge whether the landmark point matching pairs are effective, that is, on the basis of determining that the landmark point matching pairs are effective by method 1, continue to combine method 2 and / or method 3 for judgment, so that when the landmark point matching pairs meet multiple effective conditions at the same time, it is determined that these landmark point matching pairs are effective for calculating the spatial calibration relationship. This can further improve the accuracy of the subsequent obtained spatial calibration relationship. For example: when the conditions in method 1 and method 2 are met at the same time, multiple landmark point matching pairs are considered effective; or combining method 1 and method 3 for judgment, when the conditions in method 1 and method 3 are met at the same time, multiple landmark point matching pairs are considered effective; or combining method 1, method 2 and method 3 for judgment, when the conditions in method 1, method 2 and method 3 are met at the same time, multiple landmark point matching pairs are considered effective. This application does not limit this.

[0102] The specific implementation of the above three methods for judging the effectiveness of the landmark point matching pairs will be introduced below:

[0103] 1. Straight line distribution judgment method: judge whether all acquired landmark point matching pairs are in straight line distribution. If yes, the acquired landmark point matching pairs do not meet the effectiveness, and continue to execute S301-S302 to acquire more landmark point matching pairs. If not, the acquired landmark point matching pairs meet the effectiveness.

[0104] After obtaining multiple pairs of landmark point matching pairs, straight line fitting is performed on the radar coordinates or corresponding geographical coordinates of all landmark points. Both types of coordinates can be fitted, or only one type of coordinate can be fitted, because there is a specific corresponding relationship between the two types of coordinates, that is, if one type is in straight line distribution, the other type must also be in straight line distribution.

[0105] The following is an example method for judging whether the landmark point matching pairs are in straight line distribution, taking fitting all landmark point geographical coordinates as an example:

[0106] Firstly, a straight line is fitted according to the geographic coordinates of all the landmark points. After the fitting is completed, the distances of all the geographic coordinates to the fitted straight line are calculated. The geographic coordinates are all known, and the distance of each geographic coordinate to the fitted straight line can be calculated according to the distance formula of a point to a straight line in Euclidean geometry.

[0107] Secondly, it is determined whether the distance of each geographic coordinate to the straight line is less than a first threshold value, and whether the number of the geographic coordinates satisfying the distance less than the first threshold value is greater than a second threshold value. If the number of the geographic coordinates satisfying the distance less than the first threshold value is greater than the second threshold value, it can be considered that the geographic coordinates of all the landmark points are distributed on a straight line, and the condition of calculating the calibration relationship is not satisfied, and the matching pairs of the landmark points need to be re-acquired. Otherwise, it can be considered that the geographic coordinates of all the landmark points are not distributed on a straight line, and the condition of calculating the calibration relationship is satisfied, and the acquired matching pairs of the landmark points are effective, and the acquired matching pairs of the landmark points can be used to calculate the calibration relationship.

[0108] The first threshold value and the second threshold value can be set according to actual needs, and the application does not limit this. In addition, the fitted straight line mentioned above is determined in the fitting process. The fitted straight line can be determined according to the geographic coordinates of the landmark points, and the specific determination process is a technical method commonly used in the art, which will not be described here.

[0109] It should be understood that the calibration relationship we calculate is the calibration relationship between two planes, and a straight line cannot determine a plane. Therefore, if the acquired matching pairs of the landmark points are distributed in a straight line, the finally calculated calibration relationship is not accurate, in other words, the calculated transformation matrix is not unique, but a random transformation matrix that satisfies the condition. Therefore, in order to improve the accuracy of spatial calibration and ensure that the calculated transformation matrix is unique, the acquired matching pairs of the landmark points cannot be distributed in a straight line.

[0110] It should be understood that in the case of determining that the acquired matching pairs of the landmark points are not distributed in a straight line, it can be determined that the acquired matching pairs of the landmark points are effective. However, in order to further improve the accuracy of the calculated calibration relationship, one or more of the following methods can be combined to further determine the effectiveness of the matching pairs of the landmark points.

[0111] Optionally, in the case that the matching pairs of the landmark points are not distributed in a straight line, the following method 2 is combined to further determine whether the matching pairs of the landmark points are effective:

[0112] 2. Geographical area coverage judgment method: judging whether the obtained multiple landmark point matching pairs cover the intersection area (i.e. distributed in the entire intersection area), if yes, the condition for calculating the calibration relationship is met, and the obtained landmark point matching pairs are valid; if not, the condition for calculating the calibration relationship is not met.

[0113] First, after obtaining multiple landmark point matching pairs, the area of a graph containing the geographical coordinates of all landmark points is obtained.

[0114] It should be understood that the geographical coordinates of the landmark points are some sparse points on the ground, and there is no area as such. In order to judge whether the geographical coordinates of the landmark points cover all the intersection areas (i.e. distributed in the entire intersection area), the ratio relationship between the area of the graph containing the geographical coordinates of all landmark points and the area of the intersection area is used for judgment.

[0115] The graph containing the geographical coordinates of all landmark points can be a square, a rectangle, a triangle, etc., or a convex hull, which represents the smallest convex polygon containing all geographical coordinates, i.e. the convex hull is the smallest in area among the polygons containing all the geographical coordinates of the landmark points. The specific way of obtaining the convex hull can refer to the convhull function of the matrix laboratory (MATrix LABoratory, MATLAB), which will not be described here.

[0116] Secondly, after obtaining the convex hull, the proportional value of the area of the convex hull relative to the area of the intersection area is calculated, and finally, according to the relationship with the third threshold value, it is judged whether the calculation condition is met. When the proportional value is greater than the third threshold value, it can be considered that the obtained landmark point matching pairs cover all the intersection areas (i.e. distributed in the entire intersection area), and the condition for calculating the calibration relationship is met, the obtained landmark point matching pairs are valid, and the multiple landmark point matching pairs obtained can be used to calculate the calibration relationship; otherwise, it can be considered that the obtained landmark point matching pairs do not cover all the intersection areas, and the condition for calculating the calibration relationship is not met, and multiple landmark point matching pairs need to be reacquired or landmark point matching pairs continue to be acquired (increase part of the landmark point matching pairs). The third threshold value can be set according to the need, which is not limited in the present application.

[0117] It can be understood that through the above judgment method, it can be ensured that the obtained landmark point matching pairs cover all the intersection areas, and the robustness of the obtained calibration relationship is improved.

[0118] Optionally, in the case of non-straight distribution of landmark point matching pairs, the following method 3 can be combined to further judge whether the landmark point matching pairs are valid:

[0119] 3. The number of landmark point matching pair distribution judgment method: the number of landmark point matching pairs in each region in the intersection region is counted, and it is judged whether the proportion of the region where the landmark point matching pairs exist or the number of landmark point matching pairs is greater than the fourth threshold value is greater than the fifth threshold value. If yes, the obtained landmark point matching pairs are valid, and if not, it is considered that some landmark point matching pairs of the region need to be obtained.

[0120] Firstly, the intersection region is partitioned, which can be evenly divided into multiple smaller regions with equal areas.

[0121] For example, the intersection region is evenly divided into 5*5=25 small square regions. Of course, the specific way of region division and the shape of each region obtained can be divided as needed, which is not limited in the present application.

[0122] Secondly, after obtaining a plurality of landmark point matching pairs, it is judged whether each region has geographic coordinates of landmark points or the number of geographic coordinates of each region is greater than the fourth threshold value, and the proportion value of the number of regions with geographic coordinates or the number of geographic coordinates greater than the fourth threshold value to the number of all regions obtained by division is counted.

[0123] Then, after the proportion value is counted, it is judged whether the calculation condition is met according to the relationship with the fifth threshold value. If the proportion value is greater than the fifth threshold value, it can be considered that the geographic coordinates of all landmark points are uniformly distributed in the geographic region, which meets the calculation condition of the calibration relationship, and the obtained landmark point matching pairs are valid, which can be used to calculate the calibration relationship using the obtained multiple landmark point matching pairs. Otherwise, it can be considered that the geographic coordinates of all landmark points are not uniformly distributed in the geographic region, which does not meet the calculation condition of the calibration relationship, and multiple landmark point matching pairs need to be reacquired or part of the landmark point matching pairs need to be continuously acquired. The fourth threshold value and the fifth threshold value can be set as needed, which is not limited in the present application.

[0124] It is easy to understand that through the above judgment method, it can be ensured that the obtained landmark point matching pairs are uniformly distributed in the geographic region, which can reduce the spatial calibration error and improve the spatial calibration accuracy.

[0125] It should be understood that when S303 determines that the plurality of landmark point matching pairs are invalid, the spatial calibration system continues to execute S301-S302 to obtain more landmark point matching pairs, and then performs the validity judgment in S303 until S303 determines that the obtained plurality of landmark point matching pairs are valid, and then executes the subsequent S304.

[0126] S304: Obtain the calibration relationship between the radar measurement data and the geographic region according to at least two landmark point matching pairs.

[0127] Specifically, the spatial calibration system can establish a calibration relationship between the data measured by each radar device and the geographic region in the physical world according to the obtained at least two valid landmark point matching pairs.

[0128] It should be noted that the radar device is adjusted to take the moving target (for example, a moving vehicle) as a reference to adjust the measurement coordinate plane to be parallel to the ground plane when deployed, and therefore, there is an affine transformation relationship between the measurement coordinate plane of the millimeter wave radar and the XY plane of the spatial coordinate system (i.e., the coordinate system corresponding to the geographic coordinates). The affine transformation can be selected as the calibration relationship between the data measured by the radar device and the geographic region.

[0129] It should be understood that the algorithm for calculating the affine transformation is a basic content in the field of computer vision and has been widely integrated by software such as open source computer vision library (OPENCV) and MATLAB. The calculation of the affine transformation can be directly used, for example, the OPENCV library getAffineTransform function can be directly used to calculate a plurality of valid landmark point matching pairs to obtain the calibration parameters between the data measured by the radar device and the geographic region, and then obtain the corresponding calibration relationship.

[0130] It is easy to understand that after obtaining the calibration parameters corresponding to each radar device, the radar coordinates of the target measured by each radar device can be converted according to the calibration parameters to obtain the geographic coordinates corresponding to the target. However, it should be noted that the calibration parameters corresponding to the radar devices arranged at different positions are different, and the respective calibration parameters should be used for conversion to obtain a plurality of geographic coordinates corresponding to the target.

[0131] It should be understood that the steps S301 to S304 involved in the above method embodiment are only a schematic description of the general situation and should not constitute a specific limitation. The steps involved can be added, reduced or combined as needed.

[0132] After the spatial calibration of the geographical area, the spatial calibration system can store the calibration relationship between the data measured by the radar device and the geographical area. When the radar coordinates of the detected target measured by the radar device are obtained, the calibration relationship is obtained, and the geographical coordinates of the detected target in the geographical area are determined according to the calibration relationship and the radar coordinates of the detected target. The detected target can have different meanings according to different application scenarios when the spatial calibration system determines the geographical coordinates of the detected target in the geographical area according to the calibration relationship and the radar coordinates of the detected target. For example, in the application scenario of determining the illegal vehicle at the traffic intersection, the detected target is the vehicle at the traffic intersection; in the application scenario of determining the driving position of the suspicious vehicle, the detected target is the suspicious vehicle on the traffic road; in the determination of suspicious personnel in a certain area (for example, the interior of a residential area), the detected target is the suspicious personnel in the area; in the determination of dangerous situations in a certain area (for example, a factory), the detected target is the dangerous target (for example, a malfunctioning machine, a burning object, etc.) in the area.

[0133] Alternatively, the spatial calibration system can send the calculated spatial calibration relationship to a processing device, so that the processing device determines the geographical coordinates of the detected target in the geographical area according to the calibration relationship and the radar coordinates of the detected target after obtaining the radar coordinates of the detected target measured by the radar device. The processing device can be different according to different application scenarios of the calibration relationship, for example, the processing device can be a device for determining the geographical coordinates of the vehicle in the traffic management system, or a device for determining the geographical coordinates of the suspicious vehicle in the police system, or a device for determining the geographical coordinates of the suspicious personnel in the security management system, or a device for determining the geographical coordinates of the dangerous target in the dangerous investigation management system.

[0134] In summary, the spatial calibration relationship obtained by the spatial calibration system in the present application can be applied to various scenarios, and the execution subject applying the spatial calibration relationship of the present application can be the spatial calibration system itself, or other processing devices or systems receiving the spatial calibration relationship sent by the spatial calibration system. The present application does not make any limitation.

[0135] The application of the spatial calibration relationship will be described in detail below by taking the spatial calibration system itself using the calculated spatial calibration relationship to determine the illegal vehicle at the traffic intersection as an example. After the calibration relationship between the radar device and a geographical area is calculated, the spatial calibration system can store the calibration relationship, and in the subsequent time, the spatial calibration system can analyze the measurement data measured by the radar device to realize the positioning of the vehicle at the traffic intersection, obtain the geographical coordinates of the vehicle at the traffic intersection, and further determine the positional relationship between the vehicle and the traffic marking line to determine the illegal situation of the vehicle, as shown in FIG. 1. Figure 7 ​

[0136] S701: Spatial calibration is performed on the road traffic scene, and the spatial calibration relationship is stored.

[0137] Specifically, the above-mentioned Figure 3 The method calibrates the traffic road scene in space, establishes the calibration relationship between the landmark points measured by each radar device and the landmark points in the geographical area in the physical world. The spatial calibration relationship is stored in the spatial calibration system.

[0138] S702: Process the data measured by the radar device, identify and locate all vehicles.

[0139] Specifically, when illegal judgment is needed, the spatial calibration system uses a target detection algorithm to detect the video data captured by the camera, identifies all vehicles in the video data, such as using a trained neural network model such as SSD, RCNN, etc. to detect vehicles in the video data. It should be understood that the neural network model needs to be trained in advance, and the annotation of the training pictures in the training set used should include the type of target to be identified (such as motor vehicles, non-motor vehicles, etc.), so that the neural network model learns the characteristics of each type of target in the training set. After identification, the radar coordinates of all vehicles can be obtained according to the measurement results of the radar device.

[0140] S703: Target tracking is performed on all vehicles to obtain the radar coordinate sequence of all vehicles.

[0141] Specifically, target tracking refers to tracking the same target at adjacent times. The radar coordinate sequence of the target can be obtained from the radar coordinates of the target at the current time and the radar coordinates at the historical time.

[0142] S704: Calculate the motion trajectory of the vehicle in the geographical area, and complete the vehicle illegal detection according to the motion trajectory of the vehicle in the geographical area.

[0143] Specifically, the radar coordinate sequence of the vehicle obtained in S703 is actually a series of radar coordinates. For each radar coordinate in the motion trajectory sequence, the calibration parameters calculated in S304 are used to calculate the geographical coordinates corresponding to each point of the radar coordinates, and the coordinate conversion is completed.

[0144] After all the radar coordinate conversion to geographic coordinate is completed, the motion trajectory sequence of the vehicle in the geographic area of the physical world is obtained. The spatial calibration system can send the obtained motion trajectory sequence of the vehicle in the geographic area of the physical world to the traffic management system. The traffic management system can set specific detection areas for different violations, for example, for the violation of running a red light, a detection area can be set before and after the stop line of each lane, and the motion trajectory of each vehicle in the geographic area obtained from the spatial calibration system is recorded and analyzed. If the current lane signal light state is red and the vehicle trajectory passes through the two detection areas in turn, it is considered that the vehicle runs a red light; for the violation of line pressing, a lane line area where line pressing is prohibited is set, and the motion trajectory of each vehicle is recorded and analyzed. If the vehicle trajectory passes through the area, it is considered that the vehicle presses the line; for the violation of not driving in the specified lane, two detection areas can be set in two lanes. If the vehicle trajectory passes through the two detection areas in turn, it is considered that the vehicle does not drive in the specified lane, and is judged as a violation.

[0145] It can be seen that, by Figure 3 the method shown in the road traffic scene is spatially calibrated, the calibration relationship between the radar measurement data and the geographic area is established, and then the vehicle can be accurately positioned and tracked. The vehicle target, and through the analysis of the vehicle trajectory, the vehicle violation detection can be completed.

[0146] The above describes the method of the embodiments of the application in detail. In order to better implement the above-mentioned scheme of the embodiments of the application, the related equipment for cooperating with the implementation of the above-mentioned scheme is also provided.

[0147] As Figure 2 shown, the application also provides a spatial calibration system for executing the spatial calibration method of the radar device. The division of functional units in the spatial calibration system is not limited, and each unit in the spatial calibration system can be added, reduced or combined as needed. Figure 2 An exemplary division of functional units is provided:

[0148] The spatial calibration system 200 includes an acquisition unit 210, a coordinate conversion unit 220, and a calculation unit 230.

[0149] Specifically, the acquisition unit 210 is configured to perform the aforementioned step S301, and optionally perform the optional method in the aforementioned steps, to obtain the image captured by the camera and the data measured by the radar device at the same time.

[0150] The coordinate conversion unit 220 is configured to perform the aforementioned step S302 to obtain the geographic coordinates of the landmark points recorded in the image according to the pixel coordinates of the landmark points recorded in the image.

[0151] The calculation unit 230 is used to execute the aforementioned steps S303-S304, and optionally executes the optional method in the aforementioned steps, then determines at least two valid marker matching pairs and obtains the calibration relationship between the radar measurement data and the geographical area based on the at least two valid marker matching pairs.

[0152] The three units mentioned above can transmit data to each other through communication channels. It should be understood that the units included in the space calibration system 200 can be software units, hardware units, or a combination of software and hardware units.

[0153] See Figure 8 , Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 8 As shown, the computing device 800 includes a processor 810, a communication interface 820, and a memory 830, which are interconnected via an internal bus 840. It should be understood that the computing device 800 can be a computing device in a cloud environment or a computing device in an edge environment.

[0154] The processor 810 may consist of one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and hardware chips. The hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof.

[0155] Bus 840 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 840 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0156] The memory 830 can include volatile memory, such as random access memory (RAM); the memory 830 can also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 830 can also include a combination of the above-mentioned types.

[0157] It should be noted that the memory 830 of the computing device 800 stores codes corresponding to each unit of the space calibration system 200, and the processor 810 executes the codes to realize the functions of each unit of the space calibration system 200, i.e., the method of S301-S304 is executed.

[0158] As shown in Figure 9 , since each part of the space calibration system can run on multiple computing devices in different environments. Therefore, the present application also proposes a computing device system. As shown in Figure 9 , the computing device system includes multiple computing devices 900. Each computing device 900 includes a processor 910, a communication interface 920, a memory 930 and a bus 940. The processor 910, the communication interface 920 and the memory 930 communicate through the bus 940. The computing devices 900 communicate through a communication network. Among them, the processor 910 can be a CPU. The memory 930 can include volatile memory (English: volatile memory), such as RAM. The memory 930 can also include non-volatile memory, such as ROM, flash memory, HDD or SSD. The memory 930 stores executable codes, and the processor 910 executes the executable codes to execute part of the method of space calibration. The memory 930 can also include software modules required by other running processes, such as operating systems. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0159] Any computing device 900 can be a computing device in a cloud environment, or a computing device in an edge environment, or a computing device in a terminal environment, for example, the computing device 900 can be an edge computing device or a central server in the foregoing Figure 1 .

[0160] The descriptions of the processes corresponding to the respective figures above each have their own emphasis, and the parts not described in detail in a certain process can be referred to the relevant descriptions of other processes.

[0161] In the above embodiments, the implementation can be wholly or partially achieved by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions wholly or partially generate the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD), etc.

Claims

1. A spatial calibration method for radar equipment, characterized in that, include: Acquire images and radar measurement data at the same time. The images are captured by a camera located in the geographic area, and the radar measurement data is measured by a radar device located in the geographic area. The radar measurement data includes the radar coordinates of multiple marker points in the geographic area. Based on the pixel coordinates of the marker points recorded in the image, the geographic coordinates of the marker points recorded in the image are obtained; Identify at least two marker matching pairs, each marker matching pair including the geographic coordinates of one marker and the radar coordinates of the other marker; Based on the at least two marker matching pairs, the calibration relationship between the radar measurement data and the geographical area is obtained.

2. The method as described in claim 1, characterized in that, Obtaining the geographic coordinates of the markers recorded in the image based on their pixel coordinates includes: The pixel coordinates of the markers recorded in the image are determined according to the marker detection algorithm; The geographic coordinates of the marker points recorded in the image are obtained based on the pixel coordinates of the marker points and the calibration relationship between the image captured by the camera and the geographic region.

3. The method as described in claim 1 or 2, characterized in that, The camera and the radar device are independently deployed in the geographical area. The measurement area of ​​the radar device and the shooting area of ​​the camera have an overlapping area. The markers corresponding to the at least two marker matching pairs are the markers in the overlapping area.

4. The method according to any one of claims 1-2, characterized in that, The method further includes, before determining at least two flag point matching pairs, that the at least two flag point matching pairs are valid flag point matching pairs: The geographic or radar coordinates of all markers in the at least two marker matching pairs are fitted with a straight line to obtain a fitted straight line. Calculate the distance from the geographic coordinates or radar coordinates of each marker to the fitted straight line; If the number of geographic coordinates or radar coordinates of the markers whose distance to the fitted straight line is less than the first threshold is not greater than the second threshold, then the at least two marker matching pairs are determined to be valid.

5. The method as described in claim 2, characterized in that, Determining the pixel coordinates of the marker points in the image according to the marker point detection algorithm includes: The image is input into the marker detection model, and the pixel coordinates of the markers in the image are obtained based on the marker detection model. The pixel coordinates of the marker point are determined based on the pixel coordinates of the marker in the image.

6. The method as described in claim 2 or 5, characterized in that, The step of obtaining the geographic coordinates of the marker points recorded in the image based on the pixel coordinates of the marker points and the calibration relationship between the image captured by the camera and the geographic region includes: Based on the pixel coordinates of the marker points and the calibration relationship between the image captured by the camera and the geographical area, obtain the geographical coordinates of the projection points formed on the ground by the marker points in the image at the shooting angle of the camera. Obtain the parameters of the camera, which include the vertical height of the camera's position above the ground in the geographic area and the geographic coordinates of the camera's vertical projection point on the ground; The geographic coordinates of the marker are determined based on the geographic coordinates of the projection point formed on the ground by the marker at the camera's shooting angle and the parameters of the camera.

7. The method according to any one of claims 1, 2, and 5, characterized in that, The method further includes: The calibration relationship between the radar measurement data and the geographic area is sent to the processing device, so that after the processing device obtains the radar coordinates of the detected target measured by the radar device, it determines the geographic coordinates of the detected target in the geographic area according to the calibration relationship and the radar coordinates of the detected target. or, Store the calibration relationship; After obtaining the radar coordinates of the target being detected as measured by the radar device, the calibration relationship is obtained, and the geographical coordinates of the target being detected in the geographical area are determined based on the calibration relationship and the radar coordinates of the target being detected.

8. A space calibration system, characterized in that, include: An acquisition unit is used to acquire images and radar measurement data at the same time. The images are captured by a camera located in the geographical area, and the radar measurement data are measured by a radar device located in the geographical area. The radar measurement data includes the radar coordinates of multiple marker points in the geographical area. A coordinate transformation unit is used to obtain the geographic coordinates of the marker points recorded in the image based on the pixel coordinates of the marker points recorded in the image; A calculation unit is used to determine at least two marker matching pairs, each marker matching pair including the geographic coordinates of one marker and the radar coordinates of the other marker, and to obtain the calibration relationship between the radar measurement data and the geographic area based on the at least two marker matching pairs.

9. The space calibration system as described in claim 8, characterized in that, The coordinate transformation unit is specifically used for: The pixel coordinates of the markers recorded in the image are determined according to the marker detection algorithm; The geographic coordinates of the marker points recorded in the image are obtained based on the pixel coordinates of the marker points and the calibration relationship between the image captured by the camera and the geographic region.

10. The space calibration system as described in claim 8 or 9, characterized in that, The camera and the radar device are independently deployed in the geographical area. The measurement area of ​​the radar device and the shooting area of ​​the camera have an overlapping area. The markers corresponding to the at least two marker matching pairs are the markers in the overlapping area.

11. The space calibration system as described in claim 8 or 9, characterized in that, The calculation unit is further configured to: wherein the at least two marker matching pairs are valid marker matching pairs. The geographic or radar coordinates of all markers in the at least two marker matching pairs are fitted with a straight line to obtain a fitted straight line. Calculate the distance from the geographic coordinates or radar coordinates of each marker to the fitted straight line; If the number of geographic coordinates or radar coordinates of the markers whose distance to the fitted straight line is less than the first threshold is not greater than the second threshold, then the at least two marker matching pairs are determined to be valid.

12. The space calibration system as described in claim 9, characterized in that, The coordinate transformation unit is specifically used for: The image is input into the marker detection model, and the pixel coordinates of the markers in the image are obtained based on the marker detection model. The pixel coordinates of the marker point are determined based on the pixel coordinates of the marker in the image.

13. The space calibration system as described in claim 9 or 12, characterized in that, The acquisition unit is further configured to acquire parameters of the camera, including the vertical height of the camera's position above the ground in the geographic region and the geographic coordinates of the camera's vertical projection point on the ground. The coordinate transformation unit is further configured to obtain the geographic coordinates of the projection point formed by the marker point on the ground at the camera's shooting angle, based on the pixel coordinates of the marker point and the calibration relationship between the image captured by the camera and the geographic region; and to determine the geographic coordinates of the marker point based on the geographic coordinates of the projection point formed by the marker point on the ground at the camera's shooting angle and the parameters of the camera.

14. The space calibration system as described in any one of claims 8, 9, and 12, characterized in that, The calculation unit is also configured to send the calibration relationship between the radar measurement data and the geographic area to the processing device, so that after the processing device obtains the radar coordinates of the detected target measured by the radar device, it determines the geographic coordinates of the detected target in the geographic area according to the calibration relationship and the radar coordinates of the detected target. or, The calculation unit is also used to store the calibration relationship. After obtaining the radar coordinates of the target being detected measured by the radar device, the calibration relationship is obtained, and the geographical coordinates of the target being detected in the geographical area are determined according to the calibration relationship and the radar coordinates of the target being detected.

15. A computing device, characterized in that, The computing device includes a memory and a processor, the processor executing computer instructions stored in the memory, causing the computing device to perform the method according to any one of claims 1-7.

16. A computer-readable storage medium storing a computer program that, when executed by a processor, performs the method according to any one of claims 1-7.

Images