Method, device and equipment for automatic calibration of coordinate system of roadside radar and vision fusion equipment

Abstract

The present invention discloses a method, device and equipment for automatically calibrating the coordinate system of a roadside radar-visual fusion device. The method comprises the following steps: establishing an original coordinate system with the location of the radar-visual fusion device as the coordinate origin, constructing a calibration mapping model between radar coordinates and video image coordinates according to the relationship between the coordinates of the detection target in the original coordinate system and when the coordinates are converted to the normal direction and intersected at the center of the ground field of view; obtaining the image coordinates of two adjacent lane lines and performing straight line fitting to obtain the slope and intercept of the straight line; calculating the longitudinal distance of the center point of the field of view according to the slope and intercept of the straight line; inputting the radar coordinate value of the detection target and the focal length value of the device and the longitudinal distance of the center point of the field of view into the calibration mapping model to obtain the coordinate value of the radar target in the video image. The present invention has the advantages of simple operation, low cost, strong convenience, high accuracy and high efficiency.

Description

Technical Field

[0001] The present invention mainly relates to the field of intelligent transportation technology, and specifically relates to a method, device and equipment for automatically calibrating the coordinate system of a roadside radar and vision fusion device. Background Art

[0002] Intelligent road control relies on the deployment of a series of highly intelligent roadside equipment on road infrastructure, such as traffic radars, video components and other sensors. Each sensor device collects data and performs real-time data analysis in traffic road scenarios, thereby assisting in the realization of vehicle-road collaboration. In order to perform data analysis and fusion, it is necessary to integrate multi-source data information from various roadside sensor devices into a unified coordinate system to ensure the consistency and accuracy of data from different sensors. The radar and vision fusion device is an integrated device of radar and vision sensors. The radar and vision sensors respectively collect and analyze data in real time in traffic road scenarios. By fusing the target information detected by the radar and video sensors into a common coordinate system, it is possible to achieve real-time, multi-dimensional information integration and unification of various elements in the traffic scene (including pedestrians, vehicles, animals, etc.).

[0003] In the prior art, multi-sensor coordinates are usually realized by manual calibration, which is not only costly and inefficient, but also easily affected by the environment, and it is difficult to provide accurate and efficient calibration results. Chinese patent application CN115542271A radar coordinate and video coordinate calibration method, equipment and related devices, the scheme obtains matching radar targets and video targets by the radar vision device; respectively uses the radar coordinates corresponding to the radar target and the video coordinates corresponding to the video target to calculate the calibration parameters corresponding to each area within the detection range of the radar vision device, and the detection range is divided into regions based on scene information; and uses the calibration parameters of the region to map the video coordinates to the radar coordinate system. However, this type of solution is complex to implement, and it is necessary to obtain matching radar targets and video targets for a large number of calculations. The calculation and operation complexity is high, which is not conducive to the rapid realization of calibration mapping between radar and video coordinates. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a method, device and equipment for automatically calibrating the coordinate system of a roadside radar and vision fusion device, which has a simple implementation method, low cost, high accuracy, high efficiency and strong flexibility.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for automatically calibrating a coordinate system of a roadside radar and visual fusion device, wherein the radar and visual fusion device comprises a radar and a visual module, and the radar and visual fusion device is installed at the roadside end. The method comprises the following steps:

[0007] An original coordinate system is established with the location of the radar-visual fusion device as the coordinate origin, and a calibration mapping model between radar coordinates and video image coordinates is constructed according to the relationship between the coordinates of the detection target in the original coordinate system and the coordinates when the target is converted to the normal direction and intersects the center of the ground field of view; the video image coordinates in the calibration mapping model are calculated using the radar coordinates, the longitudinal distance of the center point of the field of view, and the focal length value;

[0008] Obtain the image coordinates of two adjacent lane lines and perform straight line fitting to obtain the slope and intercept of the straight line;

[0009] The longitudinal distance of the center point of the field of view is calculated according to the slope and intercept of the straight line obtained by fitting;

[0010] The radar coordinate value of the detection target detected by the radar-vision fusion device and the focal length value of the device are obtained, and the radar coordinate value, the focal length value and the calculated longitudinal distance of the center point of the field of view are input into the calibration mapping model to obtain the coordinate value of the radar target in the video image.

[0011] Furthermore, establishing an original coordinate system with the location of the radar and vision fusion device as the coordinate origin includes: taking the installation location of the radar and vision fusion device as the origin of the coordinate system, the Z axis is the normal direction, the Y axis is vertically downward, and the X axis is perpendicular to the Oyz plane to satisfy the right-hand screw rule, and the coordinates of the center position of the ground are (0, h, z), where h is the installation height of the radar and vision fusion device.

[0012] Furthermore, if the detection target is a road target, that is, the detection target has no height information, and the road coordinates in the original coordinate system are (x, h, z), then the calibration mapping model is:

[0013]

[0014]

[0015] If the height of the detection target is h1, and the coordinates of the detection target in the original coordinate system are (x,h-h1,z), then the calibration mapping model is:

[0016]

[0017]

[0018] Among them, x p ,y pare the horizontal and vertical coordinates of the detected target mapped to the video image coordinate system, x and z are the horizontal and vertical coordinates of the radar coordinates, h is the installation height of the radar-vision fusion device, lc is the longitudinal distance of the center point of the field of view, and x is the vertical distance of the center point of the field of view. pc and pc are the horizontal and vertical coordinate values ​​of the center of the image.

[0019] Furthermore, the construction of the calibration mapping model between the radar coordinates and the video image coordinates also includes the construction of an external parameter matrix, wherein if the detection target is a road target, the external parameter matrix is:

[0020]

[0021] Among them, x′, y′, z′ are the X, Y, and Z axis coordinate values ​​of the road target after conversion, and θ is the rotation angle required for the radar-visual fusion setting when the original coordinate system is converted to the normal direction intersecting the center of the ground field of view;

[0022] If the height of the detected target is h1, the external parameter matrix is:

[0023]

[0024] Among them, x1′, y1′, and z1′ are the converted X-, Y-, and Z-axis coordinate values ​​of the detection target with a height of h1.

[0025] Furthermore, when constructing the extrinsic parameter matrix, the z' dimension of the constructed extrinsic parameter matrix is ​​normalized to obtain a corresponding point P' by normalizing to a vertical plane where the distance between the device and the Z' axis is 1, wherein if the detection target is a road target, P' is:

[0026]

[0027] If the height of the detected target is h1, then P′ is:

[0028]

[0029] Furthermore, the longitudinal distance of the center point of the field of view is calculated according to the slope and intercept of the straight line obtained by fitting according to the following formula:

[0030]

[0031] Among them, h is the installation height of the radar-vision fusion device, lc is the longitudinal distance of the center point of the field of view, k is the slope of the fitted straight line, b is the intercept of the fitted straight line, and f is the focal length.

[0032] Furthermore, the step of obtaining the image coordinates of two adjacent lane lines and performing straight line fitting includes:

[0033] Build a relationship model:

[0034]

[0035] Among them, x w is the lane width, Δx wp is the lane pixel width corresponding to the position at distance z in the longitudinal direction, Δy p Indicates the pixel coordinate difference of the target distance in the Y-axis direction obtained by radar detection;

[0036] With Δx wp As the dependent variable, Δy p As the independent variable, the slope k and intercept b of the straight line are obtained by linear fitting, that is:

[0037]

[0038]

[0039] A coordinate system automatic calibration device for a roadside radar and visual fusion device, the radar and visual fusion device comprising a radar and a visual module, the radar and visual fusion device being installed at the roadside end, the calibration device comprising:

[0040] A model building module is used to establish an original coordinate system with the location of the radar-vision fusion device as the coordinate origin, and to build a calibration mapping model between radar coordinates and video image coordinates according to the relationship between the coordinates of the detection target in the original coordinate system and the coordinates when the target is converted to the normal direction and intersects the center of the ground field of view; the video image coordinates in the calibration mapping model are calculated using the radar coordinates, the longitudinal distance of the center point of the field of view, and the focal length value;

[0041] The straight line fitting module is used to obtain the image coordinates of two adjacent lane lines and perform straight line fitting to obtain the slope and intercept of the straight line;

[0042] A calculation module, used for calculating the longitudinal distance of the center point of the field of view according to the slope and intercept of the straight line obtained by fitting;

[0043] A mapping module is used to obtain the radar coordinate value of the detection target detected by the radar-vision fusion device and the focal length value of the device, input the radar coordinate value, the focal length value and the calculated longitudinal distance of the center point of the field of view into the calibration mapping model, and obtain the coordinate value of the radar target in the video image.

[0044] An electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0045] A computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] 1. The present invention constructs a calibration mapping model for the longitudinal distance and focal length value between the radar coordinates and the video image coordinates about the center point of the field of view according to the relationship between the coordinates of the detection target in the original coordinate system and the coordinates when the coordinates are converted to the normal direction and intersect the center of the ground field of view, and then uses the image coordinates of two adjacent lane lines to perform straight line fitting to obtain the slope and intercept of the straight line to calculate the longitudinal distance of the center point of the field of view, and inputs it into the calibration mapping model to efficiently and accurately realize the calibration mapping between the radar and video image coordinate systems, without the need for complex calculation and mapping processes, which can greatly improve the efficiency and convenience of calibration mapping, while also ensuring the accuracy of calibration mapping.

[0048] 2. The present invention further constructs an approximate model for solving the longitudinal distance of the center point of the field of view, and approximates the model to a linear model, so that only the image coordinates of adjacent lane lines or two lines of equal width need to be obtained on the image, and the actual center point position parameters of the field of view can be automatically calculated by straight line fitting. Combined with the calibration mapping model, the automatic calibration mapping between the 3D millimeter radar or 4D millimeter wave radar coordinate system and the video image coordinates can be quickly completed. Not only less prior information and restrictions are required, but also the implementation process of automatic calibration can be further simplified, so that the efficiency and convenience of calibration can be effectively improved while ensuring the accuracy of the automatic calibration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a schematic diagram of the implementation process of the method for automatically calibrating the coordinate system of the roadside radar and vision fusion device in this embodiment.

[0050] Figure 2 It is a schematic diagram of the principle of coordinate system construction and mapping of the radar and vision fusion device in this embodiment.

[0051] Figure 3 It is a schematic diagram of the principle of coordinate system mapping for 3D radar detection coordinates in this embodiment.

[0052] Figure 4 It is a schematic diagram of the principle of coordinate system mapping for 4D radar detection coordinates in this embodiment.

[0053] Figure 5 It is a schematic diagram of the imaging principle of lanes in the direction of the traffic road within the field of view in this embodiment. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0055] The radar and vision fusion equipment in this embodiment specifically includes a radar and a vision module. The radar and vision fusion equipment is installed at the roadside end at a height of h. The radar and vision module collect traffic data on the road in real time. Figure 1 As shown, the steps of the method for automatically calibrating the coordinate system of the roadside radar and vision fusion device in this embodiment include:

[0056] Step S1: Establish an original coordinate system with the location of the radar-visual fusion device as the coordinate origin, and construct a calibration mapping model between radar coordinates and video image coordinates according to the relationship between the coordinates of the detection target in the original coordinate system and the coordinates when the target is converted to the normal direction and intersects the center of the ground field of view; the video image coordinates in the calibration mapping model are calculated using the radar coordinates, the longitudinal distance of the center point of the field of view, and the focal length value;

[0057] Step S2: Obtain the image coordinates of two adjacent lane lines and perform straight line fitting to obtain the slope and intercept of the straight line;

[0058] Step S3: Calculate the longitudinal distance of the center point of the field of view according to the slope and intercept of the fitted straight line;

[0059] Step S4: Obtain the radar coordinate value of the detection target detected by the radar-vision fusion device and the focal length value of the device, input the radar coordinate value, focal length value and the calculated longitudinal distance of the center point of the field of view into the calibration mapping model, and obtain the coordinate value of the radar target in the video image.

[0060] The traffic scene where the radar-vision fusion device is installed is fixed, so after completing one calibration, the correspondence between the actual position coordinates of each point on the road surface and the pixel coordinates in the image is constant. This embodiment uses the above characteristics, and for the roadside radar-vision fusion device, in advance, according to the relationship between the coordinates of the detection target in the original coordinate system and when it is converted to the normal direction and intersected at the center of the ground field of view, a calibration mapping model for the longitudinal distance and focal length value between the radar coordinates and the video image coordinates about the center point of the field of view is constructed, and then the image coordinates of the two adjacent lane lines are used to fit the straight line slope and intercept to calculate the longitudinal distance of the center point of the field of view, and input it into the calibration mapping model to efficiently and accurately realize the automatic calibration mapping between the radar and video image coordinate systems, without the need for complex calculation and mapping processes, which can greatly improve the efficiency of calibration mapping and the convenience of operation, while also ensuring the accuracy of calibration mapping.

[0061] In this embodiment, the visual module (camera) calibration and the internal parameter matrix calibration need to be completed before calibration. Specifically, the camera calibration can be performed by the classic Zhang calibration method. After calibration, the internal parameter matrix is ​​obtained as follows:

[0062]

[0063] Among them, x pc and pc are the pixel coordinate positions f of the center of the horizontal and vertical directions of the image respectively x and f y In practice, they are very close, and the average of the two is taken as the focal length value, that is:

[0064]

[0065] like Figure 2 As shown, a radar-vision fusion device is installed at a height of h on the roadside end, and an original coordinate system is established with the location of the radar-vision fusion device as the origin of the coordinate system. Specifically, the installation location of the radar-vision fusion device is used as the origin of the coordinate system, that is, for the world coordinate system O-xyz, the origin is set at the installation location of the device, the Z axis is the normal direction, the Y axis is vertically downward, and the X axis is perpendicular to the Oyz plane to satisfy the right-hand screw rule. For the actual scene, the device faces forward and looks down around the X axis for θ radians. Assume from the above that the device is installed at a height of h, then the center of the ground is at (0, h, z).

[0066] The target detected by the 3D millimeter wave radar has no height information, while the 4D millimeter wave radar can detect the height of the target. This embodiment derives the construction process of the calibration mapping model for the targets detected by the above two types of radars.

[0067] (1) Calibration mapping of 3D radar detection targets and video images

[0068] Since there is no height information for the targets detected by 3D millimeter wave radar, the coordinates of all detected targets are on the road surface by default. All road surface coordinates can be expressed as (x, h, z) coordinates, that is, the known y-axis coordinate is h. Figure 3 As shown, assuming that the device is rotated by θ radians to obtain the transformed world coordinate system Ox′y′z′, the normal direction of the radar device intersects the center of the ground field of view (0, h, lc).

[0069] From the above, we can see that:

[0070]

[0071]

[0072] By rotation transformation, the ground coordinates (x, h, z) in the original world coordinate system O-xyz are converted to (x', y', z'), and the following relationship exists:

[0073]

[0074] The above is the external parameter matrix. The z' dimension is further normalized to the point P' on the vertical plane where the distance between the device and the Z' axis is 1, that is:

[0075]

[0076] Then the internal parameter matrix is:

[0077]

[0078] In the above formula, f is the ratio of the original focal length value to the physical pixel value of the sensor, and x pc and pc Then the image center can be further obtained:

[0079]

[0080]

[0081] That is, the road target with a longitudinal distance of z and a lateral distance of x can be mapped into a video image according to equations (8) and (9) as (x p ,y p ).

[0082] This embodiment constructs a calibration mapping model between the road surface target detected by the 3D radar and the video image by using the above equations (8) and (9), which can realize the automatic calibration of the coordinate system of the roadside 3D radar and the visual device. Under the premise of ensuring the calibration accuracy and ease of operation, it realizes the one-to-one calibration of the pixel coordinates and actual distance of the road surface part in the video image.

[0083] (2) Calibration mapping of 4D radar and video images

[0084] 4D millimeter wave radar can detect the height of the target, such as Figure 4 As shown, assuming that the 4D millimeter-wave radar detects a target with a height of h1, the coordinates of the target in the original O-xyz coordinate system are (x, h-h1, z), which can be obtained in the same way as the above 3D radar calibration mapping.

[0085]

[0086] Further, we can get:

[0087]

[0088] From this, we can get the corresponding coordinates of the detected target in the image:

[0089]

[0090]

[0091] From the above equations (12) and (13), we can construct the calibration mapping model when the 4D radar detects a target with a height of h1:

[0092]

[0093]

[0094] Where lc is the longitudinal distance of the center point of the field of view, x pc and pc are the horizontal and vertical coordinate values ​​of the center of the image.

[0095] According to the above calibration mapping model, whether it is 3D millimeter-wave radar coordinates or 4D millimeter-wave radar coordinates with height parameters, one-to-one mapping with video image coordinates can be quickly achieved through the corresponding calibration mapping model.

[0096] From the mapping relationship between the video image coordinates and the radar coordinates, it can be seen that by determining the longitudinal distance lc value of the center point of the field of view, the mapping relationship between the radar coordinate systems can be determined. In the traffic scene, the road where the vehicle target is traveling is divided by lane lines, and the lane width is consistent. Considering the detection field of view of the 3D millimeter wave radar, we can get:

[0097]

[0098]

[0099] Then we can get:

[0100]

[0101] In general, the focal length value f is much larger than the difference between the target pixel coordinates and the center point coordinates, and the longitudinal distance z measured by the millimeter wave radar is generally more than 30m, the center point distance is generally close to 80m, and the installation height is generally less than 6-8m, which satisfies:

[0102] f>>Δy

[0103] lc>>h

[0104] So we can get:

[0105]

[0106]

[0107] Combining the above two equations, we can get:

[0108]

[0109] In the above formula, x and Δx p They respectively represent the lateral distance in the normal direction and the lateral pixel coordinate difference of the target distance detected by the millimeter wave radar.

[0110] The imaging of the lanes in the traffic road direction within the field of view is as follows: Figure 5 As shown, assuming that the width of the lane is x w , the lane pixel width corresponding to the position at a distance z in the longitudinal axis direction is Δx wp , then according to the similarity principle, we can get:

[0111]

[0112] Right now:

[0113]

[0114] From the above formula, we can know that according to the image coordinates of the two lane lines in the field of view on the video image, we can get a series of (Δy p ,Δx wp ) coordinate point, Δx wp is the lane pixel width corresponding to the position at distance z in the longitudinal direction, Δy p Indicates the pixel coordinate difference of the target distance in the Y-axis direction obtained by radar detection, and then Δx wp As the dependent variable, Δy p As the independent variable, we can get the slope k and intercept b of the straight line by fitting the straight line, then:

[0115]

[0116]

[0117] Then we can get:

[0118]

[0119] It can be seen from formula (26) that as long as the image coordinates of two adjacent lane lines are obtained on the image, the straight line fitting method can be further used to automatically calculate the center coordinate lc value of the field of view, and the calibration mapping between the radar coordinate system and the video image coordinate system can be completed by substituting it into the calibration mapping model. Similarly, the above implementation method is also applicable to the autonomous calibration mapping process of the 4D millimeter wave radar coordinate system with height parameters and the video image coordinate system.

[0120] Based on the above principle, the steps of acquiring the image coordinates of two adjacent lane lines and performing straight line fitting in this embodiment specifically include:

[0121] Construct the relational model as shown in formula (23):

[0122]

[0123] With Δx wp As the dependent variable, Δy p As the independent variable, the slope k and intercept b of the straight line are obtained by linear fitting, as shown in equations (24) and (25);

[0124] The longitudinal distance lc of the center point of the field of view is calculated according to the slope and intercept of the straight line obtained by the above fitting according to formula (26).

[0125] This embodiment constructs an approximate model for solving the longitudinal distance lc of the center point of the field of view, and approximates the model to a linear model, so that only the image coordinates of adjacent lane lines or two lines of equal width need to be obtained on the image, and the actual center point position parameters of the field of view can be automatically calculated by straight line fitting. Combined with the calibration mapping model, the automatic calibration mapping between the 3D millimeter radar or 4D millimeter wave radar coordinate system and the video image coordinates can be quickly completed. Not only less prior information and restrictions are required, but also the implementation process of automatic calibration can be further simplified, so that the efficiency and convenience of calibration can be effectively improved while ensuring the accuracy of the automatic calibration results.

[0126] The coordinate system automatic calibration device of the roadside radar and vision fusion equipment in this embodiment includes:

[0127] The model building module is used to establish an original coordinate system with the location of the radar-vision fusion device as the coordinate origin, and to build a calibration mapping model between radar coordinates and video image coordinates based on the relationship between the coordinates of the detection target in the original coordinate system and the coordinates when the target is converted to the normal direction and intersects the center of the ground field of view; the video image coordinates in the calibration mapping model are calculated using the radar coordinates, the longitudinal distance of the center point of the field of view, and the focal length value;

[0128] The straight line fitting module is used to obtain the image coordinates of two adjacent lane lines and perform straight line fitting to obtain the slope and intercept of the straight line;

[0129] A calculation module, used for calculating the longitudinal distance of the center point of the field of view according to the slope and intercept of the fitted straight line;

[0130] The mapping module is used to obtain the radar coordinate value of the detection target detected by the radar-vision fusion device and the focal length value of the device, input the radar coordinate value, focal length value and the calculated longitudinal distance of the center point of the field of view into the calibration mapping model, and obtain the coordinate value of the radar target in the video image.

[0131] The automatic calibration device for the coordinate system of the roadside radar and vision fusion device in this embodiment corresponds one to one with the automatic calibration method for the coordinate system of the roadside radar and vision fusion device described above, and will not be described in detail here.

[0132] This embodiment further provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0133] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0134] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0135] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for automatically calibrating the coordinate system of a roadside radar and visual fusion device, wherein the radar and visual fusion device comprises a radar and a visual module, and the radar and visual fusion device is installed at the roadside, characterized in that: The steps of the method include: The original coordinate system is established with the location of the radar-visual fusion device as the coordinate origin. According to the relationship between the coordinates of the detection target in the original coordinate system and the coordinates when the target is converted to the normal direction and intersected with the center of the ground field of view, a calibration mapping model between the radar coordinates and the video image coordinates is constructed; the video image coordinates in the calibration mapping model are calculated using the radar coordinates, the longitudinal distance of the center point of the field of view, and the focal length value. If the detection target is a road target, that is, the detection target has no height information, and the road coordinates in the original coordinate system are (x, h, z), then the calibration mapping model is: If the height of the detection target is h1, and the coordinates of the detection target in the original coordinate system are (x,h-h1,z), then the calibration mapping model is: Among them, x p ,y p are the horizontal and vertical coordinates of the detected target mapped to the video image coordinate system, x and z are the horizontal and vertical coordinates of the radar coordinates, h is the installation height of the radar-vision fusion device, lc is the longitudinal distance of the center point of the field of view, and x is the vertical distance of the center point of the field of view. pc and pc are the horizontal and vertical coordinate values ​​of the center of the image; Obtain the image coordinates of two adjacent lane lines and perform straight line fitting to obtain the slope and intercept of the straight line; The longitudinal distance of the center point of the field of view is calculated according to the slope and intercept of the straight line obtained by fitting; The radar coordinate value of the detection target detected by the radar-vision fusion device and the focal length value of the device are obtained, and the radar coordinate value, the focal length value and the calculated longitudinal distance of the center point of the field of view are input into the calibration mapping model to obtain the coordinate value of the radar target in the video image.

2. The method for automatic calibration of the coordinate system of the roadside radar and vision fusion equipment according to claim 1 is characterized in that: Establishing the original coordinate system with the location of the radar and vision fusion device as the coordinate origin includes: taking the installation location of the radar and vision fusion device as the origin of the coordinate system, the Z axis is the normal direction, the Y axis is vertically downward, and the X axis is perpendicular to the Oyz plane to satisfy the right-hand screw rule. The coordinates of the center position of the ground are (0, h, z), where h is the installation height of the radar and vision fusion device.

3. The method for automatic calibration of the coordinate system of the roadside radar and vision fusion equipment according to claim 1 is characterized in that: The construction of the calibration mapping model between the radar coordinates and the video image coordinates also includes constructing an external parameter matrix, wherein if the detection target is a road target, the external parameter matrix is: Among them, x ′ ,y ′ , z′ are the X, Y, and Z axis coordinate values ​​of the road target after conversion, and θ is the rotation angle required for the radar-visual fusion setting when the original coordinate system is converted to the normal direction intersecting the center of the ground field of view; If the height of the detected target is h1, the external parameter matrix is: Among them, x1′, y1′, and z1′ are the converted X-, Y-, and Z-axis coordinate values ​​of the detection target with a height of h1.

4. The method for automatic calibration of the coordinate system of the roadside radar and vision fusion equipment according to claim 3 is characterized in that: When constructing the external parameter matrix, the z' dimension of the constructed external parameter matrix is ​​normalized to obtain a corresponding point P' by normalizing to a vertical plane where the distance between the device and the Z' axis is 1. If the detection target is a road target, P' is: If the height of the detected target is h1, then P′ is:

5. The method for automatically calibrating the coordinate system of the roadside radar and vision fusion equipment according to any one of claims 1 to 4, characterized in that: The longitudinal distance of the center point of the field of view is calculated according to the slope and intercept of the straight line obtained by fitting according to the following formula: Among them, h is the installation height of the radar-vision fusion device, lc is the longitudinal distance of the center point of the field of view, k is the slope of the fitted straight line, b is the intercept of the fitted straight line, and f is the focal length.

6. The method for automatic calibration of the coordinate system of the roadside radar and vision fusion equipment according to claim 5 is characterized in that: The step of obtaining the image coordinates of two adjacent lane lines and performing straight line fitting comprises: Build a relationship model: Among them, x w is the lane width, Δx wp is the lane pixel width corresponding to the position at distance z in the longitudinal direction, Δy p Indicates the pixel coordinate difference of the target distance in the Y-axis direction obtained by radar detection; With Δx wp As the dependent variable, Δy p As the independent variable, the slope k and intercept b of the straight line are obtained by linear fitting, that is:

7. A coordinate system automatic calibration device for a roadside radar and visual fusion device, the radar and visual fusion device comprising a radar and a visual module, the radar and visual fusion device being installed at the roadside, characterized in that: The calibration device comprises: The model building module is used to establish an original coordinate system with the location of the radar-visual fusion device as the coordinate origin, and to build a calibration mapping model between radar coordinates and video image coordinates according to the relationship between the coordinates of the detection target in the original coordinate system and when the coordinates are converted to the normal direction and intersected at the center of the ground field of view; the video image coordinates in the calibration mapping model are calculated using the radar coordinates, the longitudinal distance of the center point of the field of view, and the focal length value. If the detection target is a road target, that is, the detection target has no height information, and the road coordinates in the original coordinate system are (x, h, z), then the calibration mapping model is: If the height of the detection target is h1, and the coordinates of the detection target in the original coordinate system are (x,h-h1,z), then the calibration mapping model is: Among them, x p ,y p are the horizontal and vertical coordinates of the detected target mapped to the video image coordinate system, x and z are the horizontal and vertical coordinates of the radar coordinates, h is the installation height of the radar-vision fusion device, lc is the longitudinal distance of the center point of the field of view, and x is the vertical distance of the center point of the field of view. pc and pc are the horizontal and vertical coordinate values ​​of the center of the image; The straight line fitting module is used to obtain the image coordinates of two adjacent lane lines and perform straight line fitting to obtain the slope and intercept of the straight line; A calculation module, used for calculating the longitudinal distance of the center point of the field of view according to the slope and intercept of the straight line obtained by fitting; A mapping module is used to obtain the radar coordinate value of the detection target detected by the radar-vision fusion device and the focal length value of the device, input the radar coordinate value, the focal length value and the calculated longitudinal distance of the center point of the field of view into the calibration mapping model, and obtain the coordinate value of the radar target in the video image.

8. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images