Camera and radar calibration method and device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a camera and radar calibration method and device, electronic equipment and storage medium, which relate to the technical field of multi-dimensional perception, and include: acquiring calibration orientation data collected by a radar and calibration image data collected by a camera. Based on the calibration orientation data and the calibration image data, the position of a preset calibration point in a target object in a radar coordinate system and the position of a target detection box of the target object in the calibration image data in a pixel coordinate system are determined. Based on the position of the target detection box in the pixel coordinate system and an initial conversion relationship between the pixel coordinate system and the radar coordinate system, the position of a target region in the radar coordinate system is determined. The shortest distance from the preset calibration point in the radar coordinate system to the target region is calculated. Based on the shortest distance, the initial parameter value of a calibration parameter is adjusted to determine the target parameter value of the calibration parameter, and the target conversion relationship between the pixel coordinate system and the radar coordinate system is obtained, so as to improve the accuracy of the determined target conversion relationship.

Description

Technical Field

[0001] This application relates to the field of multidimensional sensing technology, and in particular to a camera and radar calibration method, apparatus, electronic device and storage medium. Background Technology

[0002] Millimeter-wave radar and spherical cameras (PTZ cameras) together form a monitoring system capable of detecting ships in water transport scenarios. Since the azimuth data acquired by millimeter-wave radar and the image data acquired by the PDR camera are two different modes, the implementation of this monitoring system requires precise conversion between the radar-acquired azimuth data and the PDR-acquired image data to improve the ship monitoring effect.

[0003] In related technologies, technicians conduct precise measurements of various calibration parameters in the water transport scenario on-site. Based on these measurements, a mapping relationship is determined between radar coordinates and the PTZ value of the PTZ camera. Then, based on this mapping relationship, the azimuth data acquired by the radar and the image data acquired by the PTZ camera are converted. However, this process suffers from significant errors due to the manual measurement of calibration parameters, and it requires high measurement costs in practical applications. Specifically, calibrating both the camera and radar is expensive and reduces the efficiency of calibration. Summary of the Invention

[0004] The purpose of this application is to provide a camera and radar calibration method, apparatus, electronic device, and storage medium to reduce the cost of camera and radar calibration, improve the efficiency of camera and radar calibration, and improve the accuracy of the target transformation relationship between the determined camera pixel coordinate system and the radar coordinate system. The specific technical solution is as follows:

[0005] Firstly, in order to achieve the above objectives, embodiments of this application provide a camera and radar calibration method, the method comprising:

[0006] Acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera;

[0007] Based on the calibration azimuth data and the calibration image data, the position of the preset calibration point in the target object in the radar coordinate system of the radar is determined, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera is determined.

[0008] Based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the position of the target region corresponding to the target detection box in the radar coordinate system is determined; wherein, the calibration parameter in the initial transformation relationship is an initial parameter value;

[0009] Based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system, calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system;

[0010] The initial parameter values ​​of the calibration parameters are adjusted based on the shortest distance to determine the target parameter values ​​of the calibration parameters, thereby obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system that includes the target parameter values ​​of the calibration parameters.

[0011] Optionally, determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system includes: determining the two-dimensional coordinates of a second specified point corresponding to the first specified point in the target detection box in the radar coordinate system based on the two-dimensional coordinates of the first specified point in the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system; calculating the horizontal and vertical field of view angles of the target object based on the Z-value of the camera and the visual width and visual length of the target detection box; wherein, the horizontal field of view angle of the target object is... The angle is defined as the angle between the line connecting the camera and the left edge of the target object and the line connecting the camera and the right edge of the target object; the vertical field of view of the target object is defined as the angle between the line connecting the camera and the upper edge of the target object and the line connecting the camera and the lower edge of the target object; the camera is a spherical camera; based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, the target width and target length of the target area corresponding to the target detection box in the radar coordinate system are calculated; based on the two-dimensional coordinates of the second designated point in the radar coordinate system, and the target width and target length, the position of the target area in the radar coordinate system is determined.

[0012] Optionally, calculating the horizontal and vertical field of view of the target object based on the camera's Z-value and the width and length of the target detection box includes: calculating the horizontal and vertical field of view of the target object based on the camera's Z-value, a first preset formula, and the visual width and visual length of the target detection box; wherein the first preset formula includes:

[0013] γ u =W u *FOV u (Z)

[0014] γ v =W v *FOV v (Z)

[0015] Y u W represents the horizontal field of view of the target object. u The field of view (FOV) represents the visual width of the target detection box. u (Z) represents the horizontal field of view of the camera corresponding to the Z-value; Z represents the Z-value of the camera; Y v W represents the vertical field of view of the target object. v The field of view (FOV) represents the visual length of the target detection box. v (Z) represents the vertical field of view of the camera corresponding to the Z value of the camera.

[0016] Optionally, calculating the target width and target length of the target region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, includes: calculating the initial width and initial length of the initial estimated region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, a second preset formula, and the horizontal and vertical field of view of the target object; wherein, the second preset formula includes:

[0017] w est =γu*R radar

[0018] h est =γ v *R radar

[0019] w est Indicates the initial width; γ u R represents the horizontal field of view of the target object. radar h represents the distance between the target object and the radar. est Indicates the initial length; γ vThis indicates the vertical field of view of the target object;

[0020] Based on the initial width and the initial length, the target width and target length of the target detection box in the radar coordinate system are determined.

[0021] Optionally, determining the target width and target length of the target region corresponding to the target detection box in the radar coordinate system based on the initial width and the initial length includes: adjusting the initial width and the initial length according to preset adjustment parameters to obtain the target width and target length of the target region corresponding to the target detection box in the radar coordinate system.

[0022] Optionally, the first designated point is the midpoint of the bottom edge of the target detection box;

[0023] The step of determining the position of the target region in the radar coordinate system based on the two-dimensional coordinates of the second designated point in the radar coordinate system, as well as the target width and the target length, includes: taking the second designated point as the center point of the target region, and the direction of the line connecting the target object to the camera as the direction of the short side of the target region, determining a rectangular region in the radar coordinate system with a width equal to the target width and a length equal to the target length, thereby obtaining the position of the target region in the radar coordinate system.

[0024] Optionally, the step of acquiring the calibration azimuth data of the target scene collected by the radar and the calibration image data of the target scene collected by the camera includes: acquiring various calibration azimuth data of the target scene collected by the radar at various historical moments and multiple calibration image data of the target scene collected by the camera at various historical moments.

[0025] The step of determining the position of a preset calibration point in the target object within the radar coordinate system of the radar, and the position of the target detection bounding box of the target object in the calibration image data within the pixel coordinate system of the camera, based on the calibration azimuth data and the calibration image data, includes: for each historical moment, obtaining the position of the preset calibration point of the first object in the calibration azimuth data of that historical moment within the radar coordinate system of the radar; performing target detection on each calibration image data of that historical moment to obtain the position of the detection bounding box of the second object in the calibration image data of that historical moment within the pixel coordinate system of the camera. The position in the coordinate system; combining the first object and the second object at this historical moment to obtain multiple calibration data sets; wherein, a calibration data set includes: the position of a preset calibration point of the first object in the radar coordinate system of the radar, and the position of the detection box of the second object in the pixel coordinate system of the camera; from the multiple calibration data sets, determining a target calibration data set containing the target object; obtaining the position of the preset calibration point of the target object in the target calibration data set in the radar coordinate system, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera.

[0026] Optionally, determining the target calibration data group containing the target object from the plurality of calibration data groups includes: for each calibration data group, when the attribute information of the first object and the second object in the calibration data group meets the preset filtering conditions, determining the calibration data group as the target calibration data group containing the target object; wherein, the attribute information of an object includes at least one of the following: the distance between the object and the radar, the object's speed, size, and direction of movement.

[0027] Optionally, the step of calculating the shortest distance from the preset calibration point to the target area in the radar coordinate system based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system includes: for each target calibration data group, based on the position of the preset calibration point in the target calibration data group in the radar coordinate system and the position of the target area in the calibration data group in the radar coordinate system, calculating the shortest distance from the preset calibration point in the target calibration data group to the target area in the radar coordinate system to obtain the shortest distance corresponding to the target calibration data group;

[0028] The step of adjusting the initial parameter value of the calibration parameter based on the shortest distance to determine the target parameter value of the calibration parameter and obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter includes: adjusting the initial parameter value of the calibration parameter to obtain the adjusted initial parameter value when the shortest distance corresponding to each target calibration data group does not meet the preset convergence condition; returning to execute the step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, until the shortest distance corresponding to each target calibration data group meets the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter is obtained;

[0029] The method further includes: when the shortest distance corresponding to each target calibration data group satisfies a preset convergence condition, determining the initial parameter value as the target parameter value, and obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter.

[0030] Optionally, the target scenario is a water transport scenario;

[0031] Before determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the method further includes: obtaining the water surface height of the water transport scene at each pre-recorded historical moment; determining the minimum value among the water surface heights of the water transport scene at each pre-recorded historical moment to obtain the water surface height of the calibration reference plane; for each historical moment, calculating the difference between the water surface height of the water transport scene at that historical moment and the water surface height of the calibration reference plane to obtain the water surface height difference corresponding to that historical moment; calculating the difference between the initial parameter value of the camera's reference height parameter and the water surface height difference corresponding to that historical moment, using it as the initial parameter value of the camera's height parameter at that historical moment, to obtain the initial transformation relationship between the pixel coordinate system and the radar coordinate system at that historical moment; wherein, the reference height parameter represents: the height of the camera from the calibration reference plane;

[0032] The step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system includes: for each target calibration data group, determining the position of the target region in the radar coordinate system in the target calibration data group based on the position of the target detection box in the pixel coordinate system in the target calibration data group and the initial transformation relationship of the historical time to which the target calibration data group belongs.

[0033] Optionally, the camera is a spherical camera; the radar is a millimeter-wave radar;

[0034] The calibration parameters include: the camera's height parameter; the angle between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the camera's P value is 0; the P value when the camera points in the direction the camera's base is tilted; the camera's mounting pitch angle; wherein the mounting pitch angle is the angle between the camera's base and the horizontal plane; and the error amount of the camera's T value.

[0035] Optionally, the transformation relationship between the pixel coordinate system and the radar coordinate system includes:

[0036] x = r * cos(yaw) obj +Δα)

[0037] y = r*sin(yaw) obj +Δα)

[0038]

[0039]

[0040]

[0041]

[0042] T1 = T + T0

[0043] yaw0=arctan2{sin(P-P0), cos[(P-P0)*|cosθ|]}

[0044] d v =v-0.5

[0045] d u =u-0.5

[0046] (x, y) represents the two-dimensional coordinates of the target feature point in the BEV coordinate system; r represents the distance between the target feature point and the origin of the radar coordinate system; (u, v) represents the normalized two-dimensional coordinates of the target feature point in the pixel coordinate system; pitch obj yaw represents the angle between the line connecting the optical center of the camera and the object and the horizontal plane. obj The azimuth angle of the target feature point in the radar coordinate system is represented by Δα; Δα represents the angle difference between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0; H represents the height parameter of the camera; T1 represents the actual T value of the camera. This indicates the tilt angle between the camera's base and the horizontal direction; d v d represents the offset of the normalized ordinate of the target feature point in the pixel coordinate system relative to the image center; u The value of F0V represents the offset of the normalized x-coordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the current Z value of the camera; v (Z) represents the vertical field of view (FOV) of the camera corresponding to its current Z value; u (Z) represents the horizontal and vertical field of view of the camera corresponding to the current Z value; yaw0 represents the angle through which the camera rotates on the horizontal plane when the camera rotates from P value P0 to P value P; T represents the current T value of the camera; T0 represents the error amount of the camera's T value; P represents the current P value of the camera; P0 represents the P value when the camera is pointing in the direction of tilt of the camera's base; θ represents the mounting pitch angle of the camera.

[0047] The BEV coordinate system takes the projection point of the radar in the horizontal plane as its origin, and the X-axis and Y-axis of the BEV coordinate system are located in the horizontal plane; the positive direction of the X-axis of the BEV coordinate system is the same as the positive direction of the X-axis of the radar coordinate system.

[0048] Secondly, in order to achieve the above objectives, embodiments of this application provide a camera and radar calibration device, the device comprising:

[0049] The data acquisition module is used to acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera.

[0050] The first position determination module is used to determine, based on the calibration azimuth data and the calibration image data, the position of the preset calibration point in the target object in the radar coordinate system of the radar, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera.

[0051] The second position determination module is used to determine the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system; wherein, the calibration parameter in the initial transformation relationship is an initial parameter value;

[0052] The shortest distance determination module is used to calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system.

[0053] The first transformation relationship determination module is used to adjust the initial parameter value of the calibration parameter based on the shortest distance, determine the target parameter value of the calibration parameter, and obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system that includes the target parameter value of the calibration parameter.

[0054] Optionally, the second position determination module is specifically used to determine the two-dimensional coordinates of a second specified point in the radar coordinate system corresponding to the first specified point in the target detection frame, based on the two-dimensional coordinates of the first specified point in the target detection frame in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system; and to calculate the horizontal and vertical field of view of the target object based on the Z value of the camera and the visual width and visual length of the target detection frame; wherein, the horizontal field of view of the target object is: the line connecting the camera and the left edge of the target object, and the line connecting the camera and the right edge of the target object. The angle between the lines connecting the side edges; the vertical field of view of the target object is the angle between the line connecting the camera and the upper edge of the target object and the line connecting the camera and the lower edge of the target object; the camera is a spherical camera; based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, the target width and target length of the target area corresponding to the target detection box in the radar coordinate system are calculated; based on the two-dimensional coordinates of the second designated point in the radar coordinate system, and the target width and target length, the position of the target area in the radar coordinate system is determined.

[0055] Optionally, the second position determination module is specifically used to calculate the horizontal and vertical field of view of the target object based on the Z-value of the camera, a first preset formula, and the visual width and visual length of the target detection box; wherein, the first preset formula includes:

[0056] γu =W u *FOV u (Z)

[0057] γ v =W v *FOV v (Z)

[0058] Y u W represents the horizontal field of view of the target object. u The field of view (FOV) represents the visual width of the target detection box. u (Z) represents the horizontal field of view of the camera corresponding to the Z-value; Z represents the Z-value of the camera; Y v W represents the vertical field of view of the target object. v The field of view (FOV) represents the visual length of the target detection box. v (Z) represents the vertical field of view of the camera corresponding to the Z value of the camera.

[0059] Optionally, the second position determination module is specifically used to calculate the initial width and initial length of the initial estimated region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, a second preset formula, and the horizontal and vertical field of view angles of the target object; wherein, the second preset formula includes:

[0060] w est =γu*R radar

[0061] h est =γ v *R radar

[0062] w est Indicates the initial width; γ u R represents the horizontal field of view of the target object. radar h represents the distance between the target object and the radar. est Indicates the initial length; γ v This indicates the vertical field of view of the target object;

[0063] Based on the initial width and the initial length, the target width and target length of the target detection box in the radar coordinate system are determined.

[0064] Optionally, the second position determination module is specifically used to adjust the initial width and the initial length according to preset adjustment parameters to obtain the target width and target length of the target area corresponding to the target detection box in the radar coordinate system.

[0065] Optionally, the first designated point is the midpoint of the bottom edge of the target detection box;

[0066] The second position determination module is specifically used to determine a rectangular area in the radar coordinate system with a width equal to the target width and a length equal to the target length, based on the second designated point as the center point of the target area and the direction of the line connecting the target object to the camera as the direction of the short side of the target area, thereby obtaining the position of the target area in the radar coordinate system.

[0067] Optionally, the data acquisition module is used to acquire various calibration azimuth data of the target scene collected by the radar at various historical moments, and multiple calibration image data of the target scene collected by the camera at various historical moments.

[0068] The first position determination module is specifically configured to, for each historical moment, obtain the position of a preset calibration point of a first object in the calibration azimuth data of that historical moment in the radar coordinate system; perform target detection on each calibration image data of that historical moment to obtain the position of the detection box of a second object in the calibration image data of that historical moment in the pixel coordinate system of the camera; combine the first object and the second object of that historical moment to obtain multiple calibration data sets; wherein, a calibration data set includes: the position of a preset calibration point of a first object in the radar coordinate system of the radar, and the position of the detection box of a second object in the pixel coordinate system of the camera; determine a target calibration data set containing the target object from the multiple calibration data sets; obtain the position of the preset calibration point of the target object in the target calibration data set in the radar coordinate system, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera.

[0069] Optionally, the first position determination module is specifically used to determine that the calibration data group is a target calibration data group containing the target object when the attribute information of the first object and the second object in the calibration data group meets the preset filtering conditions for each calibration data group; wherein, the attribute information of an object includes at least one of the following: the distance between the object and the radar, the object's speed, size, and direction of movement.

[0070] Optionally, the shortest distance determination module is specifically used to calculate, for each target calibration data group, the shortest distance from the preset calibration point in the target calibration data group to the target area in the radar coordinate system based on the position of the preset calibration point in the target calibration data group in the radar coordinate system and the position of the target area in the target calibration data group in the radar coordinate system, to obtain the shortest distance corresponding to the target calibration data group.

[0071] The first transformation relationship determination module is specifically used to adjust the initial parameter value of the calibration parameter to obtain the adjusted initial parameter value when the shortest distance corresponding to each target calibration data group does not meet the preset convergence condition; return to execute the step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, until the shortest distance corresponding to each target calibration data group meets the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter is obtained;

[0072] The device further includes:

[0073] The second transformation relationship determination module is used to determine the initial parameter value as the target parameter value when the shortest distance corresponding to each target calibration data group satisfies the preset convergence condition, and to obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter.

[0074] Optionally, the target scenario is a water transport scenario;

[0075] The device further includes:

[0076] The water surface height acquisition module is used to acquire the water surface height of the water transport scene at each historical moment before the second position determination module executes the determination of the position of the target area corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the initial transformation relationship between the pixel coordinate system and the radar coordinate system established in advance.

[0077] The water surface height determination module is used to determine the minimum value among the water surface heights of the water transport scenario at each pre-recorded historical moment, and to obtain the water surface height of the calibration reference plane;

[0078] The water surface height difference determination module is used to calculate the difference between the water surface height of the water transport scenario and the water surface height of the calibration reference plane for each historical moment, so as to obtain the water surface height difference corresponding to that historical moment.

[0079] The third transformation relationship determination module is used to calculate the difference between the initial parameter value of the camera's reference height parameter and the water surface height difference corresponding to the historical moment, and use it as the initial parameter value of the camera's height parameter at the historical moment to obtain the initial transformation relationship between the pixel coordinate system and the radar coordinate system at the historical moment; wherein, the reference height parameter represents the height of the camera from the calibration reference plane;

[0080] The second position determination module is specifically used to determine the position of the target region in the radar coordinate system for each target calibration data group, based on the position of the target detection box in the pixel coordinate system in the target calibration data group and the initial transformation relationship of the historical time to which the target calibration data group belongs.

[0081] Optionally, the camera is a spherical camera; the radar is a millimeter-wave radar;

[0082] The calibration parameters include: the camera's height parameter; the angle between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the camera's P value is 0; the P value when the camera points in the direction the camera's base is tilted; the camera's mounting pitch angle; wherein the mounting pitch angle is the angle between the camera's base and the horizontal plane; and the error amount of the camera's T value.

[0083] Optionally, the transformation relationship between the pixel coordinate system and the radar coordinate system includes:

[0084] x = r * cos(yaw) obj +Δα)

[0085] y = r*sin(yaw) obj +Δα)

[0086]

[0087]

[0088]

[0089]

[0090] T1 = T + T0

[0091] yaw0=arctan2{sin(P-P0), cos[(P-P0)*|cosθ|]}

[0092] d v =v-0.5

[0093] d u =u-0.5

[0094] (x, y) represents the two-dimensional coordinates of the target feature point in the BEV coordinate system; r represents the distance between the target feature point and the origin of the radar coordinate system; (u, v) represents the normalized two-dimensional coordinates of the target feature point in the pixel coordinate system; pitch obj yaw represents the angle between the line connecting the optical center of the camera and the object and the horizontal plane. obj The azimuth angle of the target feature point in the radar coordinate system is represented by Δα; Δα represents the angle difference between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0; H represents the height parameter of the camera; T1 represents the actual T value of the camera. This indicates the tilt angle between the camera's base and the horizontal direction; d v d represents the offset of the normalized ordinate of the target feature point in the pixel coordinate system relative to the image center; u The FOV represents the offset of the normalized x-coordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the current Z value of the camera; FOV v (Z) represents the vertical field of view (FOV) of the camera corresponding to its current Z value; u (Z) represents the horizontal and vertical field of view of the camera corresponding to the current Z value; yaw0 represents the angle through which the camera rotates on the horizontal plane when the camera rotates from P value P0 to P value P; T represents the current T value of the camera; T0 represents the error amount of the camera's T value; P represents the current P value of the camera; P0 represents the P value when the camera is pointing in the direction of tilt of the camera's base; θ represents the mounting pitch angle of the camera.

[0095] The BEV coordinate system takes the projection point of the radar in the horizontal plane as its origin, and the X-axis and Y-axis of the BEV coordinate system are located in the horizontal plane; the positive direction of the X-axis of the BEV coordinate system is the same as the positive direction of the X-axis of the radar coordinate system.

[0096] This application also provides an electronic device, including: a memory for storing computer programs; and a processor for executing the program stored in the memory to implement any of the camera and radar calibration methods described above.

[0097] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the camera and radar calibration methods described above.

[0098] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the camera and radar calibration methods described above.

[0099] Beneficial effects of the embodiments in this application:

[0100] This application provides a camera and radar calibration method that acquires calibration azimuth data of a target scene collected by radar and calibration image data of the target scene collected by camera. Based on the calibration azimuth data and calibration image data, the method determines the position of a preset calibration point in the target object within the radar coordinate system, and the position of the target detection box in the calibration image data within the camera's pixel coordinate system. Based on the position of the target detection box in the pixel coordinate system and a pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the method determines the position of the target region corresponding to the target detection box in the radar coordinate system, where the calibration parameters in the initial transformation relationship are initial parameter values. Based on the position of the preset calibration point in the radar coordinate system and the position of the target region corresponding to the target detection box in the radar coordinate system, the method calculates the shortest distance from the preset calibration point to the target region in the radar coordinate system. Based on the shortest distance, the initial parameter values ​​of the calibration parameters are adjusted to determine the target parameter values ​​of the calibration parameters, thus obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system, which includes the target parameter values ​​of the calibration parameters.

[0101] Based on the above processing, the target transformation relationship between the pixel coordinate system and the radar coordinate system can be automatically determined without the need for manual measurement of the target scene's calibration parameters. This reduces the cost and improves the efficiency of camera and radar calibration. Furthermore, the preset calibration points within the target object are defined as the calibration points within the target object, and the target area is defined as the region where the target object is located. Correspondingly, since the preset calibration points are located within the target area, adjusting the calibration parameter values ​​based on the shortest distance from the preset calibration points in the radar coordinate system to the target area can improve the accuracy of the determined target transformation relationship. Subsequently, precise conversion between positional data and image data can be performed based on the target transformation relationship, improving the accuracy of ship monitoring in water transport scenarios.

[0102] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0103] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0104] Figure 1 A flowchart illustrating the first camera and radar calibration method provided in this application embodiment;

[0105] Figure 2 A flowchart illustrating the second camera and radar calibration method provided in this application embodiment;

[0106] Figure 3 This is a comparison diagram of calibration image data and calibration orientation data provided in an embodiment of this application.

[0107] Figure 4 A schematic diagram illustrating the transformation relationship between a pixel coordinate system and a radar coordinate system provided in an embodiment of this application;

[0108] Figure 5 This application provides a schematic diagram of the position of a ship in calibration image data.

[0109] Figure 6 A schematic diagram illustrating the first type of camera rotation provided in this application embodiment;

[0110] Figure 7 A schematic diagram of camera installation provided for an embodiment of this application;

[0111] Figure 8 A schematic diagram illustrating a second type of camera rotation provided in an embodiment of this application;

[0112] Figure 9 A schematic diagram illustrating a camera rotation imaging tilt provided in an embodiment of this application;

[0113] Figure 10 A schematic diagram of the horizontal and vertical angular components of a ship after the camera is rotated, provided in an embodiment of this application;

[0114] Figure 11 A flowchart illustrating the third camera and radar calibration method provided in this application embodiment;

[0115] Figure 12 This application provides a schematic diagram of the position of a ship in calibration image data.

[0116] Figure 13 A schematic diagram of the position of a target area in a radar coordinate system provided in an embodiment of this application;

[0117] Figure 14 A flowchart illustrating the fourth camera and radar calibration method provided in this application embodiment;

[0118] Figure 15 A schematic diagram of a tide table provided in an embodiment of this application;

[0119] Figure 16 A structural diagram of a camera and radar calibration device provided in an embodiment of this application;

[0120] Figure 17 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0121] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0122] Ship monitoring technology can play a crucial role in waterway data statistics. However, unlike typical land transportation scenarios, ships in waterway scenarios have a wider range of movement and more widely distributed routes. Furthermore, the conditions for setting up cameras in waterway scenarios are more demanding. In most waterway scenarios, the available locations for setting up cameras are limited, and cameras can often only be set up in certain areas along the shore. All of these factors pose challenges to ship monitoring in waterway scenarios.

[0123] Therefore, traditional fixed-focus, fixed-angle cameras are not effectively adapted to water transport scenarios. A switch to a PTZ camera, capable of horizontal and vertical rotation and zoom, is necessary. PTZ cameras collect image data of water transport scenarios, enabling continuous monitoring of ships within these scenarios. However, PTZ cameras also have perception limitations. For example, during continuous monitoring, they cannot effectively detect objects outside their field of view. Furthermore, while PTZ cameras can monitor objects through rotation and zoom, they fail to capture and monitor objects in situations with poor field of view or when the object is too far away or slightly off-center.

[0124] Millimeter-wave radar is independent of visual conditions and can sensitively detect moving objects in unobstructed environments. Furthermore, it boasts a wide sensing angle and long range. These characteristics of millimeter-wave radar effectively compensate for the shortcomings of PTZ cameras. Therefore, combining millimeter-wave radar with a PTZ camera creates a millimeter-wave radar-PTZ camera monitoring system. This system can utilize the azimuth data acquired by the millimeter-wave radar to provide a global view of the water transport scene and use the image data acquired by the PTZ camera for continuous object detection, achieving superior ship monitoring results.

[0125] Since the azimuth data acquired by millimeter-wave radar and the image data acquired by PTZ camera are two different modes, the above monitoring system needs to be accurately converted between the azimuth data acquired by radar and the image data acquired by PTZ camera in order to improve the ship monitoring effect.

[0126] In a water transport scenario without water surface undulations, theoretically, given the camera's intrinsic and extrinsic parameters and the ship's position in the image data, it's possible to convert the data to the ship's position on the water surface (BEV (Bird's Eye View) plane). Furthermore, the ship's bearing data can also be processed to obtain its position in the BEV plane, and since the same ship has the same position in the BEV plane, the transformation relationship between the pixel coordinate system and the radar coordinate system can be determined.

[0127] Because the camera is constantly moving and zooming, its intrinsic and extrinsic parameters are also changing in real time. Therefore, an accurate explicit model was developed to accurately model the changes in intrinsic and extrinsic parameters during camera movement. This model yielded a function model representing the transformation relationship between the pixel coordinate system and the radar coordinate system sphere, which includes several calibration parameters with practical significance.

[0128] The calibration parameters mentioned above can be obtained through manual measurement, but manual measurement has low accuracy and cannot meet the accuracy requirements of ship monitoring systems. Furthermore, the labor and time costs of each measurement are high. Alternatively, the calibration parameters can be determined by technicians through repeated adjustments on calibration data containing ship-related POS information (ship's position, heading, and speed, etc.), but this requires extensive calibration skills, experience, and a considerable amount of time.

[0129] This application provides a simple and efficient calibration method for use in electronic devices. With only a small amount of calibration data, the electronic device automatically fits the calibration parameters and automatically regresses to determine the target parameter values, thus obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system, which includes the target parameter values ​​of the calibration parameters.

[0130] The electronic device acquires calibration azimuth data of the target scene collected by the radar, calibration image data of the target scene collected by the camera, and a pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system. The calibration parameters in the initial transformation relationship are the initial parameter values. Then, based on the calibration azimuth data, calibration image data, and the aforementioned initial transformation relationship, the shortest distance from the preset calibration point of the target object to the target area in the radar coordinate system is calculated. Based on the shortest distance, the initial parameter values ​​of the calibration parameters are adjusted to determine the target parameter values ​​of the calibration parameters, thus obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system that includes the target parameter values ​​of the calibration parameters.

[0131] Correspondingly, after establishing the target transformation relationship between the pixel coordinate system and the radar coordinate system, in addition to being applied to ship monitoring systems to improve the accuracy of ship monitoring, the target transformation relationship can also be applied to radar visual perception fusion, target detection and capture, and other fields.

[0132] The following description focuses on the camera and radar calibration method provided in this application, based on a water transport scenario. However, the applicability of the camera and radar calibration method includes, but is not limited to, water transport scenarios.

[0133] See Figure 1 , Figure 1 A flowchart of a camera and radar calibration method provided in this application embodiment, the method may include the following steps:

[0134] S101: Acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera.

[0135] S102: Based on the calibration azimuth data and calibration image data, determine the position of the preset calibration point in the radar coordinate system of the target object, and the position of the target detection box in the calibration image data in the pixel coordinate system of the camera.

[0136] S103: Based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, determine the position of the target area corresponding to the target detection box in the radar coordinate system.

[0137] Among them, the calibration parameters in the initial transformation relationship are the initial parameter values.

[0138] S104: Based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system, calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system.

[0139] S105: Adjust the initial parameter values ​​of the calibration parameters based on the shortest distance, determine the target parameter values ​​of the calibration parameters, and obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter values ​​of the calibration parameters.

[0140] Based on the camera and radar calibration method provided in this application, the target transformation relationship between the pixel coordinate system and the radar coordinate system can be determined without manually measuring the calibration parameters of the target scene. This reduces the cost and improves the efficiency of camera and radar calibration. Furthermore, the preset calibration point in the target object is the calibration point within the target object, and the target area is the area where the target object is located. Correspondingly, the preset calibration point is located within the target area. Adjusting the calibration parameter values ​​based on the shortest distance from the preset calibration point to the target area in the radar coordinate system improves the accuracy of the determined target transformation relationship. Subsequently, precise conversion between positional data and image data can be performed based on the target transformation relationship, improving the accuracy of ship monitoring in water transport scenarios.

[0141] For step S101, the target scenario can be a water transport scenario, where the camera and radar are mounted at fixed positions on the shore, and the spatial ranges detected by the camera and radar overlap, allowing the camera and radar to detect the same vessel. For example, the camera and radar can be integrated into one unit. Alternatively, after mounting the camera, the radar can be installed directly below the camera and within the camera's field of view.

[0142] Radar is used to detect the position and distance of target objects (such as ships) in water transport scenarios. The radar is a millimeter-wave radar, which emits millimeter-wave signals and, based on the millimeter-wave signals reflected by the ship, obtains azimuth data including the ship's polar coordinates in the radar coordinate system. The ship's polar coordinates in the radar coordinate system include the distance between the ship and the radar, as well as the ship's azimuth angle.

[0143] The camera is used to acquire real-time image data of water transport scenes. The camera is a spherical camera (PTZ). A PTZ camera has a gimbal and other mechanical structures, allowing for PTZ (Pan Tilt Zoom, representing omnidirectional (left / right / up / down) movement of the gimbal and lens zoom control) changes, capturing both panoramic and detailed information of the scene. P (Pan, panoramic view) represents the horizontal angle of the PTZ camera, T (Tilt, tilt) represents the pitch angle, and Z (Zoom, zoom mode) represents the zoom value. Thanks to the gimbal control structure, the PTZ camera can cover 360 degrees horizontally and more than 90 degrees vertically.

[0144] Correspondingly, electronic devices acquire calibration image data of the target scene collected by the camera, as well as calibration azimuth data of the target scene collected by the radar.

[0145] In some embodiments, for water transport scenarios, since the scanning frequency of radar acquiring azimuth data and the frame rate of camera acquiring image data may be inconsistent, the acquired image data and azimuth data are not acquired by the camera and radar at the same time. The water surface height in the water transport scenario may be different at different times, which will lead to lower accuracy in determining the target conversion relationship.

[0146] To acquire image and azimuth data from both the camera and radar simultaneously, and to further improve the accuracy of the determined target conversion relationship, the electronic device can acquire a single frame of image data captured by the camera at a historical moment, as calibration image data. Furthermore, the electronic device can extract the azimuth data from the radar-acquired azimuth data whose timestamp differs the smallest from the timestamp of the calibration image data, and use this as the calibration azimuth data.

[0147] Based on the above processing, the calibration azimuth data with the smallest difference between the timestamp and the timestamp of the calibration image data can be obtained. This allows us to obtain the image data and azimuth data collected by the camera and radar at the same time, avoiding the impact of different water surface heights in water transport scenarios at different times on the subsequent determination of target conversion relationships, and improving the accuracy of the determined target conversion relationships.

[0148] Regarding step S102, the camera's pixel coordinate system is the coordinate system of the image data acquired by the camera. The pixel coordinate system has its origin at the top left corner of the image data, and its X-axis and Y-axis are parallel to the two vertical edges of the image data, respectively. The radar coordinate system is a polar coordinate system with the radar's transmission signal outlet as its origin. The positive direction of the radar coordinate system's X-axis is consistent with the positive direction of the camera when its P-value is 0. The positive direction of the camera refers to the direction the camera lens is pointing when the camera's P-value is 0.

[0149] Preset calibration points are feature points on a target object that are pre-calibrated by technicians. For example, when a radar collects calibration azimuth data, it measures the distance between a fixed point on the target object (i.e., the radar detection point) and the radar; in this case, the preset calibration point is the radar detection point.

[0150] Calibration azimuth data is the azimuth information of the detected target object. This azimuth information is represented as the polar coordinates of the target object in the radar coordinate system. These polar coordinates include the distance between the target object and the radar and the azimuth angle of the target object in the horizontal direction. Electronic equipment directly acquires the polar coordinates of the radar detection points in the calibration azimuth data to obtain the position of the preset calibration point in the radar coordinate system.

[0151] The calibration image data is a calibration image containing the target object. Electronic devices perform target detection on the calibration image data to obtain the position of the target detection box containing the target object in the pixel coordinate system. The target detection box is the smallest bounding rectangle of the image region occupied by the target object in the calibration image data. The position of the target detection box in the pixel coordinate system is represented by: the two-dimensional coordinates of the four vertices of the target detection box in the pixel coordinate system, or the two-dimensional coordinates of two vertices of the target detection box in the pixel coordinate system, and the visual length and visual width of the target detection box.

[0152] In some embodiments, Figure 1 Based on this, see Figure 2 Step S101 may include the following steps:

[0153] S1011: Acquire the calibration azimuth data of the target scene collected by the radar at various historical moments, as well as the multiple calibration image data of the target scene collected by the camera at various historical moments.

[0154] Accordingly, step S102 may include the following steps:

[0155] S1021: For each historical moment, obtain the position of the preset calibration point of the first object in the radar coordinate system from the calibration azimuth data of that historical moment.

[0156] S1022: Perform target detection on each calibration image data of the historical moment to obtain the position of the detection box of the second object in the calibration image data of the historical moment in the camera's pixel coordinate system.

[0157] S1023: Combine the first and second objects of this historical moment to obtain multiple calibration data sets.

[0158] One calibration data set includes: the position of a preset calibration point of a first object in the radar coordinate system, and the position of a detection box of a second object in the camera pixel coordinate system.

[0159] S1024: Determine the target calibration data set containing the target object from multiple calibration data sets.

[0160] S1025: Obtain the position of the preset calibration point of the target object in the target calibration data set in the radar coordinate system, and the position of the target detection box of the target object in the calibration image data in the camera pixel coordinate system.

[0161] To improve the accuracy of the determined target conversion relationship, the electronic equipment acquires multiple calibration azimuth data of the target scene collected by the radar at various historical moments, as well as multiple calibration image data of the target scene collected by the camera at various historical moments. Subsequently, the radar and camera are calibrated based on the calibration azimuth data and calibration image data at various historical moments.

[0162] The calibration image data includes objects in the target scene (i.e., the first object), and the calibration orientation data also includes objects in the target scene (i.e., the second object). The first and second objects contain some identical objects. Therefore, the electronic device needs to determine whether the first and second objects are the same object, and identify the target object based on the identical objects in the first and second objects.

[0163] For each historical moment, the electronic device obtains the polar coordinates of the preset calibration point of the first object in the calibration azimuth data of that historical moment in the radar coordinate system. Furthermore, it performs target detection on each calibration image data of that historical moment to obtain the position of the detection box of the second object in the calibration image data of that historical moment in the pixel coordinate system.

[0164] For each historical moment, since it is not yet determined whether the first object in the calibration azimuth data and the second object in the calibration image data are the same object, the electronic device randomly combines each first object and each second object for that historical moment to obtain multiple calibration data sets. Each calibration data set includes: the polar coordinates of a preset calibration point of a first object in the radar coordinate system, and the position of the detection box of a second object in the pixel coordinate system.

[0165] Then, the electronic device can determine whether the first object and the second object are the same object based on one or more of the attribute information of the first object and the second object.

[0166] In some embodiments, step S1024 may include the following steps: for each calibration data group, when the attribute information of the first object and the second object in the calibration data group meets the preset filtering conditions, the calibration data group is determined to be a target calibration data group containing the target object.

[0167] The attribute information of an object includes at least one of the following: the distance between the object and the radar, the object's speed, size, and direction of movement. The size of an object includes its width and height.

[0168] For each first object, the electronic device can directly acquire the distance between the object and the radar from the calibration azimuth data. For each second object, the electronic device can determine the distance between the second object and the camera based on the position of the second object in the pixel coordinate system. Since the radar and the camera are installed in the same location, the distance between the second object and the radar is also obtained.

[0169] For each historical moment, the electronic device calculates the speed and direction of motion of the first object at that historical moment based on its position in the calibration image data at that historical moment, its position in the calibration image data of the previous historical moment, and the time difference between that historical moment and the previous historical moment. Furthermore, the electronic device calculates the speed and direction of motion of the second object at that historical moment based on its distance from the radar at that historical moment, its distance from the radar at the previous historical moment, and the time difference between that historical moment and the previous historical moment.

[0170] In one implementation, the preset filtering conditions include any one of the following: the difference between the distance between the first object and the radar and the distance between the second object and the radar is within a preset distance range; the difference between the movement speed of the first object and the movement speed of the second object is within a preset speed range; the movement direction of the first object is the same as the movement direction of the second object; the difference between the width of the first object and the width of the second object is within a preset length range, and the difference between the height of the first object and the height of the second object is within a preset height range.

[0171] For each calibration data set, if the attribute information of the first object and the second object in the calibration data set satisfies at least one of the above, the first object and the second object in the calibration data set are determined to be the same object.

[0172] In another implementation, for each object, the electronic device generates a sequence of feature information including the distance between the object and the radar, the object's size, speed, and direction of motion. Then, the electronic device calculates the correlation between the feature information sequences of the first object and the feature information sequences of the second object using a preset association algorithm. The preset association algorithm could be the Hungarian algorithm, the nearest neighbor algorithm, or the cosine similarity algorithm, etc.

[0173] For each calibration data group, if the correlation between the first object and the second object in the calibration data group is greater than a preset correlation threshold, the attribute information of the first object and the second object in the calibration data group is determined to meet the preset filtering conditions, that is, the first object and the second object in the calibration data group are determined to be the same object.

[0174] For each calibration data set, if the first object and the second object in that calibration data set are the same object, then that object is determined to be the target object, and that calibration data set is designated as the target calibration data set. Alternatively, the electronic device can select a specified number of calibration data sets from those containing calibration data sets where the first object and the second object are the same object to obtain the target calibration data set. Then, the polar coordinates of the preset calibration point of the target object in the target calibration data set within the radar coordinate system are obtained, as well as the position of the target detection box of the target object in the calibration image data within the camera's pixel coordinate system.

[0175] Furthermore, each target calibration data set also records the P-value, T-value, and Z-value of the camera at the corresponding historical moment. Subsequently, when the electronic device determines the position of the target detection box of the target object in the target calibration data set in the pixel coordinate system, the initial transformation relationship is the P-value, T-value, and Z-value of the camera at the corresponding historical moment of the calibration data set.

[0176] For example, see Figure 3 , Figure 3 This is a comparison diagram of calibration image data and calibration orientation data provided in an embodiment of this application. The target scenario is a water transport scenario. Figure 3 The left side of the image shows calibration image data from 16:42:27 on May 31, 2023, while the right side shows calibration bearing data from the same time. The calibration image data contains images of multiple vessels in a water transport scenario, and the calibration bearing data contains the bearing information of each vessel at the same time. For example, calibration image data labeled 17 and calibration bearing data labeled 83 form a target calibration data group, and the vessel in the detection box in the calibration image labeled 17 and the vessel in the calibration bearing data labeled 83 are the target vessels.

[0177] In some embodiments, technicians may filter the various calibration data sets to select the target calibration data set containing the same target object from multiple calibration data sets.

[0178] Based on the above processing, the same object (i.e., the target object) in the calibration azimuth data and calibration image data can be identified. Calibration is performed based on the polar coordinates of the target object's preset calibration point in the radar coordinate system and the position of the target detection box in the pixel coordinate system within the calibration image data. This improves the accuracy of the determined target transformation relationship. Furthermore, determining whether the first object and the second object are the same object based on the object's attribute information further improves the accuracy of identifying the target object, thereby enhancing the accuracy of the determined target transformation relationship.

[0179] For step S103, the initial transformation relationship is obtained by setting the parameter values ​​of the calibration parameters in the function model representing the transformation relationship between the pixel coordinate system and the radar coordinate system sphere to the initial parameter values. This function model includes: a function model between the pixel coordinate system and the BEV coordinate system, and a function model between the radar coordinate system and the BEV coordinate system.

[0180] The BEV coordinate system uses the radar's projection point on the horizontal plane as its origin. The X and Y axes of the BEV coordinate system lie within the horizontal plane, and the positive direction of the X-axis is the same as the positive direction of the X-axis in the BEV coordinate system. The radar's projection point on the horizontal plane is the projection point of the radar's transmitted signal outlet on the horizontal plane.

[0181] Given a set of calibration parameters (a1, a2, a3, a4, a5) for the function model, a1, a2, a3, a4, a5 are respectively the camera's height parameter H; the angle Δα between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the camera's P value is 0; the camera's mounting pitch angle θ; the P value P0 when the camera is pointing in the direction the camera's base is tilted; and the error T0 of the camera's T value.

[0182] The function model between the pixel coordinate system and the BEV coordinate system is used to transform the normalized two-dimensional coordinates (u, v) ∈ [0, 1] of the target feature points in the pixel coordinate system to two-dimensional coordinates (x, y) in the BEV coordinate system. Accordingly, the function model between the pixel coordinate system and the BEV coordinate system can be defined as: f1(u, v, P, T, Z, a1, a2, a3, a4, a5) → (x, y). The function model between the radar coordinate system and the BEV coordinate system is used to transform the two-dimensional coordinates (x, y) of the target feature points in the BEV coordinate system to polar coordinates (r, α) in the radar coordinate system. Accordingly, the function model between the radar coordinate system and the BEV coordinate system can be defined as: f2(x, y, a1, a2, a3, a4, a5) → (r, α). Where P, T, and Z represent the camera's P value, T value, and Z value, respectively. Furthermore, the function model between the pixel coordinate system and the radar coordinate system can be defined as: f3(u, v, P, T, Z, a1, a2, a3, a4, a5) → (r, α).

[0183] For a water transport scenario, the target object is the target vessel. The following explanation uses a water transport scenario as an example to illustrate the transformation relationship between the pixel coordinate system and the radar coordinate system.

[0184] When setting up radar and cameras, ensure that the cameras and radar meet the following conditions:

[0185] (1) The camera base is parallel to the horizontal plane;

[0186] (2) The positive direction of the X-axis of the radar coordinate system coincides with the positive direction of the camera when the camera P value is 0, and the positive direction of the X-axis of the radar coordinate system is consistent with the positive direction of rotation according to the camera P value.

[0187] The positive direction of the radar coordinate system's X-axis aligns with the positive direction of the camera's rotation. This includes situations where the camera rotates clockwise, in which case the radar coordinate system is left-handed, meaning the positive direction of the X-axis is to the left. Alternatively, if the camera rotates counter-clockwise, the radar coordinate system is right-handed, meaning the positive direction of the X-axis is to the right.

[0188] (3) When the camera's T value is 0 at the factory, the camera's optical axis is parallel to the camera base. When the camera rotates according to the T value, the downward rotation is the positive direction.

[0189] Assuming the target vessel is located at the center of the camera's field of view, then the target vessel will be located at the center of the calibrated image data when imaged. See also Figure 4 Let P, T, and Z be the current P, T, and Z values ​​of the camera, respectively. The camera's height parameter, i.e., its height above the water surface, is denoted as H. Correspondingly, the angle between the line connecting the camera and the target vessel in space and the horizontal plane is pitch. obj This refers to the camera's current T-value, or pitch. obj =T. The azimuth angle (yaw) of the target vessel in the radar coordinate system (polar coordinates). obj =P. Correspondingly, the polar coordinates (r, α) of the target vessel in the radar coordinate system can be expressed as:

[0190] Therefore, under ideal conditions, in addition to the camera's current P, T, and Z values, only the camera's height parameter H is needed to transform the normalized two-dimensional coordinates (0.5, 0.5) of the target ship in the pixel coordinate system to the two-dimensional coordinates (x, y) in the BEV coordinate system. Thus, the camera's height parameter H is one of the five calibration parameters.

[0191] It is understandable that the coordinates of the target ship in the pixel coordinate system are represented by the two-dimensional coordinates of the target feature points (such as the center point) of the target ship. Since the target ship is located at the center of the calibration image data, after normalizing the two-dimensional coordinates, the normalized two-dimensional coordinates of the target ship in the pixel coordinate system can be obtained as (0.5, 0.5).

[0192] This applies when the target vessel is not located at the center of the camera's field of view, meaning the target vessel is not located at the center of the calibration image data during imaging (i.e., Figure 5 For the case of the image center (see [image center]), see [link] Figure 5 The offset of the target ship's normalized x-coordinate in the pixel coordinate system relative to the center of the calibrated image data is denoted as d. uThe offset of the target ship's normalized ordinate in the pixel coordinate system relative to the center of the calibrated image data is denoted as d. v By consulting the Z-FOV (Field of View) table based on the camera's current Z value, the camera's horizontal field of view (FOV) can be determined. u (Z) and vertical field of view (FOV) v (Z).

[0193] Furthermore, d can be determined based on the offset direction of the target vessel relative to the center of the calibrated image data. u and d v For example, when the target vessel is offset downwards relative to the center of the calibrated image data, then d is determined. v If it is a positive value, then d is determined. v The value is negative. When the target vessel shifts to the left relative to the center of the calibration image data, d is determined. u If it is a positive value, then d is determined. u It is a negative value.

[0194] Approximately, the camera's field of view can be assumed to be uniformly distributed across the calibration image data. The horizontal field of view angle of the target ship at the center of the camera's field of view can be calculated as d. u *FOV u (Z), the vertical field of view of the target ship from the center of the camera's field of view is d. v *FOV v (Z). Based on this, Figure 4 The angle between the line connecting the camera and the target ship in the space shown and the horizontal plane is pitch. obj =T+d v *FOV v (Z), the azimuth angle (yaw) of the target ship in the polar coordinates of the radar coordinate system. obj =P+d u *FOV u (Z).

[0195] However, in actual radar and camera installation, it is often difficult to meet the above conditions. For example: the camera is not installed horizontally, i.e., the camera base is not parallel to the horizontal plane; the camera's T-value has a certain deviation at the factory; the positive direction of the camera's P-value (0) is different from the positive direction of the radar coordinate system's X-axis, i.e., the positive direction of the camera's rotation according to the P-value is different from the positive direction of the radar coordinate system's X-axis. These three actual deviations introduce four calibration parameters. These four calibration parameters include: the error T0 of the camera's T-value; the angular difference Δα between the positive direction of the camera and the positive direction of the radar coordinate system's X-axis when the camera's P-value is 0; the P-value P0 when the camera is tilted towards the direction of the camera base's tilt; and the camera's installation pitch angle θ. The direction in which the camera is tilted towards the camera base refers to the direction in which the camera lens is tilted towards the camera base.

[0196] See Figure 6 The P value P0 when the camera is pointing in the direction the camera base is tilted indicates that the camera has rotated left or right starting from the position P=O. Figure 6 When rotating in the direction indicated by P, the angle rotated to the direction pointing towards the camera's base tilt. The camera's mounting pitch angle represents the angle between the camera's base and the horizontal plane.

[0197] The above four calibration parameters, plus the camera height parameter, make a total of five calibration parameters. These parameters include: the camera height parameter; the angle between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the camera's P-value is 0; the P-value P0 when the camera is pointing in the direction the camera base is tilted; the camera's mounting pitch angle; and the error amount of the camera's T-value.

[0198] Because cameras can freely tilt and rotate horizontally, representing the installation errors of the initial pitch and roll angles during camera mounting is not intuitive. See also Figure 7 If we consider the camera as being mounted on an inclined plane, then the camera's initial pitch and roll angles can be equivalently converted into two parameters: the P-value P0 when the camera rotates to be parallel to the inclined plane and the tilt angle of the inclined plane (i.e., the camera's mounting pitch angle) θ. When the camera's T-value is fixed at 0, and the camera is rotated to any P-value, see [reference needed]. Figure 8 d p Let d represent the change in the camera's P value as the camera rotates from P value P0 to P value P. p =P-P0. At this moment, the pitch angle between the optical center of the camera and the horizontal plane is pitch0 = arcsin[cos(d p )*sinθ], and the angle through which the camera rotates in the BEV coordinate system is yaw0=arctan2{sin(d p ), cos[(d p )*|cosθ|]}.

[0199] In the expression for yaw0 above, the arctan2 function is used to return the specified coordinates of the point {sin(d) in the BEV coordinate system. p ), cos[(d p The arctangent value of )*|cosθ|]. The arctangent value is: the value of the x-axis of the BEV coordinate system rotated counterclockwise to the origin of the BEV coordinate system and the coordinate point {sin(d p ), cos[(d p The angle that the line connecting )*|cosθ|]} needs to rotate through.

[0200] See Figure 9 After the camera moves from a value of P0 to a value of P, it changes from horizontal to tilted. The tilt angle between the camera base and the horizontal direction is... The camera moves in the direction of motion according to the T value (i.e.) Figure 9 The direction shown in T1 is perpendicular to the bottom edge of the camera. See also Figure 9 The motion of the camera, according to its T-value, can be decomposed into horizontal and vertical motions. Therefore, the pitch angle between the camera's optical center and the horizontal plane after the motion is... And the angle through which the camera rotates in the BEV coordinate system is

[0201] like Figure 10 As shown, after the camera moves according to the P and T values, if the target ship is not in the center of the frame, the angular offset between the target ship and the center of the calibration image data needs to be decomposed into horizontal and vertical directions. The angular offset between the target ship and the center of the calibration image data in the horizontal direction is... The vertical angular offset between the target vessel and the center of the calibration image data.

[0202] Furthermore, the pitch angle between the line connecting the camera and the target ship in space and the horizontal plane can be obtained. obj =pitch1+d v1 *FOV v (Z), and the azimuth angle yaw of the target ship in the polar coordinate system of the radar coordinate system. obj =yaw1+d u1 *FOV u (Z).

[0203] Accordingly, the transformation relationship between the pixel coordinate system and the radar coordinate system includes:

[0204] x = r * cos(yaw) obj +Δα) (1)

[0205] y = r*sin(yaw) obj+Δα) (2)

[0206]

[0207]

[0208]

[0209]

[0210] T1 = T + T0 (7)

[0211] yaw0=arctan2{sin(P-P0), cos[(P-P0)*|cosθ|]} (8)

[0212] d v =v-0.5 (9)

[0213] d u =u-0.5 (10)

[0214] (x, y) represents the two-dimensional coordinates of the target feature point in the BEV coordinate system; r represents the distance between the target feature point and the origin of the radar coordinate system; (u, v) represents the normalized two-dimensional coordinates of the target feature point in the pixel coordinate system; pitch obj yaw represents the angle between the line connecting the optical center of the camera and the object and the horizontal plane. obj Δα represents the azimuth angle of the target feature point in the radar coordinate system; Δα represents the angle difference between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the camera's P value is 0; H represents the camera's height parameter; T1 represents the actual T value of the camera. This indicates the tilt angle between the camera's base and the horizontal direction; d v d represents the offset of the normalized ordinate of the target feature point in the pixel coordinate system relative to the image center; u The FOV represents the offset of the normalized x-coordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the current Z value of the camera; FOV v (Z) represents the camera's vertical field of view corresponding to the current Z value; FOV u (Z) represents the horizontal and vertical field of view of the camera corresponding to the current Z value; yaw0 represents the angle the camera rotates on the horizontal plane when rotating from P value P0 to P value P; T represents the current T value of the camera; T0 represents the error of the camera's T value; P represents the current P value of the camera; P0 represents the P value when the camera is pointing in the direction of the camera's base tilt; θ represents the camera's mounting pitch angle; (r, yaw obj ) represents the polar coordinates of the target feature point in the radar coordinate system.

[0215] In constructing the transformation relationship between the pixel coordinate system and the radar coordinate system, the influence of multiple error factors was considered, and these factors were accurately modeled and calculated, resulting in a high-precision transformation relationship from the pixel coordinate system to the radar coordinate system. This allows for high-precision coordinate transformations and improves usability in engineering practice, meeting the requirements of downstream high-precision multimodal sensing. Furthermore, in practical applications, ships may be located at any position in the calibration image data. The transformation relationship between the pixel coordinate system and the radar coordinate system was designed to accommodate situations where the ship is not at the center of the calibration image data, enabling mutual transformations from any position in the calibration image data to the radar coordinate system. This demonstrates strong versatility and expands the applicability of the determined target transformation relationship.

[0216] After establishing a function model with five calibration parameters, a relatively intuitive way to determine these calibration parameters is to perform regression on them. This involves obtaining the polar coordinates of multiple pairs of known preset calibration points in the radar coordinate system, and the two-dimensional coordinates of the preset calibration points in the pixel coordinate system. The polar coordinates of the preset calibration points in the radar coordinate system are then used as the coordinates (r, yaw) in formulas (1) to (10) above. obj The two-dimensional coordinates of the preset calibration point in the pixel coordinate system are used as (u, v) in the above formulas (1) to (10) to obtain a set of equations about the calibration parameters H, P0, T0, θ, and Δα. By solving the set of equations through a regression algorithm, the initial parameter values ​​of the calibration parameters H, P0, T0, θ, and Δα can be obtained. Based on the calculated initial parameter values, the polar coordinates of the known preset calibration point in the radar coordinate system are converted into two-dimensional coordinates (x1, y1) in the BEV coordinate system, and the two-dimensional coordinates of the preset calibration point in the pixel coordinate system are converted into two-dimensional coordinates (x2, y2) in the BEV coordinate system.

[0217] Correspondingly, the two-dimensional coordinates (x1, y1) and (x2, y2) in the BEV coordinate system are the coordinates of the same preset calibration point. Theoretically, the distance between the two-dimensional coordinates (x1, y1) and (x2, y2) is 0. Therefore, for each pair of known preset calibration points in the radar coordinate system polar coordinates and the preset calibration points in the pixel coordinate system two-dimensional coordinates, the distance between the two-dimensional coordinates (x1, y1) and (x2, y2) in the BEV coordinate system is calculated, and this distance is expressed as a loss function as the following formula (11):

[0218]

[0219] n represents the number of pairs of preset calibration points acquired, including their polar coordinates in the radar coordinate system and their two-dimensional coordinates in the pixel coordinate system. (x1i Y 1i ) indicates that the polar coordinates of the i-th pair of preset calibration points in the radar coordinate system are transformed to two-dimensional coordinates in the BEV coordinate system. (x 2i Y 2i This indicates that the two-dimensional coordinates of the i-th pair of preset calibration points in the pixel coordinate system are transformed to two-dimensional coordinates in the BEV coordinate system. Distance represents (x 1i Y 1i ) and (x 2i Y 2i The distance between them.

[0220] Then, the parameter values ​​of calibration parameters H, P0, T0, θ, and Δα that minimize the function value loss of the above loss function are determined, and the target parameter values ​​of calibration parameters H, P0, T0, θ, and Δα are obtained.

[0221] For general linear functions, parameter regression can be performed using methods such as least squares. However, the function model mentioned above is a nonlinear function, and it is difficult to obtain the optimal solution directly through this regression algorithm. Therefore, optimization algorithms can be used to solve the regression problem of multivariate nonlinear functions.

[0222] Solving regression problems of multivariate nonlinear functions by optimizing algorithms also requires designing corresponding loss functions. Ideally, multiple pairs of known pre-defined calibration points can be obtained in both their polar coordinates in the radar coordinate system and their two-dimensional coordinates in the pixel coordinate system. The Euclidean distance between the positions of these pre-defined calibration points in the same coordinate system can then be used as the loss function. However, in practice, when the radar detects, it detects the polar coordinates of the radar detection points within the target object. The target object in the calibration image data acquired by the camera is an image region, and current conditions do not allow for matching the polar coordinates of the radar detection points with the image region occupied by the target object. In this case, the image region occupied by the target object can be converted into the region occupied by the target object in the radar coordinate system. Then, the shortest distance from the radar detection point to the region occupied by the target object in the radar coordinate system can be used as the loss function.

[0223] Therefore, by initializing the calibration parameters H, P0, T0, θ, and Δα in formulas (1) to (10) above, the initial parameter values ​​of the calibration parameters H, P0, T0, θ, and Δα are obtained, which is to obtain the initial transformation relationship between the pixel coordinate system and the radar coordinate system.

[0224] Then, based on the initial transformation relationship, the electronic device transforms the position of the target detection box in the pixel coordinate system to the radar coordinate system, thereby obtaining the position of the target area corresponding to the target detection box in the radar coordinate system. The target area is the area occupied by the target object in the radar coordinate system.

[0225] Accordingly, in some embodiments, in Figure 1 Based on this, see Figure 11 Step S103 may include the following steps:

[0226] S1031: Based on the two-dimensional coordinates of the first specified point in the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, determine the two-dimensional coordinates of the second specified point in the radar coordinate system corresponding to the first specified point in the target detection box.

[0227] S1032: Calculate the horizontal and vertical field of view of the target object based on the camera's Z value, as well as the visual width and visual length of the target detection box.

[0228] The horizontal field of view of the target object is the angle between the line connecting the camera and the left edge of the target object and the line connecting the camera and the right edge of the target object; the vertical field of view of the target object is the angle between the line connecting the camera and the top edge of the target object and the line connecting the camera and the bottom edge of the target object; the camera is a spherical camera.

[0229] S1033: Based on the distance between the target object and the radar, as well as the horizontal and vertical field of view of the target object, calculate the target width and target length of the target area corresponding to the target detection box in the radar coordinate system.

[0230] S1034: Based on the two-dimensional coordinates of the second designated point in the radar coordinate system, as well as the target width and target length, determine the position of the target area in the radar coordinate system.

[0231] The target detection box is the smallest bounding rectangle of the image region occupied by the target object in the calibration image data. In a water transport scenario, the target object is the target vessel. If the camera is not mounted at a high height and the target vessel is relatively high, the intersection of the line connecting the upper point of the target detection box and the camera with the horizontal plane will be far beyond the actual position of the target vessel on the water. However, the points on the bottom edge of the target detection box, after transformation to the BEV coordinate system, are generally closer to the center of the target vessel, i.e., closer to its actual position. Therefore, the first designated point is the midpoint of the bottom edge of the target detection box. Alternatively, the first designated point can also be the midpoint of the bottom edge of the image region occupied by the target object in the calibration image data.

[0232] Based on the initial transformation relationship between the pixel coordinate system and the radar coordinate system, the electronic device transforms the two-dimensional coordinates of the first specified point in the target detection frame in the pixel coordinate system to obtain the two-dimensional coordinates of the second specified point corresponding to the first specified point in the target detection frame in the radar coordinate system.

[0233] Obtain the Z value of the camera at the current moment, and query the Z-FOV table to get the current horizontal and vertical field of view of the camera. Combine the visual width and visual length of the target detection box to obtain the horizontal and vertical field of view of the target object.

[0234] In some embodiments, step S1032 may include the following steps: calculating the horizontal and vertical field of view of the target object based on the camera's Z value, a first preset formula, and the visual width and visual length of the target detection box.

[0235] The first preset formula includes:

[0236] γ u =W u *FOV u (Z) (12)

[0237] γ v =W v *FOV v (Z) (13)

[0238] Y u W represents the horizontal field of view of the target object. u FOV represents the visual width of the object detection bounding box. u (Z) represents the horizontal field of view of the camera corresponding to the camera's Z-value; Z represents the camera's Z-value; Y v W represents the vertical field of view of the target object. v FOV represents the visual length of the object detection bounding box. v (Z) represents the vertical field of view of the camera corresponding to the Z value.

[0239] See Figure 12 , Figure 12 The pentagram indicates the midpoint of the bottom edge of the target detection bounding box, i.e., the first designated point, R. radar R represents the distance between the radar and the target in the calibration azimuth data. radar Let r be the polar coordinate of the target object in the radar coordinate system. Based on the above formulas (12) and (13) and the visual width and visual length of the target object's target detection box, the horizontal and vertical field of view of the target object can be calculated.

[0240] In some embodiments, step S1033 may include the following steps: calculating the initial width and initial length of the initial estimated region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, a second preset formula, and the horizontal and vertical field of view of the target object. Based on the initial width and initial length, determining the target width and target length of the target region corresponding to the target detection box in the radar coordinate system.

[0241] The second preset formula includes:

[0242] w est =γ u *R radar (14)

[0243] h est =γ v *R radar (15)

[0244] w est Indicates the initial width; γ u R represents the horizontal field of view of the target object. radar Indicates the distance between the target and the radar; h est Indicates the initial length; γ v Indicates the vertical field of view of the target object;

[0245] In one implementation, the electronic device can directly calculate the initial width and initial length of the initial estimated region according to the above formulas (14) and (15), and determine the initial estimated region as the target region. Then the initial width is the target width of the target region, and the initial length is the target length of the target region.

[0246] In another implementation, the accuracy of camera and radar calibration can be affected by factors such as the asynchronous nature of the calibration azimuth data acquired by the radar and the calibration image data acquired by the camera, the asynchronous nature of the camera's PTZ value and the acquired calibration image data, and the influence of ship wakes in water transport scenarios. Therefore, electronic equipment can adjust the initial width and initial length based on preset adjustment parameters. For example, the sum of the initial width and the preset adjustment parameters, and the sum of the initial length and the preset adjustment parameters, can be calculated to obtain the target width and target length of the target region corresponding to the target detection box in the radar coordinate system. Alternatively, the initial width and initial length can be increased by a factor of the preset adjustment parameters to obtain the target width and target length of the target region corresponding to the target detection box in the radar coordinate system. In other words, the initial estimated region is expanded outward by a certain proportion to obtain the target region, thereby reducing the impact of the above factors on the accuracy of camera and radar calibration.

[0247] In some embodiments, the first designated point is the midpoint of the bottom edge of the target detection box. Accordingly, step S1034 may include the following steps: taking the second designated point as the center point of the target area and the direction of the line connecting the target object to the camera as the direction of the short side of the target area, in the radar coordinate system, determine a rectangular area with a width equal to the target width and a length equal to the target length, thereby obtaining the position of the target area in the radar coordinate system.

[0248] See Figure 13The electronic device uses the second designated point as the center point of the target area (i.e. Figure 13 (As shown by the pentagram within the rectangular area), and the direction of the line connecting the target object to the camera is taken as the direction of the short side of the target area, and the direction perpendicular to the line connecting the target object to the camera is taken as the direction of the long side of the target area. In the radar coordinate system, a rectangular area with a width equal to the target width and a length equal to the target length is determined (i.e., Figure 13 The range of visual point estimation R in est This area is the target area, thus obtaining the position of the target area in the radar coordinate system.

[0249] Regarding step S104, the electronic device can calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system.

[0250] For example, such as Figure 13 As shown, the target region can be denoted as R. est Preset calibration points (i.e.) Figure 12 The position of the radar point in the radar coordinate system is denoted as (r, yaw). obj If the shortest distance from the preset calibration point to the target area in the radar coordinate system is d[R], then the shortest distance can be denoted as d[R]. est , (r, yaw obj )).

[0251] The target area is the region occupied by the target object in the radar coordinate system. Therefore, theoretically, if the preset calibration point of the target object is located within the target area, the shortest distance from the preset calibration point to the target area in the radar coordinate system is theoretically 0.

[0252] In one implementation, the electronic device determines whether the shortest distance from the preset calibration point to the target area in the radar coordinate system is less than a preset distance threshold. If the shortest distance is less than the preset distance threshold, it indicates that the preset calibration point is located within the target area, and the accuracy of coordinate transformation based on the initial parameter values ​​at this time is relatively high. Therefore, the initial parameter values ​​of the calibration parameters H, P0, T0, θ, and Δα at this time are determined to be target parameter values, thus obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system, which includes the target parameter values ​​of the calibration parameters.

[0253] If the shortest distance is not less than a preset distance threshold, it indicates that the preset calibration point is not located within the target area. The accuracy of coordinate transformation based on the initial parameter values ​​at this time is low. Therefore, the electronic device adjusts the initial parameter values ​​of calibration parameters H, P0, T0, θ, and Δα. Based on the adjusted initial parameter values, it recalculates the shortest distance from the preset calibration point to the target area in the radar coordinate system and determines whether the shortest distance is less than the preset distance threshold. This process continues until the determined shortest distance is less than the preset distance threshold. The electronic device then determines that the initial parameter values ​​at this time are the target parameter values, thus obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system, which includes the target parameter values ​​of the calibration parameters.

[0254] The preset distance threshold is set by technicians according to requirements. To improve the accuracy of the determined target parameter values, the preset distance threshold can be set to a smaller value, such as 0.1.

[0255] In another implementation, step S103 may include the following steps: for each target calibration data group, based on the position of the preset calibration point in the target calibration data group in the radar coordinate system and the position of the target area in the calibration data group in the radar coordinate system, calculate the shortest distance from the preset calibration point in the target calibration data group to the target area in the radar coordinate system, and obtain the shortest distance corresponding to the target calibration data group.

[0256] Accordingly, step S104 may include the following steps: when the shortest distance corresponding to each target calibration data group does not meet the preset convergence condition, adjust the initial parameter value of the calibration parameter to obtain the adjusted initial parameter value; return to execute the step of determining the position of the target area corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, until the shortest distance corresponding to each target calibration data group meets the preset convergence condition, determine the initial parameter value as the target parameter value, and obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter.

[0257] Correspondingly, the method may also include the following steps: when the shortest distance corresponding to each target calibration data group satisfies the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameters is obtained.

[0258] To improve the accuracy of the determined target transformation relationships, the electronic equipment acquires multiple target calibration data sets. For each calibration data set, the electronic equipment calculates the shortest distance from the preset calibration point in the target calibration data set to the target area in the radar coordinate system, based on the position of the preset calibration point in the target calibration data set in the radar coordinate system and the position of the target area in the target calibration data set in the radar coordinate system.

[0259] The preset convergence condition is: the shortest distance corresponding to each target calibration data group is less than a preset distance threshold. Alternatively, the sum of the shortest distances corresponding to each target calibration data group is less than the preset distance threshold.

[0260] If the shortest distances corresponding to each target calibration data set do not meet the preset convergence conditions—for example, if the shortest distance corresponding to a certain target calibration data set is not less than a preset distance threshold—it indicates that the preset calibration point in that target calibration data set is not located within the target area. The accuracy of coordinate transformation based on the initial parameter values ​​at this point is low. The electronic device then adjusts the initial parameter values ​​to obtain the adjusted parameter values, which are used as the new initial parameter values. Then, a new initial transformation relationship is determined based on the new initial parameter values, and the shortest distances corresponding to each target calibration data set are determined based on the new initial transformation relationship. This process continues until the shortest distances corresponding to each target calibration data set meet the preset convergence conditions, indicating that the accuracy of coordinate transformation based on the initial parameter values ​​at this point is high. The initial parameter values ​​at this point are then determined as target parameter values, resulting in the target transformation relationship between the pixel coordinate system and the radar coordinate system, which includes the target parameter values ​​containing the calibration parameters.

[0261] If the shortest distance corresponding to each target calibration data group meets the preset convergence condition, it indicates that the preset calibration point in each target calibration data group is located within the target area. The accuracy of coordinate transformation based on the initial parameter value at this time is relatively high. Therefore, the initial parameter value can be determined as the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter can be obtained.

[0262] In some embodiments, if it is impossible to determine the target parameter value that makes the shortest distance corresponding to each target calibration data group satisfy the preset convergence condition, the electronic device selects a new target calibration data group from the multiple calibration data groups and adjusts the initial parameter value based on the new target calibration data group, and so on, until the target parameter value that makes the shortest distance corresponding to each target calibration data group satisfy the preset convergence condition is determined.

[0263] In some embodiments, in water transport scenarios, after the camera is fixedly installed relative to the ground, the other four calibration parameters (excluding the camera height parameter) of the above five calibration parameters will not change over time. For example, the error T0 of the camera's T value is a fixed error, which remains constant regardless of the camera's PTZ value. However, the camera height parameter is an exception; it represents the real-time height of the camera above the current water surface and changes in real-time with the water level.

[0264] During calibration, the time interval between the acquisition of multiple calibration images is small, resulting in minimal differences in water level height across different images. Therefore, the water level height can be considered a fixed value in the calibration image data, meaning the camera's height parameter H is constant. However, due to various factors, such as fewer passing ships, the acquisition time interval between calibration images can become larger, leading to significant variations in water level height across different images. For instance, during high and low tides, if the acquisition interval for calibration images is 2 hours, the water level height in different images could differ by 3 meters, affecting the calibration accuracy of the camera and radar. Simply assuming a fixed water level height across multiple calibration images reduces the accuracy of the target transformation relationship between the calculated pixel coordinate system and the radar coordinate system. Therefore, it is necessary to eliminate the impact of water level height variations in the calibration image data during the calibration process.

[0265] Correspondingly, for scenarios where the target is water transport, in Figure 2 Based on this, see Figure 14 Before step S103, the method may further include the following steps:

[0266] S106: Obtain the water level of the water transport scene at each pre-recorded historical moment.

[0267] S107: Determine the minimum water level among the pre-recorded water level heights for each historical moment in the water transport scenario to obtain the water level height of the calibration reference plane.

[0268] S108: For each historical moment, calculate the difference between the water surface height of the water transport scenario at that historical moment and the water surface height of the calibration reference plane, and obtain the water surface height difference corresponding to that historical moment.

[0269] S109: Calculate the difference between the initial parameter value of the camera's reference height parameter and the water surface height difference corresponding to the historical moment, and use it as the initial parameter value of the camera's height parameter at the historical moment to obtain the initial transformation relationship between the pixel coordinate system and the radar coordinate system at the historical moment.

[0270] The reference height parameter represents the height of the camera from the calibration reference plane.

[0271] Accordingly, step S103 may include the following steps:

[0272] S1035: For each target calibration data group, based on the position of the target detection box in the pixel coordinate system in the target calibration data group and the initial transformation relationship of the historical time to which the target calibration data group belongs, determine the position of the target area in the radar coordinate system in the target calibration data group.

[0273] After acquiring calibration image data of the water transport scene collected by the camera at various historical moments, and calibration orientation data of the water transport scene collected by the radar at various historical moments, the electronic equipment obtains and determines the water surface height of the water transport scene at each historical moment as pre-recorded.

[0274] In one implementation, when collecting calibration image data and calibration orientation data of water transport scenarios at various historical moments, the water surface height of the water transport scenarios at various historical moments is measured using water surface monitoring equipment.

[0275] In another implementation, using water surface monitoring equipment is costly. To reduce costs, the water surface height from the calibration image data collected at each historical time can be obtained from the tide table of the region where the water transport scenario is located, based on the historical time of each calibration image data collection. For example, Figure 15 The table shown shows the tides in Huangqi area at different times on April 23, 2023. The tide table records the tide heights in Huangqi area at different times of the day.

[0276] The electronic device uses the tidal theoretical depth datum (i.e., the lowest tide level in the region) as the calibration reference plane. In other words, it uses the lowest water level at each historical moment as the calibration reference plane. Therefore, the minimum water level height for each historical moment in the water transport scenario is the water level height on the calibration reference plane. In this way, the calibration reference plane is consistent with the tidal table's reference plane. The height of the camera from the calibration reference plane is the camera's reference height parameter, denoted as H0.

[0277] For each historical moment, the electronic device calculates the difference between the water surface height of the water transport scene at that historical moment and the water surface height of the calibration reference plane, thus obtaining the water surface height difference corresponding to that historical moment. The initial parameter value of the camera's reference height parameter is then calculated, along with the difference between the initial parameter value and the water surface height difference corresponding to that historical moment, to obtain the initial parameter value of the camera's height parameter for that historical moment. This also yields the initial transformation relationship between the pixel coordinate system and the radar coordinate system.

[0278] For example, if the reference height parameter of the camera is H0, and the water surface height difference corresponding to each historical moment is h1, h2, ..., hn, then the height parameters of the camera at each historical moment are calculated as H0-h1, H0-h2, ..., H0-hn, where n represents the number of historical moments.

[0279] Furthermore, the electronic device uses the initial parameter values ​​of the camera's height parameters at each historical moment (i.e., H0-h1, H0-h2, ..., H0-hn) as the parameter H in the above formulas (1) to (10), and obtains the initial transformation relationship between the pixel coordinate system and the radar coordinate system at each historical moment.

[0280] To improve the accuracy of the determined target transformation relationships, for each target calibration data set, the electronic device determines the position of the target region in the radar coordinate system based on the initial transformation relationship of the historical time to which the target calibration data set belongs, and the position of the target detection box in the pixel coordinate system within the target calibration data set. Subsequently, the target transformation relationship is determined based on the shortest distance between the preset calibration points in each target calibration data set and the target region in the radar coordinate system.

[0281] In practical applications, the personnel collecting calibration materials (i.e., calibration image data) need to consult tide tables to obtain the water level height of the water transport scene at various historical moments, and then write the water level height of the historical moments in the collected calibration image data into the video file according to a standard format. Subsequently, electronic equipment can automatically read the water level height of the water transport scene at each historical moment and calculate calibration parameters based on the water level height of the water transport scene at each historical moment. Based on the above processing, camera and radar calibration can be completed even when the water level height in the calibration materials is different. The requirements for calibration materials are low, making it applicable to scenarios with few ships, and improving the accuracy of camera and radar calibration.

[0282] In some embodiments, during the calibration of the camera and radar, for each target calibration data set, when the electronic device determines the position of the target area in the radar coordinate system based on the initial transformation relationship of the historical time to which the target calibration data set belongs, the initial transformation relationship contains the P-value, T-value, and Z-value of the camera at that historical time. For example, acquiring... Figure 3 When calibrating the image data and azimuth data, the camera's P value is 352.12°; the camera's T value is 73.37°; and the camera's Z value is 1.00.

[0283] Furthermore, if the direction in which the camera's T-value is 0 when acquiring calibration image data differs from the direction in which the camera's T-value is 0 when determining the transformation relationship; or if the positive direction of camera rotation according to the T-value defined when acquiring calibration image data differs from the positive direction of camera rotation according to the T-value defined when determining the initial transformation relationship, then the camera's T-value needs to be transformed to obtain a transformed T-value. The transformed T-value is the camera's T-value when the direction in which the camera's T-value is 0 when acquiring calibration image data is the same as the direction in which the camera's T-value is 0 when determining the transformation relationship, and the positive direction of camera rotation according to the T-value defined when acquiring calibration image data is the same as the positive direction of camera rotation according to the T-value defined when determining the transformation relationship. At each subsequent time step, the transformation relationship between the pixel coordinate system and the radar coordinate system at that time contains the camera's P-value, T-value, and Z-value, which can improve the accuracy of camera and radar calibration.

[0284] and Figure 1 For the corresponding method implementation examples, see [link to relevant documentation]. Figure 16 , Figure 16 This application provides a structural diagram of a camera and radar calibration device, the device comprising:

[0285] The data acquisition module 1601 is used to acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera.

[0286] The first position determination module 1602 is used to determine, based on the calibration azimuth data and the calibration image data, the position of the preset calibration point in the target object in the radar coordinate system of the radar, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera.

[0287] The second position determination module 1603 is used to determine the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system; wherein, the calibration parameter in the initial transformation relationship is an initial parameter value;

[0288] The shortest distance determination module 1604 is used to calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system.

[0289] The first transformation relationship determination module 1605 is used to adjust the initial parameter value of the calibration parameter based on the shortest distance, determine the target parameter value of the calibration parameter, and obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter.

[0290] Optionally, the second position determination module 1603 is specifically used to determine the two-dimensional coordinates of a second specified point corresponding to the first specified point in the target detection frame in the radar coordinate system based on the two-dimensional coordinates of the first specified point in the target detection frame in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system; and to calculate the horizontal and vertical field of view of the target object based on the Z value of the camera and the visual width and visual height of the target detection frame; wherein, the horizontal field of view of the target object is: the line connecting the camera and the left edge of the target object, and the line connecting the camera and the target object. The angle between the lines connecting the right edges of the target object; the vertical field of view of the target object is the angle between the line connecting the upper edge of the camera and the lower edge of the target object; the camera is a spherical camera; based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, the target width and target length of the target area corresponding to the target detection box in the radar coordinate system are calculated; based on the two-dimensional coordinates of the second designated point in the radar coordinate system, and the target width and target length, the position of the target area in the radar coordinate system is determined.

[0291] Optionally, the second position determination module 1603 is specifically used to calculate the horizontal and vertical field of view of the target object based on the Z-value of the camera, a first preset formula, and the visual width and visual length of the target detection box; wherein, the first preset formula includes:

[0292] γ u =W u *FOV u (Z)

[0293] γ v =W v *FOV v (Z)

[0294] γ u W represents the horizontal field of view of the target object. u The field of view (FOV) represents the visual width of the target detection box. u (Z) represents the horizontal field of view of the camera corresponding to the Z-value; Z represents the Z-value of the camera; γ vW represents the vertical field of view of the target object; v The field of view (FOV) represents the visual length of the target detection box. v (Z) represents the vertical field of view of the camera corresponding to the Z value of the camera.

[0295] Optionally, the second position determination module 1603 is specifically used to calculate the initial width and initial length of the initial estimated region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, a second preset formula, and the horizontal and vertical field of view angles of the target object; wherein, the second preset formula includes:

[0296] w est =γ u *R radar

[0297] h est =γ v *R radar

[0298] w est Indicates the initial width; γ u R represents the horizontal field of view of the target object. radar h represents the distance between the target object and the radar. est Indicates the initial length; γ v This indicates the vertical field of view of the target object;

[0299] Based on the initial width and the initial length, the target width and target length of the target detection box in the radar coordinate system are determined.

[0300] Optionally, the second position determination module 1603 is specifically used to adjust the initial width and the initial length according to preset adjustment parameters to obtain the target width and target length of the target area corresponding to the target detection box in the radar coordinate system.

[0301] Optionally, the first designated point is the midpoint of the bottom edge of the target detection box;

[0302] The second position determination module 1603 is specifically used to determine a rectangular area in the radar coordinate system with a width equal to the target width and a length equal to the target length, based on the second designated point as the center point of the target area and the direction of the line connecting the target object to the camera as the direction of the short side of the target area, thereby obtaining the position of the target area in the radar coordinate system.

[0303] Optionally, the data acquisition module 1601 is used to acquire various calibration azimuth data of the target scene collected by the radar at various historical moments, and multiple calibration image data of the target scene collected by the camera at various historical moments.

[0304] The first position determination module 1602 is specifically configured to, for each historical moment, obtain the position of a preset calibration point of a first object in the calibration azimuth data of that historical moment in the radar coordinate system; perform target detection on each calibration image data of that historical moment to obtain the position of the detection box of a second object in the calibration image data of that historical moment in the pixel coordinate system of the camera; combine the first object and the second object of that historical moment to obtain multiple calibration data sets; wherein, a calibration data set includes: the position of a preset calibration point of a first object in the radar coordinate system of the radar, and the position of the detection box of a second object in the pixel coordinate system of the camera; determine a target calibration data set containing the target object from the multiple calibration data sets; obtain the position of the preset calibration point of the target object in the target calibration data set in the radar coordinate system, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera.

[0305] Optionally, the first position determination module 1602 is specifically used to determine that the calibration data group is a target calibration data group containing the target object when the attribute information of the first object and the second object in the calibration data group meets the preset filtering conditions for each calibration data group; wherein, the attribute information of an object includes at least one of the following: the distance between the object and the radar, the object's speed, size, and direction of movement.

[0306] Optionally, the shortest distance determination module 1604 is specifically used to calculate, for each target calibration data group, the shortest distance from the preset calibration point in the target calibration data group to the target area in the radar coordinate system based on the position of the preset calibration point in the target calibration data group in the radar coordinate system and the position of the target area in the calibration data group in the radar coordinate system, to obtain the shortest distance corresponding to the target calibration data group;

[0307] The first transformation relationship determination module 1605 is specifically used to adjust the initial parameter value of the calibration parameter to obtain the adjusted initial parameter value when the shortest distance corresponding to each target calibration data group does not meet the preset convergence condition; return to execute the step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, until the shortest distance corresponding to each target calibration data group meets the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter is obtained;

[0308] The device further includes:

[0309] The second transformation relationship determination module is used to determine the initial parameter value as the target parameter value when the shortest distance corresponding to each target calibration data group satisfies the preset convergence condition, and to obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter.

[0310] Optionally, the target scenario is a water transport scenario;

[0311] The device further includes:

[0312] The water surface height acquisition module is used to acquire the water surface height of the water transport scene at each historical moment before the second position determination module 1603 executes the determination of the position of the target area corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the initial transformation relationship between the pixel coordinate system and the radar coordinate system established in advance.

[0313] The water surface height determination module is used to determine the minimum value among the water surface heights of the water transport scenario at each pre-recorded historical moment, and to obtain the water surface height of the calibration reference plane;

[0314] The water surface height difference determination module is used to calculate the difference between the water surface height of the water transport scenario and the water surface height of the calibration reference plane for each historical moment, so as to obtain the water surface height difference corresponding to that historical moment.

[0315] The third transformation relationship determination module is used to calculate the difference between the initial parameter value of the camera's reference height parameter and the water surface height difference corresponding to the historical moment, and use it as the initial parameter value of the camera's height parameter at the historical moment to obtain the initial transformation relationship between the pixel coordinate system and the radar coordinate system at the historical moment; wherein, the reference height parameter represents the height of the camera from the calibration reference plane;

[0316] The second position determination module 1603 is specifically used to determine the position of the target region in the radar coordinate system for each target calibration data group based on the position of the target detection box in the pixel coordinate system and the initial transformation relationship of the historical time to which the target calibration data group belongs.

[0317] Optionally, the camera is a spherical camera; the radar is a millimeter-wave radar;

[0318] The calibration parameters include: the camera's height parameter; the angle between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the camera's P value is 0; the P value when the camera points in the direction the camera's base is tilted; the camera's mounting pitch angle; wherein the mounting pitch angle is the angle between the camera's base and the horizontal plane; and the error amount of the camera's T value.

[0319] Optionally, the transformation relationship between the pixel coordinate system and the radar coordinate system includes:

[0320] x = r * cos(yaw) obj +Δα)

[0321] y = r*sin(yaw) obj +Δα)

[0322]

[0323]

[0324]

[0325]

[0326] T1 = T + T0

[0327] yaw0=arctan2{sin(P-P0), cos[(P-P0)*|cosθ|]}

[0328] d v =v-0.5

[0329] d u =u-0.5

[0330] (x, y) represents the two-dimensional coordinates of the target feature point in the BEV coordinate system; r represents the distance between the target feature point and the origin of the radar coordinate system; (u, v) represents the normalized two-dimensional coordinates of the target feature point in the pixel coordinate system; pitch objyaw represents the angle between the line connecting the optical center of the camera and the object and the horizontal plane. obj The azimuth angle of the target feature point in the radar coordinate system is represented by Δα; Δα represents the angle difference between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0; H represents the height parameter of the camera; T1 represents the actual T value of the camera. This indicates the tilt angle between the camera's base and the horizontal direction; d v d represents the offset of the normalized ordinate of the target feature point in the pixel coordinate system relative to the image center; u The value of F0V represents the offset of the normalized x-coordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the current Z value of the camera; v (Z) represents the vertical field of view (FOV) of the camera corresponding to its current Z value; u (Z) represents the horizontal and vertical field of view of the camera corresponding to the current Z value; yaw0 represents the angle through which the camera rotates on the horizontal plane when the camera rotates from P value P0 to P value P; T represents the current T value of the camera; T0 represents the error amount of the camera's T value; P represents the current P value of the camera; P0 represents the P value when the camera is pointing in the direction of tilt of the camera's base; θ represents the mounting pitch angle of the camera.

[0331] The BEV coordinate system takes the projection point of the radar in the horizontal plane as its origin, and the X-axis and Y-axis of the BEV coordinate system are located in the horizontal plane; the positive direction of the X-axis of the BEV coordinate system is the same as the positive direction of the X-axis of the radar coordinate system.

[0332] Based on the camera and radar calibration device provided in this application embodiment, the target transformation relationship between the pixel coordinate system and the radar coordinate system can be determined without manually measuring the calibration parameters of the target scene. This reduces the cost and improves the efficiency of camera and radar calibration. Furthermore, the preset calibration point in the target object is the calibration point within the target object, and the target area is the area where the target object is located. Correspondingly, the preset calibration point is located within the target area. Adjusting the calibration parameter values ​​based on the shortest distance from the preset calibration point to the target area in the radar coordinate system improves the accuracy of the determined target transformation relationship. Subsequently, precise conversion between positional data and image data can be performed based on the target transformation relationship, improving the accuracy of ship monitoring in water transport scenarios.

[0333] This application also provides an electronic device, such as... Figure 17 As shown, it includes:

[0334] Memory 1701 is used to store computer programs;

[0335] When processor 1702 executes the program stored in memory 1701, it performs the following steps:

[0336] Acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera;

[0337] Based on the calibration azimuth data and the calibration image data, the position of the preset calibration point in the target object in the radar coordinate system of the radar is determined, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera is determined.

[0338] Based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the position of the target region corresponding to the target detection box in the radar coordinate system is determined; wherein, the calibration parameter in the initial transformation relationship is an initial parameter value;

[0339] Based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system, calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system;

[0340] The initial parameter values ​​of the calibration parameters are adjusted based on the shortest distance to determine the target parameter values ​​of the calibration parameters, thereby obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system that includes the target parameter values ​​of the calibration parameters.

[0341] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1702, the communication interface, and the memory 1701 communicating with each other via the communication bus.

[0342] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0343] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0344] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0345] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0346] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described camera and radar calibration methods.

[0347] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the camera and radar calibration methods described above.

[0348] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0349] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0350] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0351] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A camera and radar calibration method, characterized in that, The method includes: Acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera; Based on the calibration azimuth data and the calibration image data, the position of the preset calibration point in the target object in the radar coordinate system of the radar is determined, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera is determined; wherein, the type of the target object is the type of object to be monitored in the target scene; Based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the position of the target region corresponding to the target detection box in the radar coordinate system is determined; wherein, the calibration parameter in the initial transformation relationship is the initial parameter value; the target region is the area occupied by the target object in the radar coordinate system; Based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system, calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system; The initial parameter values ​​of the calibration parameters are adjusted based on the shortest distance to determine the target parameter values ​​of the calibration parameters, thereby obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system that includes the target parameter values ​​of the calibration parameters.

2. The method according to claim 1, characterized in that, The step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system includes: Based on the two-dimensional coordinates of the first specified point in the target detection frame in the pixel coordinate system, and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the two-dimensional coordinates of the second specified point corresponding to the first specified point in the target detection frame in the radar coordinate system are determined. Based on the Z-value of the camera, and the visual width and visual length of the target detection box, the horizontal and vertical field of view of the target object are calculated; wherein, the horizontal field of view of the target object is the angle between the line connecting the camera and the left edge of the target object and the line connecting the camera and the right edge of the target object; the vertical field of view of the target object is the angle between the line connecting the camera and the top edge of the target object and the line connecting the camera and the bottom edge of the target object; the camera is a spherical camera; Based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, the target width and target length of the target area corresponding to the target detection box in the radar coordinate system are calculated; Based on the two-dimensional coordinates of the second designated point in the radar coordinate system, as well as the target width and the target length, the position of the target region in the radar coordinate system is determined.

3. The method according to claim 2, characterized in that, The calculation of the horizontal and vertical field of view of the target object based on the Z-value of the camera and the visual width and visual length of the target detection box includes: Based on the camera's Z-value, a first preset formula, and the visual width and visual length of the target detection box, the horizontal and vertical field of view of the target object are calculated; wherein, the first preset formula includes: ； ； This indicates the horizontal field of view of the target object; This represents the visual width of the target detection box; The Z value represents the horizontal field of view of the camera; Z represents the Z value of the camera. This indicates the vertical field of view of the target object; This represents the visual length of the target detection box; The Z-value of the camera represents the vertical field of view of the camera.

4. The method according to claim 2, characterized in that, The step of calculating the target width and target length of the target region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, includes: Based on the distance between the target object and the radar, a second preset formula, and the horizontal and vertical field of view of the target object, the initial width and initial length of the initial estimated region corresponding to the target detection box in the radar coordinate system are calculated; wherein, the second preset formula includes: ； ； This represents the initial width; This indicates the horizontal field of view of the target object; This indicates the distance between the target object and the radar; This represents the initial length; This indicates the vertical field of view of the target object; Based on the initial width and the initial length, the target width and target length of the target detection box in the radar coordinate system are determined.

5. The method according to claim 4, characterized in that, The step of determining the target width and target length of the target region corresponding to the target detection box in the radar coordinate system based on the initial width and the initial length includes: According to preset adjustment parameters, the initial width and the initial length are adjusted to obtain the target width and target length of the target area corresponding to the target detection box in the radar coordinate system.

6. The method according to claim 2, characterized in that, The first designated point is the midpoint of the bottom edge of the target detection box; Determining the position of the target region in the radar coordinate system based on the two-dimensional coordinates of the second designated point in the radar coordinate system, as well as the target width and the target length, includes: Using the second designated point as the center point of the target area, and the direction of the line connecting the target object to the camera as the direction of the short side of the target area, a rectangular area with a width equal to the target width and a length equal to the target length is determined in the radar coordinate system, thus obtaining the position of the target area in the radar coordinate system.

7. The method according to claim 1, characterized in that, The acquisition of the calibration azimuth data of the target scene collected by the radar and the calibration image data of the target scene collected by the camera includes: Acquire the calibration azimuth data of the target scene collected by the radar at various historical moments, as well as the multiple calibration image data of the target scene collected by the camera at various historical moments; The step of determining the position of a preset calibration point in the target object within the radar coordinate system of the radar, and the position of the target detection bounding box in the calibration image data within the pixel coordinate system of the camera, based on the calibration azimuth data and the calibration image data, includes: For each historical moment, the position of the preset calibration point of the first object in the calibration azimuth data of that historical moment in the radar coordinate system is obtained from the calibration azimuth data of that historical moment. Target detection is performed on each calibration image data of the historical moment to obtain the position of the detection box of the second object in the calibration image data of the historical moment in the pixel coordinate system of the camera; The first and second objects at this historical moment are combined to obtain multiple calibration data sets; wherein, a calibration data set includes: the position of a preset calibration point of the first object in the radar coordinate system of the radar, and the position of the detection box of the second object in the pixel coordinate system of the camera; From the plurality of calibration data sets, determine the target calibration data set that contains the target object; The position of the preset calibration point of the target object in the target calibration data group in the radar coordinate system is obtained, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera is obtained.

8. The method according to claim 7, characterized in that, The step of determining the target calibration data group containing the target object from the plurality of calibration data groups includes: For each calibration data set, if the attribute information of the first object and the second object in the calibration data set meets the preset filtering conditions, the calibration data set is determined to be a target calibration data set containing the target object; wherein, the attribute information of an object includes at least one of the following: the distance between the object and the radar, the object's speed, size, and direction of movement.

9. The method according to claim 7, characterized in that, The step of calculating the shortest distance from the preset calibration point to the target area in the radar coordinate system based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system includes: For each target calibration data set, based on the position of the preset calibration point in the target calibration data set in the radar coordinate system and the position of the target area in the target calibration data set in the radar coordinate system, the shortest distance from the preset calibration point in the target calibration data set to the target area in the radar coordinate system is calculated to obtain the shortest distance corresponding to the target calibration data set. The step of adjusting the initial parameter values ​​of the calibration parameters based on the shortest distance, determining the target parameter values ​​of the calibration parameters, and obtaining the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter values ​​of the calibration parameters includes: When the shortest distance corresponding to each target calibration data group does not meet the preset convergence condition, the initial parameter value of the calibration parameter is adjusted to obtain the adjusted initial parameter value; the process returns to the step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, until the shortest distance corresponding to each target calibration data group meets the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter is obtained; The method further includes: When the shortest distance corresponding to each target calibration data group satisfies the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter is obtained.

10. The method according to claim 7, characterized in that, The target scenario is a water transport scenario; Before determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, the method further includes: Obtain the water level of the water transport scene at each pre-recorded historical moment; The minimum water level height among the water transport scenarios recorded in each historical moment is determined to obtain the water level height of the calibration reference plane; For each historical moment, the difference between the water surface height of the water transport scenario at that historical moment and the water surface height of the calibration reference plane is calculated to obtain the water surface height difference corresponding to that historical moment; The difference between the initial value of the camera's reference height parameter and the water surface height difference corresponding to the historical moment is calculated and used as the initial value of the camera's height parameter at the historical moment, thereby obtaining the initial transformation relationship between the pixel coordinate system and the radar coordinate system at the historical moment; wherein, the reference height parameter represents the height of the camera from the calibration reference plane; The step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system includes: For each target calibration data set, based on the position of the target detection box in the target calibration data set in the pixel coordinate system and the initial transformation relationship of the historical time to which the target calibration data set belongs, the position of the target region in the target calibration data set in the radar coordinate system is determined.

11. The method according to any one of claims 1 to 10, characterized in that, The camera is a spherical camera; the radar is a millimeter-wave radar; The calibration parameters include: The camera's height parameter; The angle between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0. The value of P when the camera is pointed in the direction in which the camera base is tilted; The mounting pitch angle of the camera; wherein, the mounting pitch angle is the angle between the base of the camera and the horizontal plane; The error amount of the camera's T-value.

12. The method according to any one of claims 1 to 10, characterized in that, The transformation relationship between the pixel coordinate system and the radar coordinate system includes: ； ； ； ； ； ； ； yaw0=arctan2{sin(P-P0),cos[(P-P0)*|cosθ|]}; ； ； (x, y) represents the two-dimensional coordinates of the target feature point in the BEV coordinate system; r represents the distance between the target feature point and the origin of the radar coordinate system; (u, v) represents the normalized two-dimensional coordinates of the target feature point in the pixel coordinate system. This indicates the angle between the line connecting the optical center of the camera and the object and the horizontal plane. This indicates the azimuth angle of the target feature point in the radar coordinate system; The angle difference between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0; H represents the height parameter of the camera; T1 represents the actual T value of the camera; This indicates the tilt angle between the base of the camera and the horizontal direction; This represents the offset of the normalized ordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the offset of the normalized x-coordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the current Z value of the camera. This represents the vertical field of view of the camera corresponding to its current Z value; This indicates the horizontal and vertical field of view of the camera corresponding to the current Z value. This represents the angle through which the camera rotates on the horizontal plane as the camera rotates from a value of P0 to a value of P. T represents the current T value of the camera; T0 represents the error amount of the camera's T value; P represents the current P value of the camera; P0 represents the P value when the camera is pointing in the direction the camera's base is tilted. This indicates the mounting pitch angle of the camera; The BEV coordinate system takes the projection point of the radar in the horizontal plane as its origin, and the X-axis and Y-axis of the BEV coordinate system are located in the horizontal plane; the positive direction of the X-axis of the BEV coordinate system is the same as the positive direction of the X-axis of the radar coordinate system.

13. A camera and radar calibration device, characterized in that, The device includes: The data acquisition module is used to acquire the calibration azimuth data of the target scene collected by the radar, and the calibration image data of the target scene collected by the camera. The first position determination module is used to determine, based on the calibration azimuth data and the calibration image data, the position of a preset calibration point in the target object in the radar coordinate system of the radar, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera; wherein, the type of the target object is the type of object to be monitored in the target scene; The second position determination module is used to determine the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system; wherein, the calibration parameter in the initial transformation relationship is the initial parameter value; and the target region is the area occupied by the target object in the radar coordinate system. The shortest distance determination module is used to calculate the shortest distance from the preset calibration point to the target area in the radar coordinate system based on the position of the preset calibration point in the radar coordinate system and the position of the target area corresponding to the target detection box in the radar coordinate system. The first transformation relationship determination module is used to adjust the initial parameter value of the calibration parameter based on the shortest distance, determine the target parameter value of the calibration parameter, and obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system that includes the target parameter value of the calibration parameter.

14. The apparatus according to claim 13, characterized in that, The second position determination module is specifically used to determine the two-dimensional coordinates of the second designated point in the radar coordinate system corresponding to the first designated point in the target detection frame based on the two-dimensional coordinates of the first designated point in the target detection frame in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system. Based on the Z-value of the camera, and the visual width and visual length of the target detection box, the horizontal and vertical field of view of the target object are calculated; wherein, the horizontal field of view of the target object is the angle between the line connecting the camera and the left edge of the target object and the line connecting the camera and the right edge of the target object; the vertical field of view of the target object is the angle between the line connecting the camera and the top edge of the target object and the line connecting the camera and the bottom edge of the target object; the camera is a spherical camera; Based on the distance between the target object and the radar, and the horizontal and vertical field of view of the target object, the target width and target length of the target area corresponding to the target detection box in the radar coordinate system are calculated; Based on the two-dimensional coordinates of the second designated point in the radar coordinate system, as well as the target width and the target length, the position of the target region in the radar coordinate system is determined; The second position determination module is specifically used to calculate the horizontal and vertical field of view of the target object based on the Z-value of the camera, a first preset formula, and the visual width and visual length of the target detection box; wherein, the first preset formula includes: ； ； This indicates the horizontal field of view of the target object; This represents the visual width of the target detection box; The Z value represents the horizontal field of view of the camera; Z represents the Z value of the camera. This indicates the vertical field of view of the target object; This represents the visual length of the target detection box; The Z-value of the camera represents the vertical field of view of the camera. The second position determination module is specifically used to calculate the initial width and initial length of the initial estimated region corresponding to the target detection box in the radar coordinate system based on the distance between the target object and the radar, a second preset formula, and the horizontal and vertical field of view angles of the target object; wherein, the second preset formula includes: ； ； This represents the initial width; This indicates the horizontal field of view of the target object; This indicates the distance between the target object and the radar; This represents the initial length; This indicates the vertical field of view of the target object; Based on the initial width and the initial length, the target width and target length of the target region corresponding to the target detection box in the radar coordinate system are determined; The second position determination module is specifically used to adjust the initial width and the initial length according to preset adjustment parameters to obtain the target width and target length of the target area corresponding to the target detection box in the radar coordinate system; The first designated point is the midpoint of the bottom edge of the target detection box; The second position determination module is specifically used to determine a rectangular area in the radar coordinate system with a width equal to the target width and a length equal to the target length, based on the second designated point as the center point of the target area and the direction of the line connecting the target object to the camera as the direction of the short side of the target area, thereby obtaining the position of the target area in the radar coordinate system. The data acquisition module is used to acquire the calibration azimuth data of the target scene collected by the radar at various historical moments, as well as the multiple calibration image data of the target scene collected by the camera at various historical moments. The first position determination module is specifically used to, for each historical moment, obtain the position of the preset calibration point of the first object in the radar coordinate system of the radar from the calibration azimuth data of that historical moment. Target detection is performed on each calibration image data of the historical moment to obtain the position of the detection box of the second object in the calibration image data of the historical moment in the pixel coordinate system of the camera; The first and second objects at this historical moment are combined to obtain multiple calibration data sets; wherein, a calibration data set includes: the position of a preset calibration point of the first object in the radar coordinate system of the radar, and the position of the detection box of the second object in the pixel coordinate system of the camera; From the plurality of calibration data sets, determine the target calibration data set that contains the target object; The position of the preset calibration point of the target object in the target calibration data group in the radar coordinate system is obtained, and the position of the target detection box of the target object in the calibration image data in the pixel coordinate system of the camera is obtained. The first position determination module is specifically used to determine, for each calibration data group, that the calibration data group is a target calibration data group containing the target object when the attribute information of the first object and the second object in the calibration data group meets the preset filtering conditions; wherein, the attribute information of an object includes at least one of the following: the distance between the object and the radar, the object's speed, size, and direction of movement; The shortest distance determination module is specifically used to calculate the shortest distance from the preset calibration point in the target calibration data group to the target area in the radar coordinate system for each target calibration data group, based on the position of the preset calibration point in the target calibration data group in the radar coordinate system and the position of the target area in the target calibration data group in the radar coordinate system, so as to obtain the shortest distance corresponding to the target calibration data group. The first transformation relationship determination module is specifically used to adjust the initial parameter value of the calibration parameter to obtain the adjusted initial parameter value when the shortest distance corresponding to each target calibration data group does not meet the preset convergence condition; return to execute the step of determining the position of the target region corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the pre-established initial transformation relationship between the pixel coordinate system and the radar coordinate system, until the shortest distance corresponding to each target calibration data group meets the preset convergence condition, the initial parameter value is determined to be the target parameter value, and the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter is obtained; The device further includes: The second transformation relationship determination module is used to determine the initial parameter value as the target parameter value when the shortest distance corresponding to each target calibration data group satisfies the preset convergence condition, and to obtain the target transformation relationship between the pixel coordinate system and the radar coordinate system containing the target parameter value of the calibration parameter; The target scenario is a water transport scenario; The device further includes: The water surface height acquisition module is used to acquire the water surface height of the water transport scene at each historical moment before the second position determination module executes the determination of the position of the target area corresponding to the target detection box in the radar coordinate system based on the position of the target detection box in the pixel coordinate system and the initial transformation relationship between the pixel coordinate system and the radar coordinate system established in advance. The water surface height determination module is used to determine the minimum value among the water surface heights of the water transport scenario at each pre-recorded historical moment, and to obtain the water surface height of the calibration reference plane; The water surface height difference determination module is used to calculate the difference between the water surface height of the water transport scenario and the water surface height of the calibration reference plane for each historical moment, so as to obtain the water surface height difference corresponding to that historical moment. The third transformation relationship determination module is used to calculate the difference between the initial parameter value of the camera's reference height parameter and the water surface height difference corresponding to the historical moment, and use it as the initial parameter value of the camera's height parameter at the historical moment to obtain the initial transformation relationship between the pixel coordinate system and the radar coordinate system at the historical moment; wherein, the reference height parameter represents the height of the camera from the calibration reference plane; The second position determination module is specifically used to determine the position of the target region in the radar coordinate system for each target calibration data group based on the position of the target detection box in the pixel coordinate system and the initial transformation relationship of the historical time to which the target calibration data group belongs; The camera is a spherical camera; the radar is a millimeter-wave radar; The calibration parameters include: The camera's height parameter; The angle between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0. The value of P when the camera is pointed in the direction in which the camera base is tilted; The mounting pitch angle of the camera; wherein, the mounting pitch angle is the angle between the base of the camera and the horizontal plane; The error amount of the camera's T-value; The transformation relationship between the pixel coordinate system and the radar coordinate system includes: ; ； ； ； ； ； ； yaw0=arctan2{sin(P-P0),cos[(P-P0)*|cosθ|]}; ； ； (x, y) represents the two-dimensional coordinates of the target feature point in the BEV coordinate system; r represents the distance between the target feature point and the origin of the radar coordinate system; (u, v) represents the normalized two-dimensional coordinates of the target feature point in the pixel coordinate system. This indicates the angle between the line connecting the optical center of the camera and the object and the horizontal plane. This indicates the azimuth angle of the target feature point in the radar coordinate system; The angle difference between the positive direction of the camera and the positive direction of the X-axis of the radar coordinate system when the P value of the camera is 0; H represents the height parameter of the camera; T1 represents the actual T value of the camera; This indicates the tilt angle between the base of the camera and the horizontal direction; This represents the offset of the normalized ordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the offset of the normalized x-coordinate of the target feature point in the pixel coordinate system relative to the image center; Z represents the current Z value of the camera. This represents the vertical field of view of the camera corresponding to its current Z value; This indicates the horizontal and vertical field of view of the camera corresponding to the current Z value. This represents the angle through which the camera rotates on the horizontal plane as the camera rotates from a value of P0 to a value of P. T represents the current T value of the camera; T0 represents the error amount of the camera's T value; P represents the current P value of the camera; P0 represents the P value when the camera is pointing in the direction the camera's base is tilted. This indicates the mounting pitch angle of the camera; The BEV coordinate system takes the projection point of the radar in the horizontal plane as its origin, and the X-axis and Y-axis of the BEV coordinate system are located in the horizontal plane; the positive direction of the X-axis of the BEV coordinate system is the same as the positive direction of the X-axis of the radar coordinate system.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-12.

Images