Joint calibration method, device thereof, electronic device and unmanned aerial vehicle

Abstract

Embodiments of the present application relate to a joint calibration method, a device thereof, an electronic device and a UAV. The joint calibration method comprises: obtaining pose information of a detection radar; wherein the pose information comprises a height above ground of the detection radar and a pitch angle of the detection radar; in a preset set of calibration parameters, obtaining a target calibration parameter matched with the pose information; wherein the set of calibration parameters comprises a plurality of groups of calibration parameters; each group of calibration parameters is matched with a pose information interval; the pose information interval comprises a height interval and a pitch angle interval; and based on the target calibration parameter, a spatial conversion relationship between the detection radar and an image acquisition device is determined. The method can correct and update the calibration parameter according to the attitude change of the detection radar, ensure the accuracy of the obtained spatial conversion relationship, and effectively improve the data fusion effect of the detection radar and the image acquisition device.

Description

[Technical Field]

[0001] This invention relates to the field of data fusion technology, and in particular to a joint calibration method, apparatus, electronic device, and unmanned aerial vehicle. [Background Technology]

[0002] With the continuous development of electronic information technology, more and more automated equipment, such as drones, is being widely used in various industries. These automated devices typically carry multiple types of sensors, such as millimeter-wave radar and cameras. These sensor devices each have their own advantages and characteristics, and they work together to meet the needs of various application scenarios.

[0003] How to fuse data collected by multiple sensor devices on an automated machine (e.g., millimeter-wave radar detection data and visual image data) so that the automated machine can easily integrate different sensor devices is an urgent problem that needs to be solved. [Summary of the Invention]

[0004] The joint calibration method, apparatus, electronic device, and UAV provided in this application can overcome at least some of the defects of existing data fusion methods.

[0005] In a first aspect, embodiments of this application provide a joint calibration method. This joint calibration method includes: acquiring pose information of a detection radar; the pose information includes: the ground clearance of the detection radar and the pitch angle of the detection radar; acquiring target calibration parameters matching the pose information from a preset set of calibration parameters; wherein the set of calibration parameters includes several sets of calibration parameters; each set of calibration parameters matches a pose information interval; the pose information interval includes: a height interval and a pitch angle interval; and determining the spatial transformation relationship between the detection radar and the image acquisition device based on the target calibration parameters.

[0006] Optionally, the spatial transformation relationship between the detection radar and the image acquisition device includes: the coordinate correspondence between the radar detection data of the target and the three-dimensional coordinates of the target in the detection radar coordinate system; a first coordinate transformation relationship between the detection radar coordinate system and the image acquisition device coordinate system; a second coordinate transformation relationship between the image acquisition device coordinate system and the two-dimensional image coordinate system; and a third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system.

[0007] Optionally, the coordinate correspondence is related to the pose information of the detection radar; the radar detection data includes: the distance between the detection radar and the target and the target horizontal angle between the detection radar and the target.

[0008] Optionally, the coordinate correspondence is expressed according to the following formula:

[0009]

[0010] The target's coordinates in the detection radar coordinate system are (X... r ,Y r Z r R is the distance between the target and the detection radar; O is the origin of the world coordinate system; B is the intersection of the z-axis of the detection radar coordinate system and the x-axis of the world coordinate system; C is the origin of the detection radar coordinate system; G is the intersection of the perpendicular line passing through the target and the x-axis of the world coordinate system; E is the intersection of the perpendicular line passing through point G and the z-axis of the detection radar coordinate system; H is the altitude of the detection radar above the ground; α is the elevation angle of the detection radar; θ radar To detect the horizontal angle between the radar and the target.

[0011] Optionally, the first coordinate transformation relationship is as shown in the following formula:

[0012]

[0013] The target's coordinates in the detection radar coordinate system are (X... r ,Y r Z r The target's coordinates in the image acquisition device's coordinate system are (X...). c Y c Z c R is an orthogonal rotation matrix, and t is a three-dimensional translation vector.

[0014] Optionally, the second coordinate transformation relationship is as shown in the following formula:

[0015]

[0016] Among them, the target's coordinates in the detection radar coordinate system (X) r ,Y r Z r R is the straight-line distance between the target and the detection radar; O is the origin of the world coordinate system; B is the intersection of the z-axis of the detection radar coordinate system and the x-axis of the world coordinate system; C is the origin of the detection radar coordinate system; G is the intersection of the perpendicular line passing through the target and the x-axis of the world coordinate system; E is the intersection of the perpendicular line passing through point G and the z-axis of the detection radar coordinate system; H is the altitude of the detection radar above the ground; α is the elevation angle of the detection radar; θ radar To detect the angle between the radar and the target.

[0017] Optionally, the second coordinate transformation relationship is as shown in the following formula:

[0018]

[0019] The target's coordinates in the image acquisition device's coordinate system are (X... c Y c Z c The target's coordinates in the two-dimensional image coordinate system are (x, y), and f is the focal length.

[0020] Optionally, the third coordinate transformation relationship is expressed according to the following formula:

[0021]

[0022] Wherein, the coordinates of the origin of the two-dimensional image coordinate system in the two-dimensional pixel coordinate system are (u o v o The target's coordinates in the two-dimensional pixel coordinate system are (u, v); the target's coordinates in the two-dimensional image coordinate system are (x, y); d is the ratio of the length of a single pixel in the two-dimensional pixel coordinate system to the length of a unit in the two-dimensional image coordinate system.

[0023] Optionally, obtaining target calibration parameters that match the pose information from a preset calibration parameter set further includes: refreshing the calibration parameter set when it is updated; and traversing the calibration parameter set according to the pose information of the current position of the detection radar to obtain a set of calibration parameters that match the current pose information.

[0024] Optionally, the method further includes: generating several pose information intervals; determining the pose information interval currently occupied by the detection radar, and calculating a set of calibration parameters matching the current pose information interval; after changing the pose information interval occupied by the detection radar, recalculating another set of calibration parameters matching the changed pose information interval; after all the calibration parameters matching each pose information interval have been calculated, recording each pose information interval and the matching calibration parameters to form the calibration parameter set.

[0025] Optionally, generating several pose information intervals specifically includes: dividing a preset range of ground height into m consecutive height intervals and dividing a preset range of pitch angles into n consecutive pitch angle intervals; combining each height interval with the n pitch angle intervals to generate m*n different pose information intervals.

[0026] Optionally, dividing the preset ground clearance range into m consecutive height intervals specifically includes:

[0027] Set the step value for dividing the height range;

[0028] The ground clearance range is divided into m height intervals according to the following formula:

[0029]

[0030] Where the subscript m is the index of the height interval, ceil is rounded up, and H... max H represents the upper limit of the range of ground clearance. min ΔH is the lower limit of the ground clearance range; ΔH is the division step value;

[0031] The step of dividing the preset pitch angle range into n consecutive pitch angle intervals specifically includes:

[0032] Set the step value for dividing the pitch angle range;

[0033] The pitch angle range is divided into n pitch angle intervals according to the following formula:

[0034]

[0035] Where, the subscript n is the index of the height interval, ceil is the rounding up value, and α... max α is the upper limit of the pitch angle range. min Δα is the lower limit of the pitch angle range; Δα is the division step value.

[0036] Optionally, the calculation of obtaining a set of calibration parameters that match the current pose information range specifically includes: acquiring several test coordinate data under the pose information; calculating and determining the undetermined parameters in a preset spatial transformation function as the calibration parameters using the test coordinate data; wherein the preset spatial transformation function is configured to represent the spatial transformation relationship between the detection radar and the image acquisition device.

[0037] Optionally, the spatial transformation relationship between the detection radar and the image acquisition device includes: the coordinate transformation relationship between the detection radar coordinate system and the two-dimensional pixel coordinate system; the test coordinate data includes: the first coordinate data of the test point in the detection radar coordinate system and the second coordinate data of the same test point in the two-dimensional pixel coordinate system.

[0038] Optionally, the method further includes: establishing a coordinate correspondence between the radar detection data of the target and the three-dimensional coordinates of the target in the radar coordinate system; sequentially determining a first coordinate transformation relationship between the radar coordinate system and the image acquisition device coordinate system; a second coordinate transformation relationship between the image acquisition device coordinate system and the two-dimensional image coordinate system; and a third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system; integrating the coordinate correspondence, the first coordinate transformation, the second coordinate transformation, and the third coordinate transformation to obtain the preset spatial transformation function; wherein the coordinate correspondence is related to the pose information of the radar; the radar detection data includes: the distance between the radar and the target and the horizontal angle between the radar and the target.

[0039] Optionally, the preset spatial transformation function is expressed according to the following formula:

[0040] p = K[R t]q

[0041] Wherein, coordinate q is the first coordinate data, coordinate p is the second coordinate data, K is the intrinsic parameter of the image acquisition device; R is the orthogonal rotation matrix, and t is the three-dimensional translation vector;

[0042] The orthogonal rotation matrix and the three-dimensional translation vector include several undetermined parameters, as shown in the following formula:

[0043] w = [θ x θ y θ z , t x , t y , t z ];

[0044] Where, θ x θ y and θ z These are the rotation angles of each coordinate axis; t x , t y and t z These represent the amount of movement of each coordinate axis in the corresponding direction.

[0045] Optionally, the step of calculating and determining the undetermined parameters in the preset spatial transformation function using the test coordinate data specifically includes:

[0046] The undetermined parameters are determined by calculating the nonlinear optimal solution of the following constraint function;

[0047]

[0048] Where p is the coordinate data of the test point in the two-dimensional pixel coordinate system; q is the coordinate data of the test point in the detection radar coordinate system; K is the intrinsic parameter of the image acquisition device; R is the orthogonal rotation matrix; and t is the three-dimensional translation vector.

[0049] Secondly, embodiments of this application provide a joint calibration device. This joint calibration device includes: a pose information acquisition module, used to acquire pose information of a detection radar; wherein the pose information includes: the ground clearance of the detection radar and the pitch angle of the detection radar; a calibration parameter search module, used to acquire target calibration parameters matching the pose information from a preset calibration parameter set; wherein the calibration parameter set includes several sets of calibration parameters; each set of calibration parameters matches a pose information interval; the pose information interval includes: a height interval and a pitch angle interval; and a calibration module, used to determine the spatial transformation relationship between the detection radar and the image acquisition device based on the target calibration parameters.

[0050] Thirdly, embodiments of this application provide an electronic device. The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the joint calibration method as described above.

[0051] Fourthly, this application provides an unmanned aerial vehicle (UAV). The UAV includes: a fuselage; a detection radar and an image acquisition device mounted on the fuselage; an arm connected to the fuselage; a power unit located on the arm for providing flight power to the UAV; and a flight controller located on the fuselage and communicatively connected to both the detection radar and the image acquisition device. The flight controller stores a preset set of calibration parameters and is configured to execute the joint calibration method described above to determine the correspondence between radar data from the detection radar and image data from the image acquisition device.

[0052] Optionally, the drone further includes: a gimbal; the gimbal is disposed on the underside of the fuselage; the detection radar and the image acquisition device are disposed on the gimbal; wherein the flight controller is configured to: obtain the pitch angle of the detection radar by the tilt angle of the gimbal.

[0053] Optionally, the drone further includes: an altitude-measuring radar; the altitude-measuring radar is mounted on the fuselage and is used to detect the ground clearance of the drone; wherein the flight controller is configured to: obtain the ground clearance of the detection radar based on the ground clearance of the drone detected by the altitude-measuring radar.

[0054] One advantage of the joint calibration method provided in this application is that it can correct and update the calibration parameters according to the attitude changes of the detection radar (e.g., changes in ground altitude and pitch angle), ensuring the accuracy of the obtained spatial transformation relationship and improving the data fusion effect of the detection radar and image acquisition equipment.

[0055] One advantage of the UAV provided in this application embodiment is that by storing a preset calibration parameter table, the calibration parameters can be adapted to changes in the attitude of the detection radar with relatively low computing power consumption, providing more accurate data fusion results. [Attached Image Description]

[0056] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0057] Figure 1 This is a schematic diagram of the application environment of an embodiment of this application;

[0058] Figure 2a The coordinate system correspondence diagram provided in this application illustrates the solid geometric relationship between the detection radar coordinate system and the world coordinate system;

[0059] Figure 2b This is a schematic diagram of the coordinate system correspondence provided in the embodiments of this application, illustrating the transformation of the coordinate system of the image acquisition device from a three-dimensional projection to a two-dimensional coordinate system;

[0060] Figure 2c This is a schematic diagram of the coordinate system correspondence provided in the embodiments of this application, illustrating the correspondence between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system;

[0061] Figure 3 A flowchart illustrating the joint calibration method provided in this application embodiment;

[0062] Figure 4 A flowchart illustrating a method for forming a calibration parameter set provided in this application embodiment;

[0063] Figure 5 A flowchart illustrating a method for obtaining a preset space transformation function provided in this application embodiment;

[0064] Figure 6 This is a schematic diagram of the calibration parameter set provided in the embodiments of this application, showing a calibration parameter table that records multiple sets of calibration parameters and their matching pose information intervals;

[0065] Figure 7aFunctional block diagram of the joint calibration device provided in the embodiments of this application

[0066] Figure 7b A functional block diagram of a joint calibration device provided in another embodiment of this application;

[0067] Figure 8 A schematic diagram of an electronic device provided in an embodiment of this application.

Detailed Implementation Methods

[0068] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that when an element is described as being "fixed to" another element, it can be directly on the other element, or one or more intermediate elements may exist between them. When an element is described as being "connected" to another element, it can be directly connected to the other element, or one or more intermediate elements may exist between them. The terms "upper," "lower," "inner," "outer," "bottom," etc., used in this specification indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0069] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0070] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0071] "Millimeter-wave radar" refers to a detection radar operating in the millimeter-wave band. It possesses strong penetrating power, capable of piercing through severe weather conditions such as heavy rain, heavy snow, and strong sandstorms. It can also accurately detect small targets in environments with reduced visibility due to high light intensity or in low-light conditions, thus solving the problem of low visibility and reduced perception for automated equipment (such as drones) under adverse conditions, and enhancing spatial situational awareness. This application uses millimeter-wave radar as an example for detailed description. Those skilled in the art will understand that other different types of detection radar can also be used.

[0072] "Image acquisition equipment" refers to sensors (e.g., action cameras or video cameras) that sense light signals in a target area and provide corresponding visual data. They are low-cost and offer advantages in object height and width measurement accuracy, contour recognition, and pedestrian identification accuracy, making them indispensable sensors for target classification, sign recognition, and other applications.

[0073] Typically, radar and visual data are fused together to allow the two sensors to complement each other, thereby establishing a multi-functional control system with capabilities such as sensor fusion perception, threat terrain warning, threat obstacle highlighting, and flight assistance. This enables the drone operator to have all-weather, all-terrain, and all-scenario environmental perception capabilities, thus providing sufficient time for timely avoidance of dangerous terrain and obstacles and ensuring the safe flight of the drone in any air condition.

[0074] "Joint calibration" refers to the process of determining the coordinate transformation relationship between multiple different coordinate systems. It is used to establish the correspondence between multi-source data (such as radar data and visual data), enabling data to be transformed between different coordinate systems, and is a prerequisite for data fusion.

[0075] Traditional joint calibration of millimeter-wave radar data and image data uses data models based on a two-dimensional plane, neglecting the altitude-related information of the millimeter-wave radar. However, in certain application scenarios (such as when millimeter-wave radar is mounted on a drone), the altitude and pitch angle of the millimeter-wave radar may change with the drone's flight attitude. The pitch angle refers to the angle between the radar's normal direction and the horizontal direction when the millimeter-wave radar is operating.

[0076] Therefore, traditional two-dimensional plane-based data models cannot adapt well to these usage scenarios. When the attitude of the UAV changes, its data model will fail, resulting in problems such as radar data not being accurately converted and projected into the coordinate system of image data and large deviations in depth information.

[0077] The applicant discovered that by establishing a data acquisition model based on three-dimensional space, it is possible to adapt to changes in the attitude of the UAV without losing data information from the millimeter-wave radar in terms of altitude and pitch angle. By providing calibration parameters that vary with altitude and pitch angle, data fusion between radar and visual data can be effectively achieved.

[0078] Figure 1 This is a schematic diagram illustrating the application environment provided in an embodiment of this application. The application environment uses a drone as an example. Figure 1 As shown, the drone 10 includes: fuselage 11, arms 12, power unit 13, and flight controller 14.

[0079] The fuselage 11 is the main structure of the UAV 10. It has a suitable size and shape to meet actual needs, providing sufficient space to accommodate one or more functional modules and components. For example, the fuselage 11 can be equipped with various sensor devices, including but not limited to detection radar and image acquisition devices.

[0080] In some embodiments, the underside of the fuselage may also be equipped with an adjustable gimbal or other similar structural device. The detection radar and image acquisition equipment are both mounted and fixed on the gimbal, allowing for convenient adjustment of the drone's pitch angle according to its flight altitude.

[0081] In other embodiments, the sensor device may further include an altimeter radar. This altimeter radar is a sensor device used to accurately detect the drone's altitude above the ground. Specifically, it can be any suitable type of precise distance detection device, such as millimeter-wave radar. Alternatively, other similar sensor devices, such as altimeters, can be used to detect the current altitude of the drone above the ground.

[0082] Arm 12 is the part that extends outward from the fuselage, serving as the mounting or fixing structure for the drone's power unit, such as propellers. The arm can be integrally formed with the fuselage or connected to the fuselage in a detachable manner. Typically, on a quadcopter drone, there can be four arms, extending symmetrically along the diagonal, forming four propeller mounting positions.

[0083] The power unit 13 is a structural device used to provide flight propulsion for the UAV. It can specifically employ any suitable type of power and structural design. For example, it could be a propeller driven by an electric motor, mounted and fixed at a position at the end of the arm.

[0084] The flight controller 14 is the core of the UAV control system built into the fuselage. It can be any type of electronic device with suitable logic and computational capabilities, including but not limited to processor chips based on large-scale integrated circuits, integrated system-on-a-chip (SoC), and processors and storage media connected via a bus. Depending on the functions to be implemented (e.g., executing the joint calibration method provided in the embodiments of this application), the flight controller 14 can include several different functional modules. These functional modules can be software modules, hardware modules, or a combination of software and hardware, and are modular devices used to implement one or more functions.

[0085] It should be noted that the embodiments of this application are provided for simplicity and illustrative purposes, demonstrating the application scenario of the joint calibration method in UAVs. However, those skilled in the art will understand that, based on similar principles, the joint calibration method provided in the embodiments of this application can also be applied to other application scenarios where millimeter-wave radar may experience changes in altitude and pitch angle. The inventive concept disclosed in the embodiments of this application is not limited to... Figure 1 The application shown is on the drone.

[0086] To fully illustrate the joint calibration method provided in the embodiments of this application, Figure 1 The specific application process in the application scenario shown below, combined with Figures 2a to 2c This paper provides a detailed description of the construction of a data acquisition model based on three-dimensional space. In this specific example, the data acquisition model describes the coordinate transformation relationships between the detection radar coordinate system, the image acquisition device coordinate system, the two-dimensional image coordinate system, and the two-dimensional pixel coordinate system.

[0087] Among them, the radar coordinate system is a three-dimensional coordinate system with the phase center of the transmitting antenna as the origin, satisfying the right-hand rule; the image acquisition equipment coordinate system is a three-dimensional coordinate system with the optical center of the equipment as the origin, satisfying the right-hand rule; the two-dimensional pixel coordinate system is a two-dimensional coordinate system with the upper left corner of the image plane as the origin, and the coordinate axes represent discrete pixels. The two-dimensional image coordinate system has the center of the imaging plane (such as CCD) as the origin, and its coordinate axes are parallel to the coordinate axes of the two-dimensional pixel coordinate system.

[0088] first, Figure 2a This is a schematic diagram of the three-dimensional geometric relationship between the detection radar coordinate system and the world coordinate system provided in the embodiments of this application. It shows the three-dimensional geometric relationship between the detection radar coordinate system and the world coordinate system when the millimeter-wave radar is at a specific altitude and a specific tilt angle as the UAV flies.

[0089] like Figure 2a As shown, D is any point in three-dimensional space (e.g., a target to be detected); the origin of the world coordinate system is O, and the three coordinate axes are represented as X1, Y1 and Z1 respectively; the origin of the radar coordinate system is C, and the three coordinate axes are represented as X, Y and Z respectively.

[0090] The detection radar's altitude above the ground is H, and its elevation angle is α (the angle between the radar's normal direction and the horizontal direction when the millimeter-wave radar is operating). The distance between the millimeter-wave radar and point D is R, where Rs is the center slant range of the millimeter-wave radar. The instantaneous azimuth angle of point D relative to the millimeter-wave radar is γ, and the instantaneous elevation angle of point D relative to the millimeter-wave radar is ψ. When the detection radar uses a one-dimensional linear MIMO array for angle measurement, the target horizontal angle between it and point D is obtained as θ. radar .

[0091] 1) In Figure 2a In the diagram, we can draw a line DG perpendicular to OB through point D, and a line GE perpendicular to CB through point G. Using the three perpendiculars theorem, we can determine that DE is perpendicular to BC. In the plane formed by BCJ, we can draw a line DQ perpendicular to plane BCJ. Through these auxiliary line segments, we can determine that the three-dimensional coordinates of point D in the radar coordinate system are D = [DG, -GE, CE].

[0092] 2) The millimeter-wave radar detection data for point D mainly includes the distance R between the radar and point D, and the horizontal angle of the target between the radar and point D. The specific calculation and detection process for these two radar detection data is as follows:

[0093] 2.1) For the range R, taking a frequency modulated continuous wave (FMCW) radar as an example, a millimeter-wave radar can transmit a frequency modulated continuous wave signal. The frequency of this frequency modulated continuous wave signal can change linearly within each frequency modulation cycle. When the reflected echo signal is received, the reflected echo signal can first be digitally down-converted, then the sample values ​​can be sorted into a two-dimensional matrix, and then the time-domain echo signal can be transformed into the frequency domain dimension through a two-dimensional (2-D) Fast Fourier Transform (FFT) to obtain the two-dimensional Doppler matrix (RDM) corresponding to the target. Combined with the constant false alarm rate (CFAR) detection algorithm, the target range R of the target can be obtained.

[0094] 2.2) For the target horizontal angle, taking the two-dimensional DOA (Direction of Arrival) estimation algorithm as an example, the specific detection process is as follows:

[0095] Suppose that for point D, there exists a radar array consisting of N antennas, with an element spacing of d = λ / 2, where λ is the wavelength. Assume that the angular position of point D relative to the radar in three-dimensional space is (γ, ψ). Here, γ∈(-π / 2, π / 2) and ψ∈(0, π / 2) represent the instantaneous azimuth and elevation angles of any point target, respectively. Then, the signal vector s used to estimate the direction of arrival (DOA) can be expressed by the following formula (1-1):

[0096] s=A·a(γ,ψ) (1-1)

[0097] Where A represents the scattering coefficient of any point target, and a(γ,ψ) represents the signal steering vector, which can be expressed by the following formula (1-2):

[0098] a(γ, ψ) = [1, e] -j2πdsinγcosψ / λ , ...e -j2π(N-1)sdinγcosψ / λ ] H (1-2)

[0099] For one-dimensional DOA estimation, the steering vector considering only the azimuth angle can be expressed as:

[0100] b = [1, e] -j2πdsinγ / λ , ...e -j2π(N-1)dsinγ / λ ] H (1-3)

[0101] Therefore, the estimated azimuth angle can be obtained using the following formula (1-4):

[0102]

[0103] After determining the target distance R and the target height difference H, as follows Figure 2a As shown, the instantaneous elevation angle between the detection radar and any point target D can be expressed by the following formula (1-5):

[0104]

[0105] Therefore, the azimuth steering vector corresponding to the pitch angle caused by altitude can be expressed by the following formula (1-6):

[0106]

[0107] Where d is the uniform antenna element spacing, and N is the number of receiving antennas, [ ] H This represents the transpose and conjugate of the matrix. At this point, the target horizontal angle is estimated using DOA as follows:

[0108]

[0109] 3) Based on the steering vector expression derived in the above steps, when using a one-dimensional linear MIMO array for angle measurement, the angle between the radar and the target D is θ. radar Combining Figure 2a Based on the geometric relationships, the following formula (2-1) can be determined:

[0110] sin∠θ radar =cos∠DCQ*sin∠QCE (2-1)

[0111] 3.1) According to the folding angle formula in solid geometry, the following formula (2-2) can be determined to be satisfied between different angles:

[0112] cos∠DCE=cos∠QCE*cos∠DCQ (2-2)

[0113] 3.2) Combining equations (2-1) and (2-2), we can simplify to obtain the following equation (3):

[0114]

[0115] 3.2) Combining Figure 2aBased on the geometric relationships, the above formula (3) can be further simplified to the following formula (4):

[0116] QE=DG=Rsin∠θ radar (4)

[0117] 3.3) Combining Figure 2a The geometric relationship between OB and OD satisfies the following formula (5):

[0118]

[0119] 3.4) Combining Figure 2a The geometric relationship in CE can be calculated using the following formula (6):

[0120] CE=Hsinα+OGcosα (6)

[0121] 3.5) Using the principle of similar triangles, the ratio between line segments can be determined to satisfy the following formula (7):

[0122]

[0123] BE can be obtained by subtracting CE from Rs.

[0124] Therefore, based on the radar data and pose information detected by the detection radar, the three-dimensional coordinates of any point D in three-dimensional space in the detection radar coordinate system can be calculated as follows (8):

[0125]

[0126] As shown in equation (8), when the data acquisition model determines the three-dimensional coordinates of the target in the radar coordinate system based on the radar detection data, it introduces two parameters: the ground clearance H and the pitch angle α of the detection radar, so that it can well reflect the situation of the detection radar when the ground clearance and pitch angle change.

[0127] Secondly, the coordinate transformation relationship between the detection radar coordinate system and the image acquisition coordinate system can be represented by the constructed orthogonal rotation matrix and three-dimensional translation vector. The transformation relationship between the detection radar coordinate system and the image acquisition coordinate system can be expressed by the following formula (9):

[0128]

[0129] Among them, (X) r Y r Z r (X) represents the coordinate position in the radar coordinate system. c Y c Zc Let ) represent the coordinate position in the coordinate system of the image acquisition device, R be the orthogonal rotation matrix, and t be the three-dimensional translation vector. The three-dimensional translation vector and the orthogonal rotation matrix are represented by the following formulas (9-1) and (9-2), respectively:

[0130] t=(X t Y t Z t ) T (9-1)

[0131]

[0132] Given that multiple three-dimensional spatial sample points are located in the radar coordinate system and the image acquisition equipment coordinate system, the three-dimensional rotation angle in the orthogonal rotation matrix and the translation amount in the three-dimensional translation vector can be calculated and determined by any suitable method, thereby obtaining the coordinate transformation relationship between the radar coordinate system and the image acquisition coordinate system.

[0133] It should be noted that the specific methods for calculating and determining the orthogonal rotation matrix and the three-dimensional translation vector are well known to those skilled in the art and will not be elaborated here.

[0134] again, Figure 2b This is a schematic diagram illustrating the projection relationship between the coordinate system of the image acquisition device and the two-dimensional image coordinate system provided in the embodiments of this application, showing the transformation of the coordinate system of the image acquisition device from a three-dimensional projection to a two-dimensional coordinate system.

[0135] like Figure 2b As shown, the origin of the coordinate system of the image acquisition device is O. c The three coordinate axes are represented as X c Y c and Z c The origin of the two-dimensional image coordinate system is O, and the two coordinate axes are represented as x and y, respectively. P is any point in the coordinate system of the image acquisition device, and p is the projection of point P onto the imaging plane.

[0136] 1.1) In Figure 2b In the middle, point O c The triangle formed by Cp and point O c The triangle formed by BP is a similar triangle; point O c The triangle formed by CO and point O c The triangle formed by BA is also a similar triangle. Therefore, we can obtain the following formula (10):

[0137]

[0138] Where f is the focal length, and the coordinates of point P in the coordinate system of the image acquisition device are represented as (X... c Yc Z c The coordinates of point p in the two-dimensional image coordinate system are represented as (x, y).

[0139] 1.2) After transforming formula (10), the coordinate data of point p shown in formula (11) can be obtained:

[0140]

[0141] 1.3) By rearranging equation (10), the coordinate transformation relationship between the coordinate system of the image acquisition device and the two-dimensional image coordinate system can be obtained as shown in equation (12) below:

[0142]

[0143] Finally, the coordinate data in the two-dimensional image coordinate system obtained by the transformation of formula (12) usually uses length units such as mm, rather than discrete pixels. When commonly used image acquisition devices (such as digital cameras) acquire images, they first form a standard electrical signal and then convert it into a digital image through digital-to-analog conversion. The storage form of each acquired image is an M×N array, where the value of each element in the M rows and N columns of the image represents the gray level of the image. Thus, the coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system can be further determined to facilitate the data fusion between radar data and image data.

[0144] Figure 2c This is a schematic diagram illustrating the correspondence between a two-dimensional image coordinate system and a two-dimensional pixel coordinate system provided in an embodiment of this application. For example... Figure 2c As shown, the two-dimensional image coordinate system has its origin at the center of the image plane, and its two coordinate axes are parallel to the two perpendicular sides of the image plane, denoted by X and Y respectively. Coordinates in the two-dimensional image coordinate system can be represented as (x, y), with units of mm.

[0145] The two-dimensional pixel coordinate system has its origin at the top-left corner of the image plane, and its two coordinate axes are parallel to the X-axis and Y-axis of the two-dimensional image coordinate system, respectively, and are denoted by U and V. Coordinates in the two-dimensional pixel coordinate system can be represented by (u, v).

[0146] Let 1 pixel equal dmm. The coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system can be expressed by the following formula (13):

[0147]

[0148] Among them, (u o v o) is the coordinate value of the origin of the two-dimensional image coordinate system in the two-dimensional pixel coordinate system. Further, equation (13) can be rearranged to obtain the coordinate transformation relationship shown in equation (14):

[0149]

[0150] Therefore, based on the specific example of the above data acquisition model, any point in three-dimensional space can be transformed from the radar coordinate system to the pixel coordinate system through the following formula (15), thereby realizing the data fusion of image data and radar data. Formula (15) is obtained by combining the above formulas (9), (12) and (14).

[0151]

[0152] Those skilled in the art will understand that in the above formula (15), K is an intrinsic parameter of the image acquisition device. The specific method for obtaining K is well known to those skilled in the art and can be determined by calibration methods such as the Zhang Zhengyou calibration method, which will not be elaborated here. T is a calibration parameter related to the altitude and elevation angle of the detection radar, which will change with the altitude and elevation angle of the detection radar.

[0153] It should be noted that the specific examples of the data acquisition model provided in this application are only used to illustrate how to introduce the altitude and elevation angle information of the detection radar into the coordinate transformation relationship between the detection radar coordinate system and the two-dimensional pixel coordinate system, and are not intended to limit the scope of this application. Depending on practical needs or the characteristics of specific application scenarios, those skilled in the art can easily conceive of adjusting, replacing, or changing one or more steps or parameters to obtain other data acquisition models through reasonable derivation.

[0154] One advantage of the data acquisition model provided in this application embodiment is that it takes into account the influence of the attitude change of the detection radar in three-dimensional space, effectively solving the problem of data acquisition plane failure caused by changes in altitude and pitch angle in scenarios such as UAV applications.

[0155] This application also provides a joint calibration method. This joint calibration method is based on a data acquisition model that incorporates the attitude changes of the detection radar. Figure 3 The joint calibration method provided for embodiments of this application. For example... Figure 3 As shown, the joint calibration method includes the following steps:

[0156] S310: Acquire the position and orientation information of the detection radar.

[0157] This pose information may include: the altitude of the detection radar above the ground and the pitch angle of the detection radar. In actual operation, using... Figure 1 Taking the illustrated application scenario as an example, the ground clearance of the detection radar fixedly mounted on the drone gimbal is the drone's flight altitude, which can be obtained through sensor devices such as the drone's altimeter radar, GPS module, or altitude sensor. The pitch angle of the detection radar can be determined by reading the tilt angle of the drone gimbal. In a preferred embodiment, an altimeter radar with high detection accuracy can be used to detect the ground clearance of the detection radar. Specifically, this detection radar can be a millimeter-wave radar capable of acquiring depth information of objects.

[0158] S320. Obtain the target calibration parameters that match the pose information from the preset calibration parameter set.

[0159] The "calibration parameter set" refers to a data set consisting of multiple calibration parameters. It can be stored and retrieved in any suitable data format. For example, it can be stored as a parameter table in a specific non-volatile storage medium. Within the calibration parameter set, one or more calibration parameters that match a specific pose information range can be referred to as a set of calibration parameters. The specific calibration parameters included in each set can be set according to actual needs.

[0160] Corresponding to step S310, the pose information range can also include a height range and a pitch angle range. In this embodiment, "range" refers to a specific numerical range, and the specific size and division method of the numerical range can be set according to actual needs.

[0161] "Target calibration parameters" refer to a set of calibration parameters that match the current pose information of the detection radar. In practice, the pose information range in which the detection radar is located can be determined based on its current pose information. Then, the calibration parameters that match this pose information range can be searched in the calibration parameter set in any suitable way and used as the target calibration parameters.

[0162] In some embodiments, the data of the above-mentioned calibration parameter set can be stored in the storage space of the UAV controller and directly accessed by the corresponding processor when needed. In other embodiments, the calibration parameter set can also be stored in other servers or non-volatile storage media that have a communication connection with the UAV, providing the required target calibration parameters to the UAV controller through a wireless communication connection channel.

[0163] S330. Based on the target calibration parameters, determine the spatial transformation relationship between the detection radar and the image acquisition equipment.

[0164] The "spatial transformation relationship" refers to the spatial correspondence between the detection radar and the image acquisition equipment. It can be represented by one or more rotation matrices or other similar methods, so that the radar data obtained by the detection radar and / or the visual data obtained by the image acquisition equipment can be transformed between multiple different coordinate systems, thereby completing data fusion.

[0165] Specifically, taking the above data acquisition model as an example, the spatial transformation relationship can include: the coordinate correspondence between the radar detection data of the target and the three-dimensional coordinates of the target in the detection radar coordinate system; the first coordinate transformation relationship between the detection radar coordinate system and the image acquisition device coordinate system; the second coordinate transformation relationship between the image acquisition device coordinate system and the two-dimensional image coordinate system; and the third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system.

[0166] Among them, such as Figure 2a As shown in equation (8), this coordinate relationship is a function related to the ground clearance and elevation angle of the detection radar. At a specific ground clearance and elevation angle, the target range R and the target horizontal angle θ detected by the millimeter-wave radar are... radar This allows us to calculate the target's three-dimensional coordinates in the radar coordinate system.

[0167] In other words, changes in the radar's altitude and pitch angle will cause changes in the three-dimensional coordinates of the same target in the radar's coordinate system. By incorporating the radar's attitude change information into the model in this way, more accurate data fusion between radar data and image data obtained by the radar can be achieved.

[0168] Furthermore, the first coordinate transformation relationship can be represented by equation (9), the second coordinate transformation relationship can be represented by equation (12), and the third coordinate transformation relationship can be represented by equation (14). By integrating the first, second, and third coordinate transformation relationships, the coordinate transformation function between the detection radar coordinate system and the two-dimensional pixel coordinate system, as shown in equation (15), can be obtained.

[0169] One advantage of the joint calibration method provided in this application is that the calibration parameters can be corrected accordingly based on changes in the pose information of the detection radar, thereby obtaining an accurate spatial transformation relationship that matches the current position of the detection radar. Based on this spatial transformation relationship, the radar data obtained by the detection radar can be easily converted into a two-dimensional pixel coordinate system, realizing data fusion between multiple sources such as depth information and image visual data.

[0170] The calibration parameter set in this embodiment is data information that is pre-created and stored in the UAV's flight controller through a series of methods and steps, so that it can be called up at any time during the data fusion process. Figure 4 A method for constructing a calibration parameter set provided in the embodiments of this application. For example... Figure 4 As shown, the method for constructing this calibration parameter set includes the following steps:

[0171] S410, Generate several pose information intervals.

[0172] The pose information interval can be defined by two different parameters: a height interval and a pitch angle interval. The specific number of pose information intervals can be generated according to actual needs.

[0173] In some embodiments, the method for generating several pose information intervals may specifically include: first, dividing a preset range of ground height into m consecutive height intervals, and dividing a preset range of pitch angles into n consecutive pitch angle intervals. Then, combining each height interval with the n pitch angle intervals to generate m*n different pose information intervals.

[0174] The preset ground clearance range can be from zero to the drone's maximum flight altitude. The pitch angle range can be determined by the drone's gimbal. Both m and n are positive integers and can be set or adjusted according to actual conditions (e.g., the size of the ground clearance range).

[0175] S420. Determine the current pose information range of the detection radar and calculate a set of calibration parameters that match the current pose information range.

[0176] Among them, the calibration parameters can be derived and determined based on the established data acquisition model, using multiple known test data to determine the spatial transformation relationship between the detection radar and the image acquisition device at a specific location, and then the corresponding calibration parameters can be calculated and determined.

[0177] In some embodiments, the calibration parameters can be calculated using known test coordinate data. First, several test coordinate data points are acquired under the pose information. Then, using the test coordinate data, undetermined parameters in a preset spatial transformation function are calculated and determined as calibration parameters.

[0178] The preset spatial transformation function represents the spatial transformation relationship between the detection radar and the image acquisition device. In this embodiment, the preset "spatial transformation function" can have the same or similar expression as the "spatial transformation relationship," the difference being that the spatial transformation function contains several undetermined parameters. In other words, given the values ​​of all undetermined parameters, substituting these values ​​allows the acquisition of the desired spatial transformation relationship based on the spatial transformation function.

[0179] The test coordinate data is related to the actual spatial transformation function (or spatial transformation relationship) used. Specifically, when the spatial transformation relationship between the detection radar and the image acquisition device includes the coordinate transformation relationship between the detection radar coordinate system and the two-dimensional pixel coordinate system, the test coordinate data can include: the first coordinate data of the test point in the detection radar coordinate system and the second coordinate data of the same test point in the two-dimensional pixel coordinate system.

[0180] S430. After changing the pose information range of the detection radar, recalculate another set of calibration parameters that match the changed pose information range.

[0181] Specifically, the pose information of the detection radar is changed to a new pose information range, and step S420 is repeated to obtain calibration parameters that match this pose information range. In actual operation, the above steps S420 and S430 can be executed in a computer-built simulation environment to quickly obtain the required calibration parameter information.

[0182] S440. After all the calibration parameters matching each pose information interval have been calculated, record each pose information interval and the matching calibration parameters to form a calibration parameter set.

[0183] By executing steps S420 and S430 multiple times, the calibration parameters matching each pose information interval can be determined. These calibration parameters and their matching relationships with the pose information intervals can be recorded and saved in any suitable manner, thereby forming the required set of calibration parameters.

[0184] In some embodiments, taking a specific example of the above-described data admission model, such as... Figure 5 As shown, the specific steps to obtain the space transformation function may include:

[0185] S510. Establish the coordinate correspondence between the radar detection data of the target and the three-dimensional coordinates of the target in the detection radar coordinate system.

[0186] The coordinate correspondence can be represented as shown in equation (8), which is a three-dimensional coordinate expression related to altitude and pitch angle. It can represent the three-dimensional coordinates of the target by the distance between the detection radar and the target, and the horizontal angle between the detection radar and the target.

[0187] S520. Sequentially determine the first coordinate transformation relationship between the detection radar coordinate system and the image acquisition device coordinate system, the second coordinate transformation relationship between the image acquisition device coordinate system and the two-dimensional image coordinate system, and the third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system.

[0188] The first, second, and third coordinate transformation relationships are shown in equations (9), (12), and (14), respectively. Here, "first," "second," and "third" are used only to distinguish coordinate transformation relationships between different coordinate systems, and not to limit their specific aspects such as expression.

[0189] S530. Integrate the coordinate correspondence, the first coordinate transformation relationship, the second coordinate transformation relationship, and the third coordinate transformation relationship to obtain the preset spatial transformation function.

[0190] In this embodiment, the term "integration" is used to refer to one or more data operations that combine multiple transformation relationships and three-dimensional coordinate expressions, and then simplify and / or reorganize them accordingly. Specific mathematical operations are not limited here and can be adjusted or set according to actual needs, as long as the correspondence between the target in the detection radar coordinate system and the two-dimensional pixel coordinate system can be determined.

[0191] In other embodiments, to adapt to different application scenarios or changes in actual conditions, the data of the above-mentioned calibration parameter set may further include an updating step. When the calibration parameter set is updated, a refresh operation can be performed on the calibration parameter set. Then, based on the current pose information of the detection radar, the calibration parameter set is traversed to obtain a set of calibration parameters that match the current pose information.

[0192] To fully illustrate the calibration parameter set construction method of this application embodiment, the following uses the data admission model shown in formula (15) as an example to describe in detail the specific process of constructing the calibration parameter set based on it.

[0193] 1) Divide the height into continuous intervals:

[0194] First, set the step value for dividing the height range and the range of height above the ground.

[0195] The division step value is an empirical value and can be set or adjusted by technicians according to actual needs. Preferably, the division step value can be appropriately increased to reduce the number of altitude intervals and the number of external parameter adjustment parameters. The ground clearance range can be set according to the actual flight altitude range of the UAV during normal operation, and is not specifically limited here.

[0196] Then, according to the following formula (16-1), the range of ground clearance is divided into m height intervals:

[0197]

[0198] Where the subscript m is the index of the height interval, ceil is rounded up, and H... max H represents the upper limit of the range of altitudes above the ground. min ΔH is the lower limit of the ground clearance range; ΔH is the division step value.

[0199] 2) Divide the pitch angle into continuous intervals:

[0200] First, set the step value for dividing the pitch angle interval and the pitch angle range.

[0201] The pitch angle increments are similar to those for altitude ranges; they are also empirical values ​​that can be set or adjusted by technicians based on actual needs. The pitch angle range is set based on the range of pitch angles the gimbal might adjust during normal drone operation, and is not specifically limited here. Preferably, a larger pitch angle range can be set to cover as many extreme situations as possible during drone flight, ensuring the accuracy of the calibration parameters.

[0202] Then, according to the following formula (16-2), the pitch angle range is divided into n pitch angle intervals:

[0203]

[0204] Where, the subscript n is the index of the height interval, ceil is the rounding up value, and α... max α is the upper limit of the pitch angle range. min Δα is the lower limit of the pitch angle range; Δα is the division step value.

[0205] Therefore, the m height intervals and n pitch angle intervals obtained through the above division can form m*n pose information intervals.

[0206] 3) Calculation of calibration parameters:

[0207] Assuming that for a test target in space, its coordinates in the two-dimensional pixel coordinate system are p and its coordinates in the detection radar coordinate system are q, the above formula (15) can be rearranged and transformed into the following formula (17):

[0208] p=K[R t]q (17)

[0209] Here, K is an intrinsic parameter that does not change with the altitude and pitch angle of the detection radar. It can be obtained by calibrating the image acquisition equipment through methods such as Zhang Zhengyou's calibration method. R and t are calibration parameters that need to be determined.

[0210] As described in the data admission model above, there are a total of six calibration parameters that need to be determined in R and t, as shown in the following formula (18):

[0211] w = [θ x θ y θ z , t x , t y , t z (18)

[0212] Where, θ x θ y and θ z These are the rotation angles of each coordinate axis; t x , t y and t z These represent the amount of movement of each coordinate axis in the corresponding direction.

[0213] Given several known sets of corresponding coordinates p and q, the above six calibration parameters are determined by solving the nonlinear optimal solution of the constraint function shown in the following formula (19), so as to serve as a set of calibration parameters that match the pose information range.

[0214]

[0215] 4) Generate a set of calibration parameters:

[0216] By repeatedly changing the altitude and elevation angle of the detection radar and repeating step 3), all m*n pose information intervals and their corresponding set of calibration parameters can be obtained. The calculated sets of calibration parameters and their corresponding matching relationships with the pose information intervals can be derived from... Figure 6 The calibration parameters are recorded in the table shown.

[0217] Among them, such as Figure 6 As shown, H m α represents the current altitude range of the detection radar above the ground. nThis indicates the current elevation angle range of the detection radar, [R] mn t mn ] indicates the height range H m and pitch angle range α n A matching set of calibration parameters.

[0218] In practical applications, steps 1) to 4) above, which generate the calibration parameter set, can be executed in a simulation environment pre-built on an electronic computing platform to obtain... Figure 6 The calibration parameter table is shown. In some embodiments, this calibration parameter table is stored in the UAV's local storage medium. Of course, the calibration parameter table can also be stored in other remotely arranged storage devices, not limited to the UAV's local storage medium.

[0219] exist Figure 1 In the application scenario shown, when fusing radar data and image data, the UAV first obtains its current flight altitude and gimbal pitch angle based on relevant sensor equipment, and then determines the target altitude range within which the current flight altitude is located and the target pitch angle range within which the current pitch angle is located. Next, a set of calibration parameters matching the aforementioned target altitude and pitch angle ranges is searched and read from the calibration parameter table. Finally, the read calibration parameters are used to determine the coordinate transformation relationship between the radar coordinate system and the two-dimensional pixel coordinate system, enabling the radar data to be accurately transformed to the two-dimensional pixel coordinate system, thus achieving data fusion between radar data and image data.

[0220] For example, the depth information of a target object obtained by radar can be converted into a two-dimensional pixel coordinate system to determine the depth information of the pixel where the target object is located. This enables functions such as threat terrain warning, threat obstacle highlighting, and flight assistance, thereby helping drone operators obtain all-weather, all-terrain, and all-scenario environmental perception capabilities, providing sufficient time for timely avoidance of dangerous terrain and obstacles.

[0221] Figure 7a This is a functional block diagram of the joint calibration device provided in the embodiments of this application. Figure 7a As shown, the joint calibration device 700 may include: a pose information acquisition module 710, a calibration parameter search module 720, and a calibration module 730.

[0222] The pose information acquisition module 710 is used to acquire the pose information of the detection radar. The pose information includes the radar's altitude above the ground and its pitch angle. The calibration parameter search module 720 is used to acquire target calibration parameters that match the pose information from a preset set of calibration parameters. The calibration parameter set includes several sets of calibration parameters; each set of calibration parameters matches a pose information interval; the pose information interval includes a height interval and a pitch angle interval. The calibration module 730 is used to determine the spatial transformation relationship between the detection radar and the image acquisition device based on the target calibration parameters.

[0223] In some embodiments, the spatial transformation relationship between the detection radar and the image acquisition device includes: the coordinate correspondence between the radar detection data of the target and the three-dimensional coordinates of the target in the detection radar coordinate system; a first coordinate transformation relationship between the detection radar coordinate system and the image acquisition device coordinate system; a second coordinate transformation relationship between the image acquisition device coordinate system and the two-dimensional image coordinate system; and a third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system.

[0224] Specifically, the coordinate correspondence is related to the pose information of the detection radar, and it changes with the radar's altitude and pitch angle. The radar detection data includes: the distance between the detection radar and the target, and the horizontal angle between the detection radar and the target.

[0225] In other embodiments, such as Figure 7b As shown, the joint calibration device may further include a calibration parameter calculation module 740 for generating a calibration parameter set. The calibration parameter calculation module 740 is used to: generate several pose information intervals; determine the pose information interval currently occupied by the detection radar, and calculate a set of calibration parameters matching the current pose information interval; after changing the pose information interval occupied by the detection radar, recalculate another set of calibration parameters matching the changed pose information interval; after all the calibration parameters matching each pose information interval have been calculated, record each pose information interval and the matching calibration parameters to form the calibration parameter set.

[0226] It should be noted that, in the embodiments of this application, functionally named modules are used as examples to describe in detail the method steps to be implemented by the joint calibration device provided in the embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will realize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0227] Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application. The computer software can be stored in a computer-readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0228] Figure 8 The diagram illustrates the structure of an electronic device according to an embodiment of this application. This application does not limit the specific implementation of the electronic device. For example, it could be made of... Figure 1 The image shows the flight control chip carried by the drone.

[0229] like Figure 8 As shown, the electronic device may include: a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0230] The processor 802, communication interface 804, and memory 806 communicate with each other via communication bus 808. Communication interface 804 is used to communicate with other network elements such as clients or other servers. The processor 802 executes program 810, specifically performing the relevant steps in the above-described joint calibration method embodiment.

[0231] Specifically, program 810 may include program code, which includes computer operation instructions. Specifically, it can be used to cause processor 802 to execute the joint calibration method in any of the above method embodiments.

[0232] In the embodiments of this application, depending on the type of hardware used, the processor 802 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0233] Memory 806 is used to store program 810. Memory 806 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage, flash memory device, or other non-volatile solid-state storage device.

[0234] It has a program storage area and a data storage area, which are used to store the program 810 and its corresponding data information, respectively. For example, non-volatile software programs, non-volatile computer executable programs and modules are stored in the program storage area, or the calculation results, radar data and image information are stored in the data storage area.

[0235] This application also provides a computer-readable storage medium. This computer-readable storage medium can be a non-volatile computer-readable storage medium. This computer-readable storage medium stores a computer program.

[0236] When executed by a processor, the computer program implements one or more steps of the joint calibration method disclosed in the embodiments of this application. The complete computer program product is embodied on one or more computer-readable storage media (including but not limited to, disk storage, CD-ROM, optical storage, etc.) containing the computer program disclosed in the embodiments of this application.

[0237] In summary, the data input model constructed by the joint calibration method and apparatus provided in this application takes into account the influence of the millimeter-wave radar's operating altitude and elevation angle on the Direction of Arrival (DOA) estimation. It can adaptively adjust calibration parameters to suit operating conditions at any altitude and elevation angle.

[0238] Furthermore, the aforementioned data entry model and joint calibration method are constructed based on three-dimensional space and have good scalability. They can be applied to suitable scenarios by continuing to use the derivation methods and calculation results of the embodiments in this application, and by simultaneously setting the height and pitch angle to zero, thus degenerating to a typical two-dimensional data acquisition model.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A joint calibration method, characterized in that, include: Acquire the pose information of the detection radar; The pose information includes: the ground clearance of the detection radar and the pitch angle of the detection radar; From the preset set of calibration parameters, obtain the target calibration parameters that match the pose information; The calibration parameter set includes several sets of calibration parameters; each set of calibration parameters is matched with a pose information interval; the pose information interval includes: a height interval and a pitch angle interval; Based on the target calibration parameters, the spatial transformation relationship between the detection radar and the image acquisition equipment is determined; The spatial conversion relationship between the detection radar and the image acquisition equipment includes: The radar detection data of the target and the coordinate correspondence between the target's three-dimensional coordinates in the detection radar coordinate system; The first coordinate transformation relationship between the detection radar coordinate system and the image acquisition device coordinate system; The second coordinate transformation relationship between the coordinate system of the image acquisition device and the two-dimensional image coordinate system; and The third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system; The method further includes: Several pose information intervals are generated, the pose information interval of the detection radar is currently located is determined, and a set of calibration parameters matching the current pose information interval are calculated. After changing the pose information range of the detection radar, another set of calibration parameters that matches the changed pose information range is recalculated; After all the calibration parameters matching each pose information interval have been calculated, each pose information interval and the matching calibration parameters are recorded to form the calibration parameter set. The generation of several pose information intervals specifically includes: The preset range of ground clearance is divided into m consecutive height intervals, and the preset range of pitch angles is divided into n consecutive pitch angle intervals. Each height interval is combined with n pitch angle intervals to generate m*n different pose information intervals.

2. The method according to claim 1, characterized in that, The coordinate correspondence is related to the pose information of the detection radar; The radar detection data includes: the distance between the detection radar and the target, and the horizontal angle between the detection radar and the target.

3. The method according to claim 2, characterized in that, The coordinate correspondence is shown in the following formula: The target's coordinates in the detection radar coordinate system are ( , , R is the distance between the target and the detection radar; O is the origin of the world coordinate system; B is the intersection of the z-axis of the detection radar coordinate system and the x-axis of the world coordinate system; C is the origin of the detection radar coordinate system; G is the intersection of the perpendicular line passing through the target and the x-axis of the world coordinate system; E is the intersection of the perpendicular line passing through point G and the z-axis of the detection radar coordinate system; H is the altitude of the detection radar above the ground. To detect the elevation angle of the radar; To detect the horizontal angle between the radar and the target.

4. The method according to claim 2, characterized in that, The first coordinate transformation relationship is shown in the following formula: The target's coordinates in the detection radar coordinate system are ( , , The target's coordinates in the image acquisition device's coordinate system are ( ); , , R is an orthogonal rotation matrix, and t is a three-dimensional translation vector.

5. The method according to claim 2, characterized in that, The second coordinate transformation relationship is shown in the following formula: The coordinates of the target in the coordinate system of the image acquisition device are ( , , The target's coordinates in the two-dimensional image coordinate system are (x, y), and f is the focal length.

6. The method according to claim 2, characterized in that, The third coordinate transformation relationship is shown in the following formula: Wherein, the coordinates of the origin of the two-dimensional image coordinate system in the two-dimensional pixel coordinate system are ( , The target's coordinates in the two-dimensional pixel coordinate system are (u, v); the target's coordinates in the two-dimensional image coordinate system are (x, y); d is the ratio of the length of a single pixel in the two-dimensional pixel coordinate system to the length of a unit in the two-dimensional image coordinate system.

7. The method according to any one of claims 1-6, characterized in that, The step of obtaining target calibration parameters that match the pose information from a preset set of calibration parameters further includes: When the calibration parameter set is updated, the calibration parameter set is refreshed; Based on the current pose information of the detection radar, the calibration parameter set is traversed to obtain a set of calibration parameters that match the current pose information.

8. The method according to claim 7, characterized in that, The process of dividing the preset ground clearance range into m consecutive height intervals specifically includes: Set the step value for dividing the height range; The ground clearance range is divided into m height intervals according to the following formula: Where the subscript m is the index of the height interval, and ceil is the rounding up option. This is the upper limit of the range of ground clearance. This is the lower limit of the aforementioned range of ground clearance; To divide the step values; The step of dividing the preset pitch angle range into n consecutive pitch angle intervals specifically includes: Set the step value for dividing the pitch angle range; The pitch angle range is divided into n pitch angle intervals according to the following formula: Where the subscript n is the index of the height interval, and ceil is the rounding up option. This is the upper limit of the pitch angle range. This is the lower limit of the pitch angle range; To divide the step values.

9. The method according to claim 1, characterized in that, The calculation obtains a set of calibration parameters that match the current pose information range, specifically including: Acquire several test coordinate data under the pose information; Using the test coordinate data, the undetermined parameters in the preset spatial transformation function are calculated and determined as the calibration parameters; The preset spatial transformation function is configured to represent the spatial transformation relationship between the detection radar and the image acquisition device.

10. The method according to claim 9, characterized in that, The spatial transformation relationship between the detection radar and the image acquisition device includes: the coordinate transformation relationship between the detection radar coordinate system and the two-dimensional pixel coordinate system; The test coordinate data includes: the first coordinate data of the test point in the radar coordinate system and the second coordinate data of the same test point in the two-dimensional pixel coordinate system.

11. The method according to claim 9, characterized in that, Also includes: Establish the coordinate correspondence between the radar detection data of the target and the three-dimensional coordinates of the target in the detection radar coordinate system; The first coordinate transformation relationship between the radar coordinate system and the image acquisition equipment coordinate system is determined sequentially. The second coordinate transformation relationship between the image acquisition device coordinate system and the two-dimensional image coordinate system; and the third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system; By integrating the coordinate correspondence, the first coordinate transformation relationship, the second coordinate transformation relationship, and the third coordinate transformation relationship, the preset spatial transformation function is obtained; The coordinate correspondence is related to the pose information of the detection radar; the radar detection data includes: the distance between the detection radar and the target and the horizontal angle between the detection radar and the target.

12. The method according to claim 10 or 11, characterized in that, The preset spatial transformation function is shown in the following formula: p = K[R t]q Wherein, coordinate q is the first coordinate data, coordinate p is the second coordinate data, K is the intrinsic parameter of the image acquisition device; R is the orthogonal rotation matrix, and t is the three-dimensional translation vector; The orthogonal rotation matrix and the three-dimensional translation vector include several undetermined parameters, as shown in the following formula: w＝[ , ]； in, , and These are the rotation angles of each coordinate axis; , and These represent the amount of movement of each coordinate axis in the corresponding direction.

13. The method according to claim 12, characterized in that, The step of calculating and determining the undetermined parameters in the preset spatial transformation function using the test coordinate data specifically includes: The undetermined parameters are determined by calculating the nonlinear optimal solution of the following constraint function; Where p is the coordinate data of the test point in the two-dimensional pixel coordinate system; q is the coordinate data of the test point in the detection radar coordinate system; K is the intrinsic parameter of the image acquisition device; R is the orthogonal rotation matrix; and t is the three-dimensional translation vector.

14. A combined calibration device, characterized in that, include: The pose information acquisition module is used to acquire the pose information of the detection radar; wherein, the pose information includes: the ground clearance of the detection radar and the pitch angle of the detection radar. The calibration parameter search module is used to obtain target calibration parameters that match the pose information from a preset set of calibration parameters; wherein, the set of calibration parameters includes several sets of calibration parameters; each set of calibration parameters matches a pose information interval; the pose information interval includes: a height interval and a pitch angle interval; The calibration module is used to determine the spatial transformation relationship between the detection radar and the image acquisition device based on the target calibration parameters. The calibration parameter calculation module is used to generate several pose information intervals, determine the pose information interval where the detection radar is currently located, and calculate a set of calibration parameters that match the current pose information interval; after changing the pose information interval where the detection radar is located, recalculate another set of calibration parameters that match the changed pose information interval; after all the calibration parameters that match each pose information interval have been calculated, record each pose information interval and the matching calibration parameters to form the calibration parameter set; The generation of several pose information intervals specifically includes: The preset range of ground clearance is divided into m consecutive height intervals, and the preset range of pitch angles is divided into n consecutive pitch angle intervals. Each height interval is combined with n pitch angle intervals to generate m*n different pose information intervals; The spatial conversion relationship between the detection radar and the image acquisition equipment includes: The radar detection data of the target and the coordinate correspondence between the target's three-dimensional coordinates in the detection radar coordinate system; The first coordinate transformation relationship between the detection radar coordinate system and the image acquisition device coordinate system; The second coordinate transformation relationship between the coordinate system of the image acquisition device and the two-dimensional image coordinate system; and The third coordinate transformation relationship between the two-dimensional image coordinate system and the two-dimensional pixel coordinate system.

15. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the joint calibration method according to any one of claims 1-13.

16. An unmanned aerial vehicle (UAV), characterized in that, include: The fuselage; the fuselage is equipped with detection radar and image acquisition equipment; The arm is connected to the machine body; A power unit, located on the arm, is used to provide the drone with the power to fly; as well as A flight controller is located on the fuselage and is communicatively connected to the detection radar and the image acquisition equipment, respectively. The flight controller stores a preset set of calibration parameters and is configured to execute the joint calibration method as described in any one of claims 1-13 to determine the correspondence between the radar data of the detection radar and the image data of the image acquisition device.

17. The UAV according to claim 16, characterized in that, Also includes: Gimbal; The gimbal is mounted on the underside of the fuselage; the detection radar and the image acquisition device are mounted on the gimbal. The flight controller is configured to obtain the pitch angle of the detection radar by measuring the tilt angle of the gimbal.

18. The UAV according to claim 16, characterized in that, Also includes: Altimeter radar; The altitude-measuring radar is mounted on the fuselage and is used to detect the ground altitude of the UAV. The flight controller is configured to obtain the ground altitude of the detection radar by measuring the ground altitude of the UAV detected by the altimeter radar.

Images