Load cooperation method and device, electronic equipment, readable storage medium and chip

Abstract

The invention provides a load collaboration method and device, electronic equipment, a readable storage medium and a chip, the load collaboration method is used for an unmanned aerial vehicle, the unmanned aerial vehicle comprises an unmanned aerial vehicle platform system and an unmanned aerial vehicle fuselage, and the unmanned aerial vehicle fuselage is provided with a photoelectric load and a radar load. The load cooperation method comprises the following steps: acquiring a first parameter of a photoelectric load and a second parameter of a radar load; position information parameters corresponding to the unmanned aerial vehicle are obtained through the unmanned aerial vehicle platform system; determining a second detection range corresponding to the radar load according to the second parameter; performing equivalent angle conversion setting on the first parameter according to the position information parameter, and determining a first detection range corresponding to the photoelectric load; determining a first image acquired by a photoelectric load and a second image acquired by a radar load; and determining a global situation map corresponding to the unmanned aerial vehicle according to the first image and the second image. Through the scheme of the invention, the cooperative efficiency between loads during unmanned aerial vehicle reconnaissance is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle payload coordination technology, and in particular to a payload coordination method, device, electronic device, readable storage medium and chip. Background Art

[0002] As a new type of aerial combat force, UAVs are characterized by high degree of autonomy, strong flexibility and mobility, and high risk resistance. They can be quickly deployed to harsh environments that are inconvenient to enter, highly polluted places, or complex battlefields that are difficult to clear with air defense firepower. They can perform flight missions or support missions with high complexity, strong penetration, and high risk for a long time, with high reliability and intensity. UAVs are equipped with reconnaissance payloads such as optoelectronics and radar to perform tasks such as address surveying, reconnaissance surveillance, or tracking and positioning. Optoelectronic payloads have the advantages of strong concealment, high resolution, strong detection capability, and intuitive display results, but they are easily affected by factors such as clouds, light, and humidity, and cannot be used all day. Radar payloads can provide more accurate detection and positioning capabilities, are not restricted by weather and light conditions, and can work in all weather and all time conditions, but the resolution is not high and it is not easy to interpret. When a UAV performs a long-term reconnaissance mission, the operator needs to process the information of the two payloads for a long time, and the coordinated control of the two payloads is difficult, and the burden of joint information processing is heavy. Summary of the invention

[0003] In view of this, the present invention aims to solve the problems of difficulty in coordinated control of optoelectronic payloads and radar payloads carried by UAVs and heavy burden of joint information processing when the UAVs perform long-term reconnaissance missions.

[0004] Specifically, the present invention is achieved through the following technical solutions:

[0005] A first aspect of the present invention provides a load coordination method.

[0006] A second aspect of the present invention provides a load coordination device.

[0007] A third aspect of the present invention provides a drone.

[0008] A fourth aspect of the present invention provides an electronic device.

[0009] A fifth aspect of the present invention provides a readable storage medium.

[0010] A sixth aspect of the present invention provides a chip.

[0011] The load coordination method provided by the present invention is used for an unmanned aerial vehicle. The unmanned aerial vehicle includes an unmanned aerial vehicle platform system and an unmanned aerial vehicle fuselage. An optoelectronic load and a radar load are arranged on the unmanned aerial vehicle fuselage. The load coordination method includes: obtaining a first parameter of the optoelectronic load and a second parameter of the radar load; obtaining a position information parameter corresponding to the unmanned aerial vehicle through the unmanned aerial vehicle platform system; determining a second detection range corresponding to the radar load according to the second parameter; performing an equivalent angle conversion setting on the first parameter according to the position information parameter to determine a first detection range corresponding to the optoelectronic load, wherein the first detection range and the second detection range are equal; determining a first image collected by the optoelectronic load and a second image collected by the radar load; and determining a global situation map corresponding to the unmanned aerial vehicle according to the first image and the second image, wherein the global situation map includes target features of a bird's-eye view.

[0012] In some technical solutions, optionally, determining a second detection range corresponding to the radar payload based on a second parameter includes: determining an operating mode of the radar payload based on the second parameter; acquiring a second detection parameter corresponding to the operating mode; and determining a second detection range corresponding to the radar payload based on the second detection parameter.

[0013] In some technical schemes, optionally, an equivalent angle conversion setting is performed on the first parameter according to the position information parameter to determine the first detection range corresponding to the photoelectric load, including: determining a navigation coordinate system centered on the UAV; determining the pitch angle parameter of the photoelectric load in the navigation coordinate system according to the position information parameter and the first parameter; determining the first detection parameter of the photoelectric load; determining the first horizontal field of view angle of the photoelectric load according to the pitch angle parameter and the first detection parameter; and determining the first detection range corresponding to the photoelectric load according to the first horizontal field of view angle.

[0014] In some technical solutions, optionally, determining a first image captured by the optoelectronic load and a second image captured by the radar load includes: determining a first acquisition parameter of the optoelectronic load in a first detection range; performing image acquisition according to the first acquisition parameter to determine a first image corresponding to the optoelectronic load; determining a second acquisition parameter of the radar load in a second detection range; performing image acquisition according to the second acquisition parameter to determine a second image corresponding to the radar load.

[0015] In some technical solutions, optionally, a global situation map corresponding to the drone is determined based on the first image and the second image, including: determining a first feature point cloud corresponding to the first image; determining a second feature point cloud corresponding to the second image; fusing the first feature point cloud and the second feature point cloud based on a bird's-eye view encoder to determine a bird's-eye view feature; and determining the global situation map based on the bird's-eye view feature.

[0016] The second aspect of the present invention provides a load coordination device, including: an acquisition module, used to obtain a first parameter of the optoelectronic load and a second parameter of the radar load; a UAV module, which obtains a position information parameter corresponding to the UAV through a UAV platform system; a determination module, which determines a second detection range corresponding to the radar load according to the second parameter; a range determination module, which performs an equivalent angle conversion setting on the first parameter according to the position information parameter to determine the first detection range corresponding to the optoelectronic load; a collection module, which determines a first image collected by the optoelectronic load and a second image collected by the radar load; and a construction module, which determines a global situation map corresponding to the UAV according to the first image and the second image.

[0017] A third aspect of the present invention provides an unmanned aerial vehicle, comprising an unmanned aerial vehicle platform system and an unmanned aerial vehicle fuselage, on which an optoelectronic load and a radar load are arranged; and a load coordination device as in the second aspect.

[0018] An embodiment of the fourth aspect of the present invention provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the steps in the first aspect when executed by the processor.

[0019] An embodiment of the fifth aspect of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps in the first aspect are implemented.

[0020] An embodiment of the sixth aspect of the present invention provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps in the first aspect.

[0021] The technical solution provided by the present invention brings at least the following beneficial effects:

[0022] The present invention proposes a payload coordination method, device, electronic device, readable storage medium and chip, which automatically align the detection ranges of the optoelectronic payload and radar payload carried by the UAV platform, and perform fusion detection of the optical image and radar image based on the fusion technology of the bird's-eye view, thereby reducing the burden of joint processing of reconnaissance information and improving the coordination efficiency between payloads during UAV reconnaissance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A schematic diagram of a flow chart of a load coordination method provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a portion of the flow chart of the load coordination method provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a portion of the flow chart of the load coordination method provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a portion of the flow chart of the load coordination method provided by an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a portion of the flow chart of the load coordination method provided by an embodiment of the present invention;

[0030] Figure 6 A schematic block diagram of the structure of a load coordination device provided by an embodiment of the present invention;

[0031] Figure 7 A schematic block diagram of the structure of a drone provided by an embodiment of the present invention;

[0032] Figure 8 A schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0033] Fig. 9 A partial flow chart of a load coordination method provided in an embodiment of the present invention.

[0034] in, Figures 6 to 8 The corresponding relationship between the component names and numbers in is as follows:

[0035] 900: payload coordination device; 902: acquisition module; 904: drone module; 906: determination module; 908: range determination module; 910: acquisition module; 912: construction module; 3000: drone; 3002: drone platform system; 3004: drone body; 3006: optoelectronic payload; 3008: radar payload; 1000: electronic equipment; 1109: memory; 1110: processor. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 In a first aspect, the present invention provides a load coordination method for a drone, wherein the drone includes a drone platform system and a drone fuselage, and an optoelectronic load and a radar load are arranged on the drone fuselage. The load coordination method includes the following steps:

[0038] Step S100: Acquire a first parameter of an optoelectronic payload and a second parameter of a radar payload;

[0039] Step S102: obtaining location information parameters corresponding to the drone through the drone platform system;

[0040] Step S104: determining a second detection range corresponding to the radar payload according to the second parameter;

[0041] Step S106: performing an equivalent angle conversion setting on the first parameter according to the position information parameter to determine a first detection range corresponding to the photoelectric load;

[0042] Step S108: determining the first image collected by the optoelectronic payload and the second image collected by the radar payload;

[0043] Step S110: Determine a global situation map corresponding to the UAV according to the first image and the second image.

[0044] According to the load coordination method provided by the present invention, the detection ranges of the optoelectronic load and the radar load on the UAV are aligned, the first detection range of the optoelectronic load and the second detection range of the radar load at the same time are synchronized, and then the optical images and synthetic aperture radar (SAR) images detected by the optoelectronic load and the radar load are fused and detected to determine the global situation map corresponding to the detection or monitoring of the UAV, and the global situation map is a three-dimensional global situation map with a bird's-eye view (BEV) target feature, which is used for tasks such as image segmentation, target detection and track detection. Specifically, the basic information such as the optoelectronic resolution, azimuth, and pitch angle corresponding to the optoelectronic load is obtained as the first parameter, and the optoelectronic load is a visible light and infrared passive dynamic detection device, and any one of the two-dimensional imaging devices, which has technical advantages such as high resolution, wide spectrum, small size, light weight and low power consumption, and is classified from the detection band, and the corresponding optoelectronic parameter data can be used as the first parameter; the second parameter corresponding to the radar load is obtained, that is, the radar load operation setting parameter, and the radar load type is determined by the operation setting parameter, and the radar load type is a synthetic aperture radar. The UAV platform system is a data transmission and control system set in the UAV. Users or operators can remotely control the UAV or obtain UAV data information in real time through the UAV platform system. The UAV data information includes the position information parameters corresponding to the UAV fuselage, namely, the UAV longitude, UAV latitude, UAV altitude, UAV roll, UAV pitch and other information, as well as the optical image and SAR image obtained by the optoelectronic load and radar load. The detection range of the radar load in the spotlight synthetic aperture radar mode is a fixed value, and the fixed detection range corresponding to the radar load, namely, the second detection range, can be directly determined by the second parameter; according to the azimuth and pitch angle of the optoelectronic load, the posture disturbance of the UAV itself due to the azimuth, pitch and roll angle is analyzed to determine the horizontal field of view of the optoelectronic load, thereby setting the parameters of the optoelectronic load and the radar load so that the detection range of the optoelectronic load and the radar load at the same time is equal. The UAV performs automatic reconnaissance according to the optoelectronic load and the radar load with equal detection range, and determines the first image corresponding to the optoelectronic load and the second image corresponding to the radar load. The first image is an optical image, and the second image is a synthetic aperture radar image, and the first image and the second image are obtained by the optoelectronic load and the radar load with equal detection range. Determine an optical image bird's-eye view feature corresponding to the first image and a synthetic aperture radar point cloud bird's-eye view feature corresponding to the second image, fuse the optical image bird's-eye view feature and the synthetic aperture radar point cloud bird's-eye view feature according to a bird's-eye view encoder, determine a fused bird's-eye view feature, and establish a three-dimensional global situation map corresponding to the fused bird's-eye view feature.

[0045] It can be understood that optical images and synthetic aperture radar images have different result characteristics. The unified fusion of optical images and synthetic aperture radar images based on the bird's-eye view will retain the geometric structure and semantic density of the image when converted to the bird's-eye view space, reduce the geometric distortion of the optical image and synthetic aperture radar image in the unified conversion process, and improve the accuracy of the fusion of optical images and synthetic aperture radar images.

[0046] In some embodiments, optionally, Figure 2 As shown, determining a second detection range corresponding to the radar payload according to the second parameter includes:

[0047] Step S1042: determining the working mode of the radar payload according to the second parameter;

[0048] Step S1044: Acquire a second detection parameter corresponding to the working mode;

[0049] Step S1046: Determine a second detection range corresponding to the radar payload according to the second detection parameter.

[0050] In this embodiment, the UAV operator determines that the type of radar payload used by the UAV is a synthetic aperture radar (SAR) according to the second parameter, i.e., the radar operation setting parameter, and the corresponding working mode is a spotlight SAR mode. The detection range distance of the radar payload in the spotlight SAR mode is d, i.e., the second detection parameter; the second detection range is determined to be d×dm according to the detection range distance d. 2 , that is, the detection range of the radar payload in the spotlight SAR mode is a fixed value, which is d×d.

[0051] In some embodiments, optionally, Figure 3 As shown, performing an equivalent angle conversion setting on the first parameter according to the position information parameter to determine a first detection range corresponding to the photoelectric load includes:

[0052] Step S1062: Determine a navigation coordinate system centered on the drone;

[0053] Step S1064: determining a pitch angle parameter of the optoelectronic payload in the navigation coordinate system according to the position information parameter and the first parameter;

[0054] Step S1066: determining a first detection parameter of the photoelectric load;

[0055] Step S1068: determining a first horizontal field of view angle of the photoelectric payload according to the pitch angle parameter and the first detection parameter;

[0056] Step S1070: Determine a first detection range corresponding to the photoelectric load according to the first horizontal field of view.

[0057] In this embodiment, since the optoelectronic load is a visible light and infrared dynamic detection device, as well as any one of the two-dimensional imaging devices, its detection range is a dynamic detection range, and the dynamic detection range of the optoelectronic load needs to be adjusted to ensure that the detection ranges of the optoelectronic load and the radar load corresponding to the drone are the same and aligned at the same time. And since the optoelectronic load is set up on the drone body, it will be disturbed by the attitude during the operation of the drone, so it is necessary to determine the position information parameters of the drone, and adjust the horizontal field of view angle and the pitch field of view angle of the optoelectronic load according to the position information parameters of the drone, wherein the position information parameters corresponding to the drone are obtained through the drone platform system, and the position information parameters include: drone longitude, drone latitude, drone altitude, drone roll, drone pitch and other information. The optoelectronic resolution, azimuth and pitch angle of the optoelectronic load can also be determined by the drone platform system. Specifically, the navigation coordinate system centered on the drone is determined by the drone platform system. For example, in the northeast sky, the pitch angle is 0° with the horizontal line, upward is positive, and downward is negative; according to the azimuth angle A and pitch angle B of the optoelectronic load obtained by the drone platform system, and the azimuth angle Y, pitch angle P, and roll angle R corresponding to the drone, the equivalent azimuth angle θ and pitch angle of the optoelectronic load in the navigation coordinate system are calculated. The values ​​of the equivalent azimuth and elevation angles are the elevation angle parameters, which are calculated as follows:

[0058]

[0059] Equivalent azimuth angle θ, elevation angle for:

[0060]

[0061]

[0062] Among them, x, y, z are the coordinate values ​​of the x-axis, y-axis, and z-axis in the geographic coordinate system respectively.

[0063] The first detection parameter of the photoelectric load is determined according to the equivalent azimuth and pitch angle, which corresponds to the detection range d of the photoelectric load. The visible light of the photoelectric load obtained by the UAV platform system is 1920×1080 resolution. The horizontal field of view angle and the pitch field of view angle are 2α and 2β respectively, where:

[0064] The detection range of the photoelectric load detection overlooking field angle is d1;

[0065]

[0066] d1 = d;

[0067]

[0068] Where H is the aircraft height (in meters), and the values ​​of β and α are calculated;

[0069] The detection range distance of the horizontal field angle of the photoelectric load detection is d2;

[0070]

[0071] d2=d;

[0072]

[0073] Where H is the aircraft altitude (unit: m), and the values ​​of β and α can be calculated accordingly.

[0074] Compare the two α values ​​and take the smaller of the two values ​​as the first horizontal field of view angle as the calculation result, so that the parameters of the optoelectronics and radar can be set so that the detection ranges of the two loads are consistent.

[0075] It can be understood that by adjusting the dynamic detection range of the optoelectronic payload, that is, determining a horizontal field of view angle with a smaller value during the operation of the UAV as the first horizontal field of view angle, the amount of information processing during the adjustment of the optoelectronic payload detection range can be reduced, and the detection range of the optoelectronic payload can be aligned with the detection range of the radar payload more quickly at the same time, thereby improving the coordination efficiency between the optoelectronic payload and the radar payload of the UAV during long-term reconnaissance missions.

[0076] In some embodiments, optionally, Figure 4 As shown, determining the first image collected by the optoelectronic payload and the second image collected by the radar payload includes:

[0077] Step S1082: determining a first acquisition parameter of the photoelectric load in a first detection range;

[0078] Step S1084: performing image acquisition according to the first acquisition parameter to determine a first image corresponding to the photoelectric load;

[0079] Step S1086: determining a second acquisition parameter of the radar payload in a second detection range;

[0080] Step S1088: performing image acquisition according to the second acquisition parameter to determine a second image corresponding to the radar payload.

[0081] In this embodiment, the UAV platform system performs data adjustment based on the optoelectronic load and radar load corresponding to the first detection range and the second detection range that have been aligned, and determines the image acquisition data of the optoelectronic load in the first detection range and the radar load in the second detection range, that is, the first acquisition parameter and the second acquisition parameter. The image acquisition data is used to determine the image acquisition related settings of the optoelectronic load and the radar load in the motion state of the UAV, and is used to determine the image acquisition accuracy of the optoelectronic load and the radar load corresponding to the UAV. The optoelectronic load performs image acquisition according to the first acquisition parameter to determine the first image, which is an optical image; the radar load performs image acquisition according to the second acquisition parameter to determine the second image, which is a synthetic aperture radar image.

[0082] In some embodiments, optionally, Figure 5 As shown, determining a global situation map corresponding to the UAV according to the first image and the second image includes:

[0083] Step S1102: determining a first feature point cloud corresponding to the first image;

[0084] Step S1104: determining a second feature point cloud corresponding to the second image;

[0085] Step S1106: fusing the first feature point cloud and the second feature point cloud based on the bird's-eye view encoder to determine a bird's-eye view feature;

[0086] Step S1108: Determine the global situation map based on the bird's-eye view features.

[0087] In this embodiment, determining a first feature point cloud corresponding to a first image, i.e., an optical image, includes: performing depth prediction on the optical image, determining a discrete depth distribution of each image in the optical image, converting a two-dimensional optical image feature into a frustum feature through discrete depth prediction, and determining a depth probability of each feature pixel in the optical image according to the frustum feature; scattering each feature pixel to at least one discrete point along a camera ray, determining a scaling feature according to the depth probability corresponding to the feature pixel, and determining a feature point cloud corresponding to the optical image according to at least one scaling feature, i.e., a first feature point cloud. Obtaining a feature point cloud corresponding to a synthetic aperture radar image by projecting a synthetic aperture radar image in a bird's-eye view space, performing preprocessing such as denoising and filtering on the synthetic aperture radar image, extracting image features such as corner points, edge points, geometric shapes, and texture features from the preprocessed image, and projecting the extracted image feature points into a BEV space in combination with information such as terrain height, sensor attitude, and image rotation angle, and organizing the feature point coordinates into a point cloud data structure after mapping, i.e., a second feature point cloud. In the UAV platform system, the first feature point cloud and the second feature point cloud are fused through the bird's-eye view encoder (BEV Encoder) to determine the bird's-eye view features, and based on the projection of the bird's-eye view features (Fused BEV Features) in the bird's-eye view space, a three-dimensional global situation map with bird's-eye view target features is established. The global situation map is used as the detection result after the coordination of the UAV payload and is transmitted to the operator through the UAV platform system. The operator can perform tasks such as image segmentation, target detection and track prediction based on the global situation map.

[0088] In a specific embodiment, the present invention synchronizes the reconnaissance ranges of optoelectronics and radar at the same time by aligning the reconnaissance ranges of optoelectronics and radar, and then performs fusion detection on the optical image and SAR image reconnaissance of optoelectronics and radar, such as Fig. 9 As shown, the process includes:

[0089] Step S1902: Obtaining drone platform information;

[0090] Step S1904: Acquire information of optoelectronic payload and radar payload;

[0091] Step S1906: Aligning the detection range based on the equivalent angle;

[0092] Step S1908: Setting the parameters of the optoelectronic payload and radar payload and automatically conducting reconnaissance;

[0093] Step S1910: constructing multi-sensor fusion information based on a bird's eye view (BEV);

[0094] Step S1912: Collaborative reconnaissance and target detection.

[0095] Among them, the UAV platform information includes: UAV longitude, latitude, altitude, roll, pitch and other information; the optoelectronic payload information includes: optoelectronic resolution, azimuth, pitch angle and other information; the radar payload information includes radar detection range and other information. The detection range alignment based on equivalent angle depends on the optoelectronic horizontal field of view and optoelectronic vertical field of view of the optoelectronic payload; the UAV obtains optoelectronic video or image information and radar SAR image information through the optoelectronic payload and radar payload parameter settings and automatic reconnaissance; the global situation map is constructed based on the multi-sensor fusion information of the bird's eye view (BEV), and the global situation map is used for tasks such as image segmentation and target detection.

[0096] Specifically, step S906: based on the reconnaissance range alignment of equivalent angle conversion, the detection range of the radar in the spotlight SAR mode is a fixed value, set to d×dm 2 .

[0097] Where d is the radar payload detection range distance.

[0098] According to the azimuth (set as A) and pitch angle (set as B) of the optoelectronic payload, the UAV platform itself has attitude disturbance (azimuth Y, pitch angle P, roll angle R) (navigation system: northeast sky, pitch angle is 0° with the horizontal line, upward is positive, and downward is negative), calculate the equivalent azimuth (set as θ) and pitch angle (set as ), the calculation method is as follows:

[0099]

[0100] Equivalent azimuth angle θ, elevation angle for:

[0101]

[0102] Among them, x, y, z are the coordinate values ​​of the x-axis, y-axis, and z-axis in the geographic coordinate system respectively.

[0103] The visible light of the optoelectronic payload generally adopts a resolution of 1920×1080, and the horizontal field of view and the elevation field of view are

[0104]

[0105] The detection range of the photoelectric load detection overlooking field angle is d1;

[0106]

[0107] d1 = d;

[0108]

[0109] Where H is the aircraft height (in meters), and the values ​​of β and α are calculated;

[0110] The detection range distance of the horizontal field angle of the photoelectric load detection is d2;

[0111]

[0112] d2=d;

[0113]

[0114] Where H is the aircraft altitude (unit: m), and the values ​​of β and α can be calculated accordingly.

[0115] Compare the two α values ​​and take the smaller one as the calculation result, so that the parameters of the optoelectronics and radar can be set so that the detection range of the two loads is consistent.

[0116] Step S910: Multi-sensor fusion information construction based on bird's-eye view (BEV): The detection results of the two sensors, optoelectronics and SAR radar, have different features, and different features can exist in different views. For example, optical features are located in perspective views, while SAR radar features are typically located in 3D / bird's-eye view views. Even for optical features, each has a different observation angle (i.e., front, back, left, and right). This view difference makes feature fusion difficult because the same elements in different feature tensors may correspond to completely different spatial positions (and in this case, simple element-by-element feature fusion will not work). Therefore, it is crucial to find a shared representation so that all sensor features can be easily converted to it without losing information, and suitable for different types of tasks.

[0117] Inspired by RGB-D data, the point cloud obtained by SAR image transformation can be projected to the camera plane and render 2.5D sparse depth, but this transformation is geometrically lossy, and two adjacent points on the depth map may be far apart in 3D space. This makes the camera view ineffective for tasks that focus on object / scene geometry, such as 3D object detection. In addition, when optical images are fused with SAR point clouds, camera and SAR features have completely different densities, resulting in less than 5% of camera features being matched to SAR points.

[0118] Therefore, the present invention adopts the bird's-eye view (BEV) as a unified representation for fusion, which is friendly to almost all perception tasks because the output space is also in BEV. More importantly, the conversion to BEV preserves the geometric structure and semantic density. On the one hand, the projection of the SAR point cloud to BEV flattens the sparse SAR features along the height dimension, so no geometric distortion is produced in the image. On the other hand, the projection of the camera to BEV projects each camera feature pixel back to a ray in 3D space, which can generate a dense BEV feature map that retains all the semantic information of the camera. The algorithm for transforming the optical image to BEV space is as follows:

[0119] Performing depth prediction on the optical image to predict the discrete depth distribution of each pixel, wherein a discrete depth feature corresponding to the optical image is determined by acquiring the discrete depth distribution of the optical image, a discrete depth distribution map corresponding to the optical image is determined according to a per-pixel outer product in the discrete depth feature, and a length D and a width C of the discrete depth distribution map are determined according to the outer product;

[0120] Construct a three-dimensional coordinate system, which includes the d-axis, u-axis, and v-axis. By determining the coordinates F(u, v) of the image feature point (ImageFeature) in the three-dimensional coordinate system, and determining the corresponding distribution coordinates D(u, v) of the discrete depth distribution prediction based on the corresponding data in the discrete depth distribution map, the image features of the two-dimensional optical image are converted into frustum features (Frustum Features), and the coordinates G(u, v) of the frustum features are determined. The feature points are converted through the distribution on the coordinate axis u and the coordinate axis v, and the depth d on the coordinate axis d is converted according to the discrete features. i = 0 is converted into a two-dimensional optical image at depth d on coordinate axis d i =D frustum feature, and obtain discrete depth distribution under three-dimensional perspective.

[0121] The three-dimensional features are represented by A, B, and C respectively. Each feature pixel is scattered to D discrete points along the camera ray, and the relevant features are rescaled according to the corresponding depth probability. The A feature pixels are rescaled according to the depth probabilities of 0.1, 0.1, 0.2 and 0.6 respectively; the B feature pixels are rescaled according to the depth probabilities of 0.1, 0.1, 0.5 and 0.3 respectively; the C feature pixels are rescaled according to the depth probabilities of 0.1, 0.5, 0.2 and 0.2 respectively.

[0122] Generate a camera feature point cloud of size N×H×W×D, where D is the number of discrete points, N is the number of cameras, and (H,W) is the size of the camera feature map. Such a feature point cloud is quantized along the x and y axes with a step size of r (for example, 0.4m). Use the BEV pooling operation to aggregate all features within each r×r BEV grid and flatten the features along the z axis. The optical image BEV features and the SAR point cloud BEV features are fused simultaneously through the BEV encoder to obtain the fused BEV features. Specifically:

[0123] The optical image bird's-eye view features (Camera Feat (in BEV)) and the synthetic aperture radar point cloud bird's-eye view features (SAR Point Cloud (in BEV)) are fused through the bird's-eye view encoder (BEV Encoder) to determine the fused bird's-eye view features (Fused BEV Features). Combined with the projection of the detection results in the BEV space, a three-dimensional global situation map with BEV target features is established for tasks such as image segmentation, target detection, and track prediction.

[0124] like Figure 6 As shown, the second aspect of the present invention provides a load coordination device 900, which includes: an acquisition module 902, used to obtain a first parameter of the optoelectronic load and a second parameter of the radar load; a UAV module 904, which obtains a position information parameter corresponding to the UAV through a UAV platform system; a determination module 906, which determines a second detection range corresponding to the radar load according to the second parameter; a range determination module 908, which performs an equivalent angle conversion setting on the first parameter according to the position information parameter, and determines the first detection range corresponding to the optoelectronic load; an acquisition module 910, which determines a first image collected by the optoelectronic load and a second image collected by the radar load; and a construction module 912, which determines a global situation map corresponding to the UAV according to the first image and the second image.

[0125] According to the load coordination device 900 provided by the present invention, the detection ranges of the optoelectronic load and the radar load on the drone are aligned, the first detection range of the optoelectronic load and the second detection range of the radar load at the same time are synchronized, and then the optical images and synthetic aperture radar (SAR) images detected by the optoelectronic load and the radar load are fused and detected to determine the global situation map corresponding to the drone detection or monitoring. Specifically, the acquisition module 902 acquires the first parameter corresponding to the optoelectronic load and the second parameter of the radar load, wherein the first parameter is the basic information such as the optoelectronic resolution, azimuth, and pitch angle of the optoelectronic load; and the second parameter is the radar load operation setting parameter. The drone module 904 acquires the position information parameters corresponding to the drone through the drone platform system, namely the drone longitude, drone latitude, drone altitude, drone roll, drone pitch and other information, as well as the optical images and SAR images acquired by the optoelectronic load and the radar load. The determination module 906 is used to determine the working mode of the radar load, and determine the second detection range corresponding to the radar load according to the working mode. The range determination module 908 adjusts the dynamic detection range of the optoelectronic payload, and by determining the first detection range corresponding to the optoelectronic payload, makes the optoelectronic payload and the radar payload of the UAV have the same and aligned detection range at the same time. The acquisition module 910 enables the optoelectronic payload and the radar payload to perform image acquisition while determining the detection range, and determines the optical image of the optoelectronic payload and the synthetic aperture radar image of the radar payload. The construction module 912 fuses the first feature point cloud and the second feature point cloud through the bird's-eye view encoder (BEV Encoder), determines the bird's-eye view features, and establishes a three-dimensional global situation map with bird's-eye view target features based on the projection of the bird's-eye view features (Fused BEV Features) in the bird's-eye view space, and uses the global situation map as the detection result after the coordination of the UAV payload.

[0126] like Figure 7 As shown, the third aspect of the present invention provides a UAV 3000, which includes: a UAV platform system 3002 and a UAV fuselage 3004, on which an optoelectronic payload 3006 and a radar payload 3008 are arranged; and a payload coordination device as in the second aspect of the present invention.

[0127] Among them, the drone platform system 3002 is an information collection and processing system built into the drone 3000. The drone platform system 3002 can collect the position information parameters of the drone body 3004, that is, the longitude of the drone 3000, the latitude of the drone 3000, the altitude of the drone 3000, the roll of the drone 3000, the pitch of the drone 3000, etc. It can also collect the data information of the optoelectronic load 3006 and the radar load 3008 set on the drone body 3004, as well as the optical images and SAR images obtained by the optoelectronic load 3006 and the radar load 3008. The collected images or data are remotely transmitted to the operator through wireless or Bluetooth data transmission.

[0128] like Figure 8 As shown, the fourth aspect of the present invention provides an electronic device 1000, including a processor 1110, a memory 1109, and a program or instruction stored in the memory 1109 and executable on the processor 1110. When the program or instruction is executed by the processor 1110, the various processes of the embodiment of the above-mentioned power balancing scheduling method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] Among them, the processor 1110 is used to obtain a first parameter of the optoelectronic payload and a second parameter of the radar payload; obtain a position information parameter corresponding to the drone through the drone platform system; determine a second detection range corresponding to the radar payload according to the second parameter; perform an equivalent angle conversion setting on the first parameter according to the position information parameter to determine a first detection range corresponding to the optoelectronic payload, and the first detection range and the second detection range are equal; determine a first image collected by the optoelectronic payload and a second image collected by the radar payload; determine a global situation map corresponding to the drone according to the first image and the second image, and the global situation map includes target features of a bird's-eye view.

[0130] Optionally, the processor 1110 is further used to determine a working mode of the radar payload according to a second parameter; obtain a second detection parameter corresponding to the working mode; and determine a second detection range corresponding to the radar payload according to the second detection parameter.

[0131] Optionally, the processor 1110 is also used to determine a navigation coordinate system centered on the UAV; determine a pitch angle parameter of the optoelectronic load in the navigation coordinate system based on the position information parameter and the first parameter; determine a first detection parameter of the optoelectronic load; determine a first horizontal field of view angle of the optoelectronic load based on the pitch angle parameter and the first detection parameter; and determine a first detection range corresponding to the optoelectronic load based on the first horizontal field of view angle.

[0132] Optionally, the processor 1110 is further used to determine a first acquisition parameter of the optoelectronic load in a first detection range; perform image acquisition according to the first acquisition parameter to determine a first image corresponding to the optoelectronic load; determine a second acquisition parameter of the radar load in a second detection range; perform image acquisition according to the second acquisition parameter to determine a second image corresponding to the radar load.

[0133] Optionally, processor 1110 is further used to determine a first feature point cloud corresponding to the first image; determine a second feature point cloud corresponding to the second image; fuse the first feature point cloud and the second feature point cloud based on a bird's-eye view encoder to determine a bird's-eye view feature; and determine a global situation map based on the bird's-eye view feature.

[0134] In a fifth aspect, the present invention provides a readable storage medium, on which a program or instruction is stored, which, when executed by a processor, implements each process of the embodiment of the above-mentioned load coordination method, and can achieve the same technical effect, which will not be described here to avoid repetition. In addition, the data storage capacity and data processing speed corresponding to the image registration method in this application are improved by a readable storage medium.

[0135] The methods may be implemented in a variety of different ways depending on the specific features and / or example applications. For example, the methods may be implemented by a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the above functions, and / or combinations thereof.

[0136] A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited thereto. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory card, floppy disk, encoding mechanical device (such as a punch card or a groove with a raised structure with instructions recorded) and any suitable combination of the above devices. The computer readable storage medium used herein should not be understood as a transmission signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium, or an electrical signal transmitted through a wire, etc.

[0137] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0138] In a sixth aspect, the present invention provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, the processor is used to run a program or instruction, implement each process of the embodiment of the above-mentioned load coordination method, and can achieve the same technical effect, to avoid repetition, it is not repeated here. In addition, the chip is used to improve the data processing speed corresponding to the image registration method in this application.

[0139] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the features may work as above in certain combinations and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0140] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0141] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0142] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0143] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A load coordination method, characterized in that: For a drone, the drone includes a drone platform system and a drone fuselage, the drone fuselage is provided with an optoelectronic load and a radar load, and the load coordination method includes: Acquire a first parameter of the optoelectronic payload and a second parameter of the radar payload; Acquire location information parameters corresponding to the drone through the drone platform system; determining a second detection range corresponding to the radar payload according to the second parameter; Performing an equivalent angle conversion setting on the first parameter according to the position information parameter to determine a first detection range corresponding to the photoelectric load, wherein the first detection range is equal to the second detection range; Determining a first image collected by the optoelectronic payload and a second image collected by the radar payload; A global situation map corresponding to the drone is determined based on the first image and the second image, wherein the global situation map includes a bird's-eye view target feature.

2. The load coordination method according to claim 1, characterized in that: The determining, according to the second parameter, a second detection range corresponding to the radar payload comprises: determining an operating mode of the radar payload according to the second parameter; Acquiring a second detection parameter corresponding to the working mode; A second detection range corresponding to the radar load is determined according to the second detection parameter.

3. The load coordination method according to claim 1, characterized in that: The performing an equivalent angle conversion setting on the first parameter according to the position information parameter to determine a first detection range corresponding to the photoelectric load includes: Determining a navigation coordinate system centered on the drone; Determine a pitch angle parameter of the optoelectronic payload in the navigation coordinate system according to the position information parameter and the first parameter; Determining a first detection parameter of the photoelectric load; Determine a first horizontal field of view angle of the photoelectric payload according to the pitch angle parameter and the first detection parameter; A first detection range corresponding to the photoelectric load is determined according to the first horizontal field of view angle.

4. The load coordination method according to claim 1, characterized in that: The determining of the first image collected by the optoelectronic payload and the second image collected by the radar payload includes: Determining a first acquisition parameter of the photoelectric load in the first detection range; Perform image acquisition according to the first acquisition parameter to determine a first image corresponding to the photoelectric load; determining a second acquisition parameter of the radar payload in the second detection range; Image acquisition is performed according to the second acquisition parameter to determine a second image corresponding to the radar payload.

5. The load coordination method according to claim 4, characterized in that: The determining of a global situation map corresponding to the UAV according to the first image and the second image comprises: determining a first feature point cloud corresponding to the first image; determining a second feature point cloud corresponding to the second image; fusing the first feature point cloud and the second feature point cloud based on a bird's-eye view encoder to determine a bird's-eye view feature; A global situation map is determined based on the bird's-eye view features.

6. A load coordination device, characterized in that: include: An acquisition module, used to acquire a first parameter of an optoelectronic payload and a second parameter of a radar payload; The UAV module obtains the position information parameters corresponding to the UAV through the UAV platform system; A determination module, determining a second detection range corresponding to the radar payload according to the second parameter; a range determination module, performing an equivalent angle conversion setting on the first parameter according to the position information parameter, and determining a first detection range corresponding to the photoelectric load; An acquisition module, determining a first image acquired by the optoelectronic payload and a second image acquired by the radar payload; A construction module is used to determine a global situation map corresponding to the UAV based on the first image and the second image.

7. A drone, characterized in that: include: An unmanned aerial vehicle platform system and an unmanned aerial vehicle fuselage, wherein the unmanned aerial vehicle fuselage is provided with an optoelectronic payload and a radar payload; A load coordination device as claimed in claim 6.

8. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the load coordination method as described in any one of claims 1 to 5.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the load coordination method according to any one of claims 1 to 5 are implemented.

10. A chip, characterized in that: The chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the load coordination method as described in any one of claims 1 to 5.

Images