A method and system for detecting targets with large field of view by unmanned aerial vehicles by combining SAR, visible light sensor and infrared sensor

Abstract

The present invention discloses a large-field-of-view target detection method for an unmanned aerial vehicle (UAV) that combines SAR, a visible light sensor, and an infrared sensor, comprising the following steps: S1, obtaining SAR imaging of a target area to obtain a first image set; obtaining a second image set based on the first image set; S2, obtaining a first data set based on the second image set; the first data set includes target longitude and latitude information; S3, locking a target with a visible light sensor and an infrared sensor based on the first data set, performing visible light imaging and infrared imaging, and obtaining a third image set and a fourth image set respectively; S4, obtaining a fifth image set based on the third image set; obtaining a sixth image set based on the fourth image set; S5, fusing the fifth image set and the sixth image set based on the first condition to obtain a seventh image set, and completing target detection. The present invention also provides a large-field-of-view target detection system for an unmanned aerial vehicle (UAV) that combines SAR, a visible light sensor, and an infrared sensor.

Description

Technical Field

[0001] The present invention relates to the technical field of large-field-of-view target detection for unmanned aerial vehicles, and in particular to a large-field-of-view target detection method and system for unmanned aerial vehicles combining SAR, visible light sensors and infrared sensors. Background Art

[0002] UAV technology has been widely used in both civil and military fields. In the military field, UAVs can perform many tasks such as military target reconnaissance, surveillance and attack, battlefield situation awareness and combat effectiveness evaluation; in the civil field, UAVs have been widely used in outdoor aerial photography, logistics transportation, smart transportation, camp monitoring, field search and rescue, geological disaster rescue and other tasks. The sensors currently carried by UAVs mainly include visible light sensors and infrared sensors. In recent years, lightweight SAR (Synthetic Aperture Radar) sensors have gradually been applied to the UAV field, further expanding the application scope of UAV technology.

[0003] SAR imaging is not affected by adverse weather such as dense fog, rain and snow, can penetrate clouds, and is not afraid of complex scenes such as dense smoke; SAR imaging can image a large range and field of view, which is suitable for large-scale regional target search and detection, but its image is not intuitive enough and it is difficult to directly complete image interpretation. The optoelectronic pod in the drone is often equipped with visible light sensors and infrared sensors. The optoelectronic pod of the drone has low efficiency in searching the target area of ​​a large-scale scene, and it is difficult to efficiently complete the reconnaissance of targets in a large field of view. In addition, the visible light image is easily disturbed by dense smoke, dense fog and thick clouds, and cannot be clearly imaged. The infrared image is easily affected by interference such as fire. The combined SAR, visible light sensor and infrared sensor can make full use of the advantages of different modes and is suitable for efficient detection of targets in a large field of view. In the prior art, the registration between SAR, visible light and infrared images is first completed, and then the image pixel level, feature level and decision level fusion are performed on the registered SAR, visible light and infrared images to achieve target detection of SAR, visible light and infrared images. However, the SAR and visible light images obtained by drones are quite different and cannot be automatically registered. Feature points need to be manually selected for registration, which consumes a lot of manpower and material resources and is inefficient. They cannot be deployed to the edge computing end of the drone for real-time completion and do not fully utilize the feature advantages of different sensors. They are not suitable for target detection tasks in large areas.

[0004] Therefore, a new technical solution is urgently needed to solve the technical problem of how to efficiently perform large-field-of-view target detection on UAVs. Summary of the invention

[0005] The present invention provides a method and system for detecting targets with a large field of view of an unmanned aerial vehicle (UAV) by combining SAR, a visible light sensor and an infrared sensor, so as to solve the technical problem of how to efficiently detect targets with a large field of view of an unmanned aerial vehicle (UAV).

[0006] To achieve the above object, the present invention provides a method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, a visible light sensor and an infrared sensor, comprising the following steps:

[0007] S1. Acquire SAR imaging of a target area to obtain a first image set; obtain a second image set based on the first image set; the second image set includes SAR images containing target annotations.

[0008] S2. Obtain a first data set according to the second image set; the first data set includes target latitude and longitude information.

[0009] S3. According to the first data set, the visible light sensor and the infrared sensor are used to lock the target, and visible light imaging and infrared imaging are performed to obtain a third image set and a fourth image set, respectively.

[0010] S4. Obtain a fifth image set based on the third image set; obtain a sixth image set based on the fourth image set; the fifth image set includes visible light images containing target annotations; and the sixth image set includes infrared images containing target annotations.

[0011] S5. According to the first condition, the fifth image set and the sixth image set are fused to obtain a seventh image set, thereby completing target detection; the seventh image set includes images with target annotations.

[0012] Preferably, obtaining the second image set according to the first image set includes:

[0013] The second image set is obtained according to the first image set combined with the first model; the first model includes a trained SAR image target detection model.

[0014] Obtaining the second image set according to the first image set and the first model includes:

[0015] Each image in the first image set is cropped into slice images of a preset size according to a preset first overlap degree, and the slice images of each image are grouped into the same group to obtain a first input.

[0016] Each group of slice images in the first input is sequentially input into the first model for inference; until all the images in the first input are input into the first model.

[0017] The first model obtains a slice image result set after inferring a single group of slice images, and splices the slice image results in the slice image result set according to a first overlap degree to obtain a model output result.

[0018] After the first model infers the first input, a second image set is obtained.

[0019] Preferably, it also includes:

[0020] The training process of the first model includes:

[0021] Build a SAR image target detection model.

[0022] SAR imaging of a random target area is acquired, and targets in the images are pre-labeled to obtain an eighth image set.

[0023] The eighth image set is divided according to a preset ratio to obtain a first training set and a first test set; and the images in the first training set are cropped into slice images of a preset size.

[0024] The SAR image target detection model is trained based on the first training set.

[0025] The SAR image target detection model is tested according to the first test set. If the test meets the preset conditions, the output is the trained SAR image target detection model, that is, the first model.

[0026] If the test does not meet the preset conditions, return to retrain the SAR image target detection model until the test meets the preset conditions.

[0027] Preferably, obtaining the first data set according to the second image set comprises:

[0028] The first data set is obtained according to the second image set combined with the second model; the second model includes SAR post-processed images containing longitude and latitude coordinates and a registration model for registering the second image set with the SAR post-processed images containing longitude and latitude coordinates.

[0029] Obtaining a first data set according to the second image set and the second model includes:

[0030] The second image set is registered with the SAR post-processed image containing longitude and latitude coordinates through the registration model to obtain the target longitude and latitude information, that is, the first data set.

[0031] Preferably, the registration model includes:

[0032] The first branch and the second branch;

[0033] The first branch is the feature point detection head branch: the input image is downsampled to obtain a size of The first feature map; extract the features of the first feature map; add a channel, the size of the first feature map becomes ; Detect the feature points of the first feature map and obtain a heat map of the feature points.

[0034] The second branch is the descriptor detection head branch: the input image is downsampled twice, and the first downsampling is The second feature map of size is obtained by downsampling for the second time The second feature map of size; Descriptor detection is performed on a second feature map of the same size; the size of the second feature map is restored to the size of the input image by upsampling twice; and the downsampling and upsampling are connected by a skip connection layer.

[0035] After the first branch and the second branch, the first branch result and the second branch result are added to obtain the final feature point matching result, that is, the result of the model used to align the second image set with the SAR post-processed image containing latitude and longitude coordinates.

[0036] Preferably, locking the visible light sensor and the infrared sensor to the target according to the first data set includes:

[0037] The optical image target positioning inverse solution is performed according to the first data set to obtain the pitch angle and yaw angle information required to adjust the visible light sensor and the infrared sensor to lock the target area, including:

[0038] ;

[0039] in, Respectively represent the target points longitude, latitude and altitude; Indicates the conversion from WGS-84 coordinate system to ECEF coordinate system; Indicates the conversion of the longitude and latitude coordinates of the UAV platform in the WGS-84 coordinate system to the coordinates in the ECEF coordinate system; Indicates the target point in the captured image The corresponding unit direction vector is converted to the unit direction vector in the ECEF coordinate system; the target point is known The vertical relative height between the camera and the value.

[0040] The visible light sensor and the infrared sensor are adjusted according to the pitch angle and the yaw angle information, and the visible light sensor and the infrared sensor are locked onto the target.

[0041] Preferably, obtaining the fifth image set according to the third image set includes:

[0042] A fifth image set is obtained based on the third image set combined with the third model; the third model includes a trained visible light image target detection model.

[0043] The training process of the third model includes:

[0044] Build a visible light image object detection model.

[0045] Obtain visible light imaging of a random target area, and pre-label the targets in the image to obtain a ninth image set.

[0046] The ninth image set is divided according to a preset ratio to obtain a second training set and a second test set; and the images in the second training set are cropped into slice images of a preset size.

[0047] The visible light image object detection model is trained based on the second training set.

[0048] The visible light image target detection model is tested according to the second test set. If the test meets the preset conditions, the output is the trained visible light image target detection model, that is, the third model.

[0049] If the test does not meet the preset conditions, the visible light image target detection model is retrained until the test meets the preset conditions.

[0050] Preferably, obtaining the sixth image set according to the fourth image set includes:

[0051] The sixth image set is obtained according to the fourth image set combined with the fourth model; the fourth model includes a trained infrared image target detection model.

[0052] The training process of the fourth model includes:

[0053] Build an infrared image target detection model.

[0054] Infrared imaging of a random target area is acquired, and the targets in the image are pre-labeled to obtain a tenth image set.

[0055] The tenth image set is divided according to a preset ratio to obtain a third training set and a third test set; and the images in the third training set are cropped into slice images of a preset size.

[0056] The infrared image target detection model is trained based on the third training set.

[0057] The infrared image target detection model is tested according to the third test set. If the test meets the preset conditions, the output is the trained infrared image target detection model, that is, the fourth model.

[0058] If the test does not meet the preset conditions, the infrared image target detection model is retrained until the test meets the preset conditions.

[0059] Preferably, fusing the fifth image set and the sixth image set according to the first condition includes:

[0060] The first condition includes the confidence probability formula, which can be expressed by the following relationship:

[0061] ;

[0062] in, and represents the weighting coefficient; represents the weight function; and Represents the categories of visible light and infrared image pairs respectively consensus and average support; Indicates that the fusion result is more inclined to or ;when When the visible light and infrared image fusion result is leading; when When leading.

[0063] The targets in the fifth image set and the sixth image set are fused by using a confidence probability formula to obtain a seventh image set including images with target annotations.

[0064] The present invention also provides a large-field-of-view target detection system for an unmanned aerial vehicle (UAV) that combines SAR, a visible light sensor, and an infrared sensor, which is used in the method of the present invention, and the system includes an unmanned aerial vehicle, a SAR, an optoelectronic pod, an edge computing platform, and an information transfer platform;

[0065] SAR, optoelectronic pod, edge computing platform and information transfer platform are all installed on the UAV; SAR and optoelectronic pod are both connected to the information transfer platform; the information transfer platform is connected to the edge computing platform; the optoelectronic pod includes visible light sensors and infrared sensors.

[0066] SAR is used to perform SAR imaging, obtain a first image set, and send the first image set to the edge computing platform through the information transfer platform.

[0067] The edge computing platform is used to obtain the second image set according to the first image set, obtain the first data set according to the second image set, and send information through the information transfer platform according to the first data set to control the visible light sensor and the infrared sensor to lock the target for visible light imaging and infrared imaging;

[0068] Visible light imaging and infrared imaging are used to lock the target for visible light imaging and infrared imaging, and obtain the third image set and the fourth image set respectively; the optoelectronic pod is used to send the third image set and the fourth image set to the edge computing platform through the information transfer platform.

[0069] The edge computing platform is also used to obtain a fifth image set based on the third image set, to obtain a sixth image set based on the fourth image set, and to fuse the fifth image set and the sixth image set based on the first condition to obtain a seventh image set.

[0070] The edge computing platform is also used to perform all data processing tasks in the system and store all mathematical models required for data processing in the system.

[0071] The information transfer platform is used to process information transmission between the edge computing platform and the optoelectronic pod, and is also used to process information transmission between the system and the outside world.

[0072] The present invention has the following beneficial effects:

[0073] The invention relates to a method for detecting targets with a large field of view of a drone that combines SAR, a visible light sensor, and an infrared sensor. It proposes to obtain the latitude and longitude coordinate position information of the target through SAR image detection, guide the optoelectronic pod to rotate to the target area, and complete the visible light and infrared image fusion recognition of the target after focusing on the target area. This method can improve the search efficiency of the drone for regional targets, efficiently realize the accurate recognition of targets with a large field of view, and increase the drone's ability to distinguish camouflaged targets. The method of the invention can combine SAR, a visible light sensor, and an infrared sensor to perform efficient operations for detecting targets with a large field of view of a drone.

[0074] The unmanned aerial vehicle large field target detection system of the present invention, which combines SAR, visible light sensor and infrared sensor, is used in the method of the present invention. Compared with the existing large-scene SAR, visible light and infrared image target detection devices, the system of the present invention does not need to manually extract features for different modal registration, can run in real time, has higher detection accuracy, can make full use of the advantages of different sensors such as SAR, visible light and infrared, and adapt to scenes with dense fog, dense smoke and fire interference; at the same time, the present invention has high operating efficiency, breaks through the practical application barriers of unmanned aerial vehicle SAR, visible light and infrared image joint large-scene target detection, and provides a practical large-field test application solution. The system of the present invention can combine SAR, visible light sensor and infrared sensor to perform efficient operation of unmanned aerial vehicle large-field target detection.

[0075] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0077] Figure 1 It is a schematic diagram of a method flow of a preferred embodiment of the present invention.

[0078] Figure 2 Schematic diagram of image slices according to a preferred embodiment of the present invention.

[0079] Figure 3 It is a schematic diagram of a SAR image with target annotation according to a preferred embodiment of the present invention.

[0080] Figure 4 It is a schematic diagram of the registration network model structure of a preferred embodiment of the present invention.

[0081] Figure 5 It is a schematic diagram of the longitude and latitude coordinates of the target obtained by alignment according to the preferred embodiment of the present invention.

[0082] Figure 6 It is a schematic diagram of a target area cropped out by focusing on a visible light image and a corresponding detection effect according to a preferred embodiment of the present invention.

[0083] Figure 7 It is a schematic diagram of a target area cropped out by focusing on an infrared image and a corresponding detection effect according to a preferred embodiment of the present invention.

[0084] Figure 8 It is a schematic diagram of large-field-of-view image target detection by fusing visible light and infrared images in a real scene in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0085] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0086] The experimental test carried out in this embodiment has a test scene area of ​​about 1,000,000 square meters, including ground and water scenes, with a ground range of 500 meters × 1,200 meters and a lake range of 500 meters × 800 meters. The SAR band used by the drone is the KU band, the frequency is between 12.5 and 18 GHz, and the working system is frequency modulated continuous wave. The visible light band of the optoelectronic pod is between 380 and 750 nm, and the obtained video frame rate is 30 Hz and the resolution is 1920 × 1080 pixels. The infrared band of the optoelectronic pod is between 8 and 14 um, and the obtained video frame rate is 30 Hz and the resolution is 640 × 480 pixels.

[0087] See also Figure 1 In a preferred embodiment of the present invention, a method for detecting a target with a large field of view of an unmanned aerial vehicle by combining SAR, a visible light sensor and an infrared sensor is provided, comprising the following steps:

[0088] S1. Acquire SAR imaging of a target area to obtain a first image set; obtain a second image set based on the first image set; the second image set includes SAR images containing target annotations.

[0089] In a preferred embodiment of the present invention, obtaining the second image set according to the first image set includes:

[0090] The second image set is obtained according to the first image set combined with the first model; the first model includes a trained SAR image target detection model.

[0091] Obtaining the second image set according to the first image set and the first model includes:

[0092] See also Figure 2 , each image in the first image set is cropped into slice images of a preset size according to a preset first overlap degree, and the slice images of each image are grouped into the same group to obtain a first input.

[0093] In a preferred embodiment of the present invention, the first overlap degree is 20%; the preset size is 256×256 pixels.

[0094] In a preferred embodiment of the present invention, cutting the slice image into a preset size according to a preset first overlap degree includes:

[0095] The image is sliced ​​at a preset size, and a first degree of overlap exists with a previous adjacent slice image during slicing.

[0096] Each group of slice images in the first input is sequentially input into the first model for inference; until all the images in the first input are input into the first model.

[0097] In a preferred embodiment of the present invention, a strategy of traversing sliced ​​full-size SAR images and putting all small-size images into a batch for simultaneous inference of SAR image detection can significantly improve the accuracy of target detection in model SAR images.

[0098] The first model obtains a slice image result set after inferring a single group of slice images, and splices the slice image results in the slice image result set according to a first overlap degree to obtain a model output result.

[0099] After the first model infers the first input, a second image set is obtained.

[0100] In a preferred embodiment of the present invention, the SAR images containing target annotations in the second image set refer to Figure 3 .

[0101] In a preferred embodiment of the present invention, the training process of the first model includes:

[0102] Build a SAR image target detection model.

[0103] SAR imaging of a random target area is acquired, and targets in the images are pre-labeled to obtain an eighth image set.

[0104] In a preferred embodiment of the present invention, the target in the pre-annotated image is annotated in the form of a rectangular frame.

[0105] The eighth image set is divided according to a preset ratio to obtain a first training set and a first test set; and the images in the first training set are cropped into slice images of a preset size.

[0106] The SAR image target detection model is trained based on the first training set.

[0107] The SAR image target detection model is tested according to the first test set. If the test meets the preset conditions, the output is the trained SAR image target detection model, that is, the first model.

[0108] If the test does not meet the preset conditions, return to retrain the SAR image target detection model until the test meets the preset conditions.

[0109] The present invention designs a random cropping of full-size SAR images to obtain a fixed-size training SAR image target detection network model, which better helps the network model to extract target features.

[0110] S2. Obtain a first data set according to the second image set; the first data set includes target latitude and longitude information.

[0111] In a preferred embodiment of the present invention, obtaining the first data set according to the second image set includes:

[0112] The first data set is obtained according to the second image set combined with the second model; the second model includes a SAR post-processed image containing longitude and latitude coordinates and a model for registering the second image set with the SAR post-processed image containing longitude and latitude coordinates.

[0113] In a preferred embodiment of the present invention, the longitude and latitude coordinate information of each pixel of the real-time image is obtained by aligning the SAR real-time image and the SAR post-processed image with known longitude and latitude coordinates through feature point registration, thereby obtaining the longitude and latitude coordinate information of the target.

[0114] Obtaining a first data set according to the second image set and the second model includes:

[0115] The second image set is registered with the SAR post-processed image containing longitude and latitude coordinates through the registration model to obtain the target longitude and latitude information, that is, the first data set.

[0116] See also Figure 4 In a preferred embodiment of the present invention, the model for registering the second image set with the SAR post-processed image containing latitude and longitude coordinates includes:

[0117] A first branch and a second branch.

[0118] The first branch is the feature point detection head branch: the input image is downsampled to obtain a size of The first feature map; extract the features of the first feature map; add a channel, the size of the first feature map becomes ; Detect the feature points of the first feature map and obtain a heat map of the feature points.

[0119] The second branch is the descriptor detection head branch: the input image is downsampled twice, and the first downsampling is The second feature map of size is obtained by downsampling for the second time The second feature map of size; Descriptor detection is performed on a second feature map of the same size; the size of the second feature map is restored to the size of the input image by upsampling twice; and the downsampling and upsampling are connected by a skip connection layer.

[0120] After the first branch and the second branch, the first branch result and the second branch result are added to obtain the final feature point matching result, that is, the result of the model used to align the second image set with the SAR post-processed image containing latitude and longitude coordinates.

[0121] In a preferred embodiment of the present invention, Figure 5 For experimental verification, the SAR real-time image and the SAR post-processed image containing longitude and latitude coordinates are registered to solve the longitude and latitude coordinate information of the target.

[0122] S3. According to the first data set, the visible light sensor and the infrared sensor are used to lock the target, and visible light imaging and infrared imaging are performed to obtain a third image set and a fourth image set, respectively.

[0123] In a preferred embodiment of the present invention, locking the visible light sensor and the infrared sensor to the target according to the first data set includes:

[0124] The optical image target positioning inverse solution is performed according to the first data set to obtain the pitch angle and yaw angle information required to adjust the visible light sensor and the infrared sensor to lock the target area, including:

[0125] ;

[0126] in, Respectively represent the target points longitude, latitude and altitude; Indicates the conversion from WGS-84 (World Geodetic System-1984 Coordinate System) to ECEF (Earth-Centered Earth-Fixed, ECEF) coordinate system; Indicates the conversion of the longitude and latitude coordinates of the UAV platform in the WGS-84 coordinate system to the coordinates in the ECEF coordinate system; Indicates the target point in the captured image The corresponding unit direction vector is converted to the unit direction vector in the ECEF coordinate system; the target point is known The vertical relative height between the camera and the value.

[0127] The visible light sensor and the infrared sensor are adjusted according to the pitch angle and the yaw angle information, and the visible light sensor and the infrared sensor are locked onto the target.

[0128] S4. Obtain a fifth image set based on the third image set; obtain a sixth image set based on the fourth image set; the fifth image set includes visible light images containing target annotations; and the sixth image set includes infrared images containing target annotations.

[0129] In a preferred embodiment of the present invention, obtaining the fifth image set according to the third image set includes:

[0130] A fifth image set is obtained based on the third image set combined with the third model; the third model includes a trained visible light image target detection model.

[0131] In a preferred embodiment of the present invention, the training process of the third model includes:

[0132] Build a visible light image object detection model.

[0133] Obtain visible light imaging of a random target area, and pre-label the targets in the image to obtain a ninth image set.

[0134] The ninth image set is divided according to a preset ratio to obtain a second training set and a second test set; and the images in the second training set are cropped into slice images of a preset size.

[0135] The visible light image object detection model is trained based on the second training set.

[0136] The visible light image target detection model is tested according to the second test set. If the test meets the preset conditions, the output is the trained visible light image target detection model, that is, the third model.

[0137] If the test does not meet the preset conditions, the visible light image target detection model is retrained until the test meets the preset conditions.

[0138] In a preferred embodiment of the present invention, obtaining the sixth image set according to the fourth image set includes:

[0139] The sixth image set is obtained according to the fourth image set combined with the fourth model; the fourth model includes a trained infrared image target detection model.

[0140] In a preferred embodiment of the present invention, the training process of the fourth model includes:

[0141] Build an infrared image target detection model.

[0142] Infrared imaging of a random target area is acquired, and the targets in the image are pre-labeled to obtain a tenth image set.

[0143] The tenth image set is divided according to a preset ratio to obtain a third training set and a third test set; and the images in the third training set are cropped into slice images of a preset size.

[0144] The infrared image target detection model is trained based on the third training set.

[0145] The infrared image target detection model is tested according to the third test set. If the test meets the preset conditions, the output is the trained infrared image target detection model, that is, the fourth model.

[0146] If the test does not meet the preset conditions, the infrared image target detection model is retrained until the test meets the preset conditions.

[0147] The preferred embodiment of the present invention uses fishing boats, simulated camouflaged tanks and simulated camouflaged combat vehicles as target types, and collects 10 segments of original visible light and infrared videos, with a total duration of about 360 seconds and a total of 10,800 frames of images. According to the target longitude and latitude coordinate information provided by the SAR image, the corresponding size of 256×256 visible light and infrared images containing the target area are cropped to construct a data set. The ratio of the visible light and infrared image training set to the test set is divided into 6:4. 6 segments of visible light and infrared video are used for training, and 4 segments are used for testing. The visible light and infrared video extracts video frames and saves them as images, and the specified target type in the image is marked with a rectangular frame.

[0148] In a preferred embodiment of the present invention, the visible light and infrared models are trained based on the training set to obtain optimized visible light and infrared network model weights. Figure 6 and Figure 7 When conducting experimental verification for this embodiment, according to the target longitude and latitude coordinate information provided by the SAR image, the corresponding size of 256×256 visible light and infrared images containing the target area are cropped, and the 256×256 small-size image blocks containing the target area are transmitted to the visible light and infrared image detection network models to obtain the detection results of the visible light and infrared images respectively. Figure 6 and Figure 7 The T in the figure indicates that the target area is located according to the target coordinate information of the SAR image.

[0149] S5. According to the first condition, the fifth image set and the sixth image set are fused to obtain a seventh image set, thereby completing target detection; the seventh image set includes images with target annotations.

[0150] In a preferred embodiment of the present invention, fusing the fifth image set and the sixth image set according to the first condition includes:

[0151] The first condition includes the confidence probability formula, which can be expressed by the following relationship:

[0152] ;

[0153] in, and represents the weighting coefficient; represents the weight function; and Respectively represent the consensus and average support of visible light and infrared images for category A; Indicates that the fusion result is more inclined to or ;when When the visible light and infrared image fusion result is leading; when When leading.

[0154] The targets in the fifth image set and the sixth image set are fused by using a confidence probability formula to obtain a seventh image set including images with target annotations.

[0155] In the preferred embodiment of the present invention, through the fusion of confidence probability formulas, the single-modal target missed detection or misdetection caused by the visible light mode being easily disturbed by thick fog and smoke and the infrared mode being disturbed by fire is alleviated, and ultimately large-field-of-view and efficient target detection is achieved.

[0156] In a preferred embodiment of the present invention, Figure 8 The final target detection result of the method is used for experimental verification. It can be seen that the method of the present invention can meet the needs of actual applications, overcome the scenario where a single sensor is easily interfered by external factors, and can achieve large-field-of-view target detection in complex scenarios by utilizing the advantages of different modalities of SAR, visible light and infrared images within the large field of view of the UAV, and has strong application value. Figure 8 The T in the figure indicates that the target area is located according to the target coordinate information of the SAR image.

[0157] The invention provides a large-field-of-view target detection method for unmanned aerial vehicles combining SAR, visible light sensors and infrared sensors, and proposes obtaining the latitude and longitude coordinate position information of the target through SAR image detection, guiding the optoelectronic pod to rotate to the target area, and completing the visible light and infrared image fusion identification of the target after focusing on the target area. This can improve the search efficiency of the unmanned aerial vehicle for regional targets, efficiently realize the accurate identification of large-field-of-view targets, and at the same time increase the unmanned aerial vehicle's ability to distinguish camouflaged targets.

[0158] In a preferred embodiment of the present invention, a UAV large field of view target detection system combining SAR, visible light sensor and infrared sensor is also provided, which is used in the method of the present invention, and the system includes a UAV, a SAR, an optoelectronic pod, an edge computing platform and an information transfer platform;

[0159] SAR, optoelectronic pod, edge computing platform and information transfer platform are all installed on the UAV; SAR and optoelectronic pod are both connected to the information transfer platform; the information transfer platform is connected to the edge computing platform; the optoelectronic pod includes visible light sensors and infrared sensors.

[0160] SAR is used to perform SAR imaging, obtain a first image set, and send the first image set to the edge computing platform through the information transfer platform.

[0161] The edge computing platform is used to obtain the second image set based on the first image set, obtain the first data set based on the second image set, and send information through the information transfer platform based on the first data set to control the visible light sensor and the infrared sensor to lock the target for visible light imaging and infrared imaging.

[0162] Visible light imaging and infrared imaging are used to lock the target for visible light imaging and infrared imaging, and obtain the third image set and the fourth image set respectively; the optoelectronic pod is used to send the third image set and the fourth image set to the edge computing platform through the information transfer platform.

[0163] The edge computing platform is also used to obtain a fifth image set based on the third image set, to obtain a sixth image set based on the fourth image set, and to fuse the fifth image set and the sixth image set based on the first condition to obtain a seventh image set.

[0164] The edge computing platform is also used to perform all data processing tasks in the system and store all mathematical models required for data processing in the system.

[0165] The information transfer platform is used to process information transmission between the edge computing platform and the optoelectronic pod, and is also used to process information transmission between the system and the outside world, such as transmitting information to and from the ground.

[0166] The system of the present invention is used for the method of the present invention, and the unmanned aerial vehicle large-field-of-view target detection system combining SAR, visible light sensor and infrared sensor of the present invention is used for the method of the present invention. Compared with the existing large-scene SAR, visible light and infrared image target detection devices, the system of the present invention does not need to manually extract features for different modal alignment, can run in real time, has higher detection accuracy, can make full use of the advantages of different sensors such as SAR, visible light and infrared, and adapt to scenes with thick fog, thick smoke and fire interference; at the same time, the present invention has high operating efficiency, breaks through the practical application barrier of unmanned aerial vehicle SAR, visible light and infrared image combination for large-scene target detection, and provides a solution for practical large-field-of-view test application.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting targets with a large field of view on an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor, characterized in that: The following steps are involved: S1, acquiring SAR imaging of a target area to obtain a first image set; obtaining a second image set based on the first image set; the second image set includes SAR images containing target annotations; S2. Obtain a first data set according to the second image set; the first data set includes target latitude and longitude information; S3, locking the target with the visible light sensor and the infrared sensor according to the first data set, performing visible light imaging and infrared imaging, and obtaining a third image set and a fourth image set respectively; S4, obtaining a fifth image set according to the third image set; obtaining a sixth image set according to the fourth image set; the fifth image set includes visible light images containing target annotations; the sixth image set includes infrared images containing target annotations; S5. According to the first condition, the fifth image set and the sixth image set are fused to obtain a seventh image set, thereby completing target detection; the seventh image set includes images with target annotations; The first condition includes a confidence probability formula, which can be expressed by the following relationship: ； in, and represents the weighting coefficient; represents the weight function; and Respectively represent the consensus and average support of visible light and infrared images for category A; Indicates that the fusion result is more inclined to or ;when When the visible light and infrared image fusion result is leading; when When leading.

2. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 1 is characterized in that: Obtaining a second image set according to the first image set comprises: A second image set is obtained based on the first image set combined with a first model; the first model includes a trained SAR image target detection model; The obtaining of the second image set according to the first image set and the first model comprises: Cutting each image in the first image set into slice images of a preset size according to a preset first overlap, and grouping the slice images of each image into the same group to obtain a first input; Inputting each group of slice images in the first input into the first model in sequence for inference; until all images in the first input are input into the first model; The first model obtains a slice image result set after reasoning a single group of slice images, and splices the slice image results in the slice image result set according to the first overlap degree to obtain a model output result; After the first model infers the first input, the second image set is obtained.

3. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 2 is characterized in that: Also includes: The training process of the first model includes: Construct a SAR image target detection model; Acquire SAR images of random target areas and pre-label targets in the images to obtain an eighth image set; Dividing the eighth image set according to a preset ratio to obtain a first training set and a first test set; and cropping the images in the first training set into slice images of a preset size; Training the SAR image target detection model according to the first training set; Testing the SAR image target detection model according to the first test set, and if the test meets the preset conditions, outputting the trained SAR image target detection model, that is, the first model; If the test does not meet the preset conditions, then return to retrain the SAR image target detection model until the test meets the preset conditions.

4. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 3 is characterized in that: Obtaining a first data set according to the second image set comprises: A first data set is obtained according to the second image set in combination with a second model; the second model includes a SAR post-processed image including longitude and latitude coordinates and a registration model for registering the second image set with the SAR post-processed image including longitude and latitude coordinates; Obtaining a first data set according to the second image set in combination with the second model includes: The second image set is registered with the SAR post-processed image containing longitude and latitude coordinates through the registration model to obtain target longitude and latitude information, that is, the first data set.

5. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 4 is characterized in that: The registration model includes: The first branch and the second branch; The first branch is the feature point detection head branch: the input image is downsampled to obtain a size of The first feature map; extract the features of the first feature map; add a channel, the size of the first feature map becomes ; Detecting feature points of the first feature map to obtain a heat map of the feature points; The second branch is the descriptor detection head branch: the input image is downsampled twice, and the first downsampling is The second feature map of size is obtained by downsampling for the second time The second feature map of size; Descriptor detection is performed on a second feature map of the size; the size of the second feature map is restored to the size of the input image by upsampling twice; downsampling and upsampling are connected by a skip connection layer; After the first branch and the second branch, the first branch result and the second branch result are added to obtain the final feature point matching result, that is, the result of the model for registering the second image set with the SAR post-processed image containing latitude and longitude coordinates.

6. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 5 is characterized in that: Locking the visible light sensor and the infrared sensor on the target according to the first data set includes: The optical image target positioning inverse solution is performed according to the first data set to obtain the pitch angle and yaw angle information required to adjust the visible light sensor and the infrared sensor to lock the target area, including: ； in, Respectively represent the target points longitude, latitude and altitude; Indicates the conversion from WGS-84 coordinate system to ECEF coordinate system; Indicates the conversion of the longitude and latitude coordinates of the UAV platform in the WGS-84 coordinate system to the coordinates in the ECEF coordinate system; Indicates the target point in the captured image The corresponding unit direction vector is converted to the unit direction vector in the ECEF coordinate system; the target point is known The vertical relative height between the camera and the value; The visible light sensor and the infrared sensor are adjusted according to the pitch angle and the yaw angle information, and the visible light sensor and the infrared sensor are locked onto a target.

7. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 6 is characterized in that: Obtaining a fifth image set according to the third image set comprises: A fifth image set is obtained according to the third image set combined with the third model; the third model includes a trained visible light image target detection model; The training process of the third model includes: Build a visible light image target detection model; Obtain visible light imaging of a random target area, and pre-label the targets in the image to obtain a ninth image set; Dividing the ninth image set according to a preset ratio to obtain a second training set and a second test set; and cropping the images in the second training set into slice images of a preset size; Training the visible light image object detection model according to the second training set; Testing the visible light image target detection model according to the second test set, and if the test meets the preset conditions, outputting the trained visible light image target detection model, that is, the third model; If the test does not meet the preset conditions, the visible light image target detection model is retrained until the test meets the preset conditions.

8. The method for detecting targets with a large field of view of an unmanned aerial vehicle by combining SAR, visible light sensor and infrared sensor according to claim 7 is characterized in that: Obtaining a sixth image set according to the fourth image set comprises: A sixth image set is obtained according to the fourth image set combined with the fourth model; the fourth model includes a trained infrared image target detection model; The training process of the fourth model includes: Build infrared image target detection model; Acquire infrared images of random target areas and pre-label the targets in the images to obtain a tenth image set; Dividing the tenth image set according to a preset ratio to obtain a third training set and a third test set; and cutting the images in the third training set into slice images of a preset size; Training the infrared image target detection model according to the third training set; Testing the infrared image target detection model according to the third test set, and if the test meets the preset conditions, outputting the trained infrared image target detection model, i.e., the fourth model; If the test does not meet the preset conditions, the infrared image target detection model is retrained until the test meets the preset conditions.

9. A large-field-of-view target detection system for unmanned aerial vehicles combining SAR, visible light sensor and infrared sensor, used in the method according to any one of claims 1 to 8, characterized in that: The system includes a drone, a SAR, an optoelectronic pod, an edge computing platform, and an information transfer platform; The SAR, optoelectronic pod, edge computing platform and information transfer platform are all installed on the UAV; the SAR and optoelectronic pod are both connected to the information transfer platform; the information transfer platform is connected to the edge computing platform; the optoelectronic pod includes a visible light sensor and an infrared sensor; The SAR is used to perform SAR imaging to obtain a first image set, and send the first image set to the edge computing platform through the information transfer platform; The edge computing platform is used to obtain a second image set according to the first image set, obtain a first data set according to the second image set, and send information through the information transfer platform according to the first data set to control the visible light sensor and the infrared sensor to lock the target for visible light imaging and infrared imaging; The visible light imaging and infrared imaging are used to lock the target for visible light imaging and infrared imaging, and obtain a third image set and a fourth image set respectively; the optoelectronic pod is used to send the third image set and the fourth image set to the edge computing platform through the information transfer platform; The edge computing platform is further used to obtain a fifth image set according to the third image set, obtain a sixth image set according to the fourth image set, and fuse the fifth image set and the sixth image set according to the first condition to obtain a seventh image set; The edge computing platform is also used to perform all data processing tasks in the system and store all mathematical models required for data processing in the system; The information transfer platform is used to process information transmission between the edge computing platform and the optoelectronic pod, and is also used to process information transmission between the system and the outside world.

Images