Target multi-dimensional information acquisition and fusion processing system and method based on unmanned aerial vehicle

Abstract

The invention provides an unmanned aerial vehicle-based target multi-dimensional information acquisition and fusion processing system and method, and the system comprises an unmanned aerial vehicle platform which comprises a photoelectric configuration unmanned aerial vehicle, an oblique photography configuration unmanned aerial vehicle and a hyperspectral configuration unmanned aerial vehicle and is used for completing the acquisition of target multi-dimensional data; and the ground command and control platform comprises an integrated command and control unit, an information fusion processing unit, a measurement and control link unit and a comprehensive information processing unit, and the ground command and control platform realizes integrated route planning, command and control, load information processing, information fusion and information generation of the multi-unmanned aerial vehicle platform.

Description

Technical Field

[0001] This application belongs to the technical field of UAV command and control and intelligence fusion, and particularly relates to a target multi-dimensional information acquisition, fusion and processing system and method based on UAVs. Background Art

[0002] For a long time, the target information acquisition technology has been limited by the load-carrying platform, and it is unable to effectively collect information on targets in complex terrain areas. With the rapid development of UAV technology, multi-rotor platforms with characteristics such as convenient takeoff and landing, autonomous cruising, and fixed-point hovering have gradually become the preferred load-carrying platforms for mission payloads.

[0003] The target information acquisition system based on the multi-rotor UAV platform can overcome the influence of many adverse factors such as time and space, and has advantages such as economy, randomness, and concealment. In addition, by combining data fusion technology, the target information acquisition system based on the rotor UAV platform can perform relevant processing such as preprocessing, association, and integration on multi-source data information from different sensors, provide more valuable information, and assist in decision-making, and has been widely applied in fields such as pipeline patrol and anti-terrorism reconnaissance.

[0004] However, most of the current target information acquisition systems based on the rotor UAV platform only support controlling a single UAV platform to collect target information, and have disadvantages such as a small survey range, a single mission payload, a high collection time cost, poor target accuracy, and a low dimension of target information, and are unable to effectively meet the increasingly complex target information acquisition requirements. Summary of the Invention

[0005] Objective of the present invention: Due to the influence of complex terrain and target camouflage, higher requirements are put forward for the acquisition and recognition of target information. This patent hopes to solve the problem of difficult data collection in complex areas by researching the use of multiple rotor UAV platforms carrying mission payloads to achieve rapid acquisition of target data; based on the integrated command and control and joint mission planning technology for multi-UAV collaborative detection, solve the problem of collaborative work between multiple UAVs; adopt automated three-dimensional reconstruction, data fusion processing technology, and automated intelligence generation technology to achieve comprehensive analysis and fusion of target information, and provide more perfect decision-making support information for decision-makers.

[0006] In a first aspect, the present application provides a target multi-dimensional information acquisition, fusion and processing system based on UAVs, and the system includes:

[0007] A UAV platform, including an optoelectronic configuration UAV, an oblique photography configuration UAV, and a hyperspectral configuration UAV, and the UAV platform completes the acquisition of target multi-dimensional data;

[0008] The ground command and control platform includes an integrated command and control unit, an intelligence fusion processing unit, a measurement and control link unit, and a comprehensive information processing unit. The ground command and control platform realizes integrated route planning, command and control of multiple unmanned aerial vehicle (UAV) platforms, as well as payload information processing, information fusion, and intelligence generation.

[0009] Preferably, the UAV platform selects a multi-rotor UAV platform, which is equipped with an optoelectronic reconnaissance payload, an oblique photography payload, and a hyperspectral payload to complete the acquisition of target multi-dimensional data, providing target images, 3D modeling data, and spectral data for data fusion processing.

[0010] Preferably, the integrated command and control unit adopts a standardized control seat, and internally deploys a mission planning module and a command and control module;

[0011] Among them, the mission planning module realizes joint mission planning for multiple UAVs, ensuring effective coordination between UAV platforms to complete the acquisition of reconnaissance data in the target area;

[0012] The command and control module is based on integrated command and control technology and adopts a general standardized operation interface to realize integrated control of the UAV platform.

[0013] Preferably, the UAV platform accesses the remote control and telemetry through the measurement and control link unit to the ground command and control platform;

[0014] The comprehensive information processing unit processes the remote control and telemetry information, assisting the integrated command and control unit to realize multi-aircraft joint monitoring and information comprehensive processing throughout the data acquisition process.

[0015] Preferably, the intelligence fusion processing unit internally deploys an optoelectronic reconnaissance module, a 3D modeling module, a spectral analysis module, a data fusion module, and an intelligence generation module.

[0016] Preferably, the optoelectronic reconnaissance module realizes the extraction of target features from optoelectronic reconnaissance data;

[0017] The 3D modeling module completes the 3D reconstruction of the target from oblique photography data;

[0018] The spectral analysis module realizes the recognition of ground objects in the target area, effectively identifying target camouflage;

[0019] The data fusion module fuses the target processing data of the optoelectronic reconnaissance module, the 3D modeling module, and the spectral analysis module to complete the accurate recognition of targets in the area;

[0020] The intelligence generation module completes the automatic generation of reconnaissance intelligence.

[0021] In a second aspect, the present application also provides a method for collecting, fusing, and processing multi-dimensional information of a target based on an unmanned aerial vehicle. The method includes:

[0022] The integrated command and control unit completes the joint mission planning of multiple unmanned aerial vehicles according to the reconnaissance requirements and formulates the flight routes of the unmanned aerial vehicles.

[0023] The unmanned aerial vehicle platform carries the mission payload and executes the multi-dimensional information collection task of the target according to the preset route.

[0024] During the process of collecting multi-dimensional information of the unmanned aerial vehicle target, the unmanned aerial vehicle platform accesses the remote control and telemetry to the ground command and control platform through the measurement and control link unit, and the comprehensive information processing unit processes the remote control and telemetry information to assist the integrated command and control unit to realize the multi-aircraft joint monitoring and information comprehensive processing of the whole process of data collection.

[0025] After the data collection is completed, the unmanned aerial vehicle returns and completes the data unloading, and the intelligence fusion processing unit carries out the target information processing, data fusion, and intelligence generation work.

[0026] Preferably, the intelligence fusion processing unit carries out the target information processing, data fusion, and intelligence generation work, including:

[0027] The optoelectronic reconnaissance module of the intelligence fusion processing unit completes the automatic recognition and extraction of the target from the optoelectronic reconnaissance data, and obtains the target image and position information.

[0028] The 3D modeling processing unit of the intelligence fusion processing unit completes the 3D modeling of the target area based on the regional oblique photography reconnaissance target, and repairs the 3D model of the target based on the optoelectronic reconnaissance image to improve the accuracy of the regional 3D, and combines the autonomous GIS engine to generate the regional 3D map data.

[0029] The spectral analysis module of the intelligence fusion processing unit analyzes the target spectral data, completes the spectral analysis of the target area, and completes the camouflage analysis and target differentiation.

[0030] The data fusion module of the intelligence fusion processing unit realizes the fusion processing of multi-dimensional target information and realizes the accurate identification and analysis of the target within the area.

[0031] The intelligence generation module of the intelligence fusion processing unit automatically completes the generation of the target reconnaissance intelligence.

[0032] Advantageous technical effects of the present invention:

[0033] (1) Based on multi-aircraft joint mission planning, it solves the different mission requirements brought by different mission payloads to UAV mission planning due to different working mechanisms, acquisition methods, and working modes. It can support 3 or more types of mission payloads, and the joint reconnaissance of no less than 10 UAV platforms, reducing the complexity and difficulty of mission planning. The autonomous mission planning time is no more than 5 seconds, ensuring the orderly operation among UAV platforms and improving the detection efficiency;

[0034] (2) Through the fusion processing of visible light / infrared, oblique photography, and hyperspectral data, on the basis of realizing the camouflage analysis and accurate identification of the target, it synchronously provides the terrain and geomorphic information of the target and its surrounding areas, and can provide a more comprehensive information reference;

[0035] (3) Based on the automated intelligence generation of a unified intelligence template, it can organically integrate data such as the images, spectra, and models of the target according to the fusion results, conduct knowledge reasoning, and automatically generate intelligence products according to the intelligence template, providing effective support for the subsequent decision-making of decision-makers. Description of the Drawings

[0036] Figure 1 Schematic diagram of a target multi-dimensional information acquisition and fusion processing system based on UAVs provided by an embodiment of the present application;

[0037] Figure 2 Working principle diagram of a target multi-dimensional information acquisition and fusion processing system based on UAVs provided by an embodiment of the present application;

[0038] Figure 3 Schematic diagram of the external shape of an integrated command and control unit provided by an embodiment of the present application;

[0039] Figure 4 Technical principle diagram of a target multi-dimensional information acquisition and fusion processing system based on UAVs provided by an embodiment of the present application;

[0040] Figure 5 Working flow chart of a target multi-dimensional information acquisition and fusion processing system provided by an embodiment of the present application. Detailed Implementation Modes

[0041] In view of the many shortcomings of the current target information acquisition system, an integrated command and control unit is designed for the target multi-dimensional information acquisition and fusion processing system based on unmanned aerial vehicles (UAVs). Through joint mission planning in the multi-aircraft and multi-payload working mode, it supports different mission payloads on multiple UAV platforms to perform multi-dimensional acquisition tasks of target information. Traditional target data reconnaissance focuses on the discovery and tracking of targets, mainly completing target recognition and tracking based on target image information, infrared information, or radar information. This system is based on target image information and combines technologies such as three-dimensional reconstruction and spectral analysis to achieve the fusion of target images, models, and spectral analysis. While completing target tracking, it realizes terrain reconstruction of the surrounding area of the target, completes target spectral recognition, and provides more abundant and complete information for decision-makers while achieving accurate target recognition, which helps commanders make more accurate decisions.

[0042] The information acquisition system of this application is constructed by carrying mission payloads on multiple rotor UAV platforms, realizing rapid information acquisition in complex areas. Based on the multi-dimensional information fusion processing of the data collected by UAVs, it realizes accurate target recognition in the target area and supports rapid data acquisition.

[0043] This application has the following main technical points:

[0044] (1) Joint mission planning for multi-platform collaborative detection:

[0045] Since the functions of the optoelectronic, oblique photography, and hyperspectral payloads carried by UAV platforms are different, each payload has different requirements for the flight route and working mode of the UAV when completing tasks normally. Therefore, issues such as cooperation, joint mission planning, route planning, and collision avoidance among multiple UAVs need to be considered. This system conducts joint mission planning design for multi-platform system collaborative detection to solve these problems.

[0046] Joint mission planning includes mission initialization, payload initialization, mission scheduling, and mission allocation strategies. Among them, mission initialization is to set the position and status of mission points, payload initialization includes initializing the status parameters of image payloads, and mission scheduling uses the method of average distribution scheduling to enable the same type of tasks at each mission point to be completed within the same time period.

[0047] After mission planning, the UAV can execute tasks according to the planned task sequence. However, the battlefield environment is dynamic. Due to reasons such as situation changes, sudden threats, or new tasks, the pre-set task sequence may become unusable. To avoid mission failure caused by this situation, at this time, it is necessary to implement a task allocation strategy according to the changes in the current mission space, and use the contract net algorithm for UAV dynamic mission planning to achieve the goal of UAVs collaborating to complete tasks. The core idea of the contract net protocol is to achieve negotiation and cooperation among agents by simulating the contract transaction process in the market mechanism, and to achieve global optimal resource allocation based on the pursuit of local optimality. The dynamic mission planning of UAVs is to improve the ability of the entire system to adapt to battlefield situation changes. The application of the contract net algorithm in specific problems is mainly reflected in the auction rules, that is, the dynamic mission planning principle. Applying the contract net algorithm to the dynamic mission planning of UAVs needs to meet the principles of the shortest time, local optimization, and balance.

[0048] (2) Synchronization of acquisition task status information based on improved consensus protocol:

[0049] During the process of multi-aircraft joint execution of target information acquisition, it is necessary to keep the status information of multiple UAV platforms unified, such as timing information, target location information, task information, etc. How to effectively achieve the unification of status information between multiple UAV platforms will directly affect the result of multi-platform intelligence fusion.

[0050] This system distributes a unified time reference signal to each UAV platform through the ground command and control system to keep unified with each UAV in terms of time reference; realizes the alignment of reconnaissance data time through extrapolation and interpolation techniques; this system is based on an autonomous GIS engine, sets a standard coordinate system through the ground command and control system, and ensures the spatial unity of the system through coordinate transformation; in addition, this system realizes the unification of task information of each unmanned platform during flight through an improved consensus protocol algorithm and a wireless communication delay simulation calculation model.

[0051] (3) Multi-platform data adaptive nearest neighbor association based on human-in-the-loop:

[0052] Data association is the basis of multi-platform data fusion, and the correctness of association directly determines the judgment of the number and attributes of battlefield targets. Target changes caused by missed association and mis-association will have an adverse impact on data fusion products.

[0053] This system adopts a multi-platform data adaptive nearest neighbor association algorithm based on human-in-the-loop. The traditional nearest neighbor algorithm belongs to unsupervised learning algorithm, which cannot effectively apply existing sample information for model training, and the neighborhood range is fixed, and the data similarity threshold cannot be effectively adjusted according to the problem scenario. Therefore, on the basis of automatic algorithm association, this system introduces a human-in-the-loop machine learning method. Through a simple human-computer interaction operation interface, the operator is involved in the fusion processing process to mark and adjust the association model, and combined with machine intelligence, the association accuracy rate is guaranteed.

[0054] (4) Terrain reconstruction based on fully automatic three-dimensional real scene modeling

[0055] The system adopts advanced technologies such as photogrammetry, computer vision, and artificial intelligence, and creatively solves key technologies such as large-scale oblique photography area network adjustment, dense matching, surface reconstruction and optimization, seamless texture mapping, and image compensation, realizing the comprehensive reconstruction of the terrain in the reconnaissance area and providing comprehensive topographic and geomorphic information support for command and decision-making.

[0056] Large-scale oblique image aerotriangulation technology for ultra-large scenes: supports CPU+GPU collaborative massive image tie point matching, supports aerotriangulation solution for more than one million images, supports parallel solution of multi-node aerotriangulation area network, and supports solutions with or without POS and with or without control points.

[0057] Dense point cloud matching technology for large scenes integrating multiple methods and strategies: integrates multiple gross error rejection algorithms such as epipolar constraint and weighted iteration; pixel-by-pixel matching, supports point cloud output;

[0058] Hierarchical detail construction of three-dimensional model data in ultra-large range: When performing large-scale LOG construction, data recursive dissection, grid simplification and terrain crack elimination technologies are adopted, and the pyramid+quadtree strategy is used to cut into multi-level fragment files, seamlessly establish LOD node data, and realize real-time browsing and network publishing;

[0059] (5) Reconnaissance intelligence generation based on knowledge graph:

[0060] To comprehensively reflect the target information collected by the UAV system, the system combines expert knowledge to model the relationship between target entities and attribute information, combines structured data and unstructured data, and establishes a target information knowledge graph based on a graph database. Through the intelligence generation module, different dimensional information such as the optoelectronic information, three-dimensional reconstruction model, and spectral information of the target can be used for graph-based knowledge reasoning according to the recognition results of fusion processing, and reconnaissance intelligence can be automatically generated to provide basic intelligence support for subsequent decision-making. At the same time, the current new target multi-dimensional information can be used as input to add the current target information to the original knowledge graph for incremental update, promoting the intelligent iteration of the knowledge graph.

[0061] Please refer toFigure 1 - Figure 5 , the basic composition of the target multi-dimensional information acquisition and fusion processing system designed in this application is as Figure 1 shown, mainly including the UAV platform and the ground command and control platform. The UAV platform includes three types of UAV configurations, namely, optoelectronic configuration UAVs, oblique photography configuration UAVs, and hyperspectral configuration UAVs, to achieve the acquisition of target information.

[0062] The ground command and control platform includes an integrated command and control unit, an intelligence fusion processing unit, a measurement and control link unit, and a comprehensive information processing unit, etc., to achieve the integrated route planning, command and control of multiple UAV platforms, as well as the processing of payload information, information fusion, intelligence generation, etc. The internal cross-linking of the target multi-dimensional information acquisition and fusion processing system based on UAVs is as Figure 2 shown. The design of the main units is as follows:

[0063] (1) UAV platform: The UAV platform selects a multi-rotor UAV platform, equipped with optoelectronic reconnaissance payloads, oblique photography payloads, and hyperspectral payloads to complete the acquisition of target multi-dimensional data, providing target images, 3D modeling data, and spectral data, etc. for data fusion processing, and solving the acquisition of target data;

[0064] (2) The integrated command and control unit adopts a standardized control seat, referring to the second-generation naval standard console, with good man-machine efficiency. The schematic diagram is as Figure 3 shown; The core software such as the mission planning module and the command and control module are deployed internally; The mission planning module realizes the joint mission planning of multiple UAVs, ensuring the effective coordination between UAV platforms and completing the acquisition of reconnaissance data in the target area; The command and control module is based on the integrated command and control technology and adopts a general standardized operation interface to realize the integrated control of the UAV platform;

[0065] (3) The hardware design of the intelligence fusion processing unit is the same as that of the integrated command and control unit. The optoelectronic reconnaissance module, 3D modeling module, spectral analysis module, data fusion module, and intelligence generation module are deployed internally; The optoelectronic reconnaissance module realizes the extraction of target features from optoelectronic reconnaissance data, completing target acquisition and payload; The 3D modeling module completes the 3D reconstruction of the target from oblique photography data; The spectral analysis module realizes the recognition of ground objects in the target area, effectively identifying target camouflage; The data fusion module fuses the target processing data of the optoelectronic reconnaissance module, 3D modeling module, and spectral analysis module to complete the accurate recognition of targets in the area, and the intelligence generation module completes the automatic generation of reconnaissance intelligence.

[0066] The technical principle and working principle diagram of the target multi-dimensional information acquisition and fusion processing system based on UAVs are as Figure 4 、 Figure 5 shown. The main working process of the system is as follows:

[0067] (1) The integrated command and control unit completes the joint mission planning of multiple UAVs according to the reconnaissance needs and formulates the UAV flight routes.

[0068] (2) The UAV platform carrying the mission payload executes the multi-dimensional information acquisition mission of the target according to the pre-set route.

[0069] (3) During the multi-dimensional information acquisition of the UAV target, the UAV platform accesses the remote control and telemetry to the ground command and control platform through the measurement and control link unit. The comprehensive information processing unit processes the remote control and telemetry information, assisting the integrated command and control unit to realize the multi-aircraft joint monitoring and information comprehensive processing of the whole process of data acquisition.

[0070] (4) After the data acquisition is completed, the UAV returns and completes the data unloading. The intelligence fusion processing unit conducts the target information processing, data fusion and intelligence generation work. The main processes are as follows:

[0071] a) The optoelectronic reconnaissance module of the intelligence fusion processing unit trains the BP neural network according to the historical information, and completes the automatic recognition and extraction of the target from the optoelectronic reconnaissance data through the BP neural network to obtain the target image and position information, etc.

[0072] b) The 3D modeling processing unit of the intelligence fusion processing unit completes the 3D modeling of the target area based on the regional oblique photography reconnaissance target, and uses the support vector machine method and various features of the reconnaissance target to effectively correct the edge of the target area. And based on the optoelectronic reconnaissance image, the 3D model of the target is modeled and repaired to improve the accuracy of the regional 3D, and combined with the autonomous GIS engine to generate the regional 3D map data.

[0073] c) The spectral analysis module of the intelligence fusion processing unit analyzes the target spectral data, completes the spectral analysis of the target area, and completes the camouflage analysis and target discrimination.

[0074] d) The data fusion module of the intelligence fusion processing unit realizes the fusion processing of multi-dimensional target information and realizes the accurate identification and analysis of the targets in the area.

[0075] e) The intelligence generation module of the intelligence fusion processing unit constructs the association relationship between the target reconnaissance information and the intelligence information based on the knowledge graph, and automatically completes the generation of the target reconnaissance intelligence.

[0076] f) The target data used in the generated intelligence information is automatically stored in the historical information processing system as sample data to support the neural network training.

[0077] g) The intelligence information is stored in the graph database to support manual annotation by personnel and added as expert knowledge.

Claims

1. A target multi-dimensional information collection and fusion processing system based on unmanned aerial vehicles, characterized in that: The system comprises: UAV platforms, including optoelectronic configuration UAVs, oblique photography configuration UAVs and hyperspectral configuration UAVs, the UAV platforms complete the collection of multi-dimensional data of targets; The ground command and control platform includes an integrated command and control unit, an intelligence fusion processing unit, a measurement and control link unit, and an integrated information processing unit. The ground command and control platform realizes integrated route planning, command and control, payload information processing, information fusion, and intelligence generation for multiple UAV platforms.

2. The system according to claim 1, characterized in that The UAV platform uses a multi-rotor UAV platform, equipped with an optoelectronic reconnaissance payload, an oblique photography payload, and a hyperspectral payload to complete target multi-dimensional data acquisition, and provide target images, three-dimensional modeling data, and spectral data for data fusion processing.

3. The system according to claim 1, characterized in that The integrated command and control unit adopts a standardized control seat, and internally deploys a mission planning module and a command and control module; The mission planning module implements multi-UAV joint mission planning to ensure effective collaboration between UAV platforms and complete reconnaissance data collection in the target area. The command and control module is based on integrated command and control technology and adopts a universal standardized operation interface to achieve integrated control of the UAV platform.

4. The system according to claim 1, characterized in that The UAV platform connects remote control and telemetry to the ground command and control platform through the measurement and control link unit; The comprehensive information processing unit performs processing on remote control and telemetry information, and assists the integrated command and control unit in realizing multi-machine joint monitoring and comprehensive information processing of the entire data collection process.

5. The system according to claim 1, characterized in that The intelligence fusion processing unit internally deploys an optoelectronic reconnaissance module, a three-dimensional modeling module, a spectrum analysis module, a data fusion module and an intelligence generation module.

6. The system according to claim 5, characterized in that The photoelectric reconnaissance module realizes target feature extraction of photoelectric reconnaissance data; The three-dimensional modeling module completes the target three-dimensional reconstruction of the oblique photography data; The spectral analysis module realizes the identification of ground objects in the target area and effectively identifies the target camouflage; The data fusion module fuses the target processing data of the optoelectronic reconnaissance module, the three-dimensional modeling module, and the spectrum analysis module to achieve accurate identification of targets in the area; The intelligence generation module completes the automatic generation of reconnaissance intelligence.

7. A method for collecting and fusion processing multi-dimensional information of a target based on an unmanned aerial vehicle, characterized in that: The method comprises: The integrated command and control unit completes multi-UAV joint mission planning and formulates UAV flight routes based on reconnaissance needs. The UAV platform carries the mission payload and performs the multi-dimensional information collection task according to the pre-set route; During the multi-dimensional information collection process of the UAV target, the UAV platform connects the remote control and telemetry to the ground command and control platform through the measurement and control link unit. The comprehensive information processing unit processes the remote control and telemetry information, assisting the integrated command and control unit to realize multi-machine joint monitoring and information comprehensive processing of the entire data collection process. After data collection is completed, the UAV returns to the base and completes data unloading. The intelligence fusion processing unit carries out target information processing, data fusion and intelligence generation.

8. The method according to claim 7, characterized in that The intelligence fusion processing unit carries out target information processing, data fusion and intelligence generation, including: The optoelectronic reconnaissance module of the intelligence fusion processing unit automatically identifies and extracts targets from optoelectronic reconnaissance data, and obtains target images and location information; The 3D modeling processing unit of the intelligence fusion processing unit completes the 3D modeling of the target area based on the regional oblique photography reconnaissance target, and models and repairs the target 3D model based on the optoelectronic reconnaissance image to improve the accuracy of the regional 3D, and combines with the autonomous GIS engine to generate regional 3D map data; The spectrum analysis module of the intelligence fusion processing unit analyzes the target spectrum data, completes the spectrum analysis of the target area, completes the camouflage analysis and target differentiation; The data fusion module of the intelligence fusion processing unit realizes the fusion processing of multi-dimensional target information and achieves accurate identification and analysis of targets in the area; The intelligence generation module of the intelligence fusion processing unit automatically completes the generation of target reconnaissance intelligence.

Images