Real-time processing and intelligent analysis software platform for image data of unmanned aerial vehicle

Abstract

The invention discloses an unmanned aerial vehicle image data real-time processing and intelligent analysis software platform, and relates to the technical field of image processing, and the software platform comprises a task target module which is used for building a task target database and a three-dimensional virtual model library, and after the task target module receives task target data, the task target database and the three-dimensional virtual model library are built; the task target module is used for transmitting task target data to a target database, and analyzing original image feature points of the task target data and multi-source image data associated with the task target data. After the task target database obtains the image data of the task target, the image processing module is utilized to process and analyze the task target in real time according to the processing state selected by the dynamic adjustment module, so that whether the collected image is subjected to cooperative work or not is selected conveniently, and the overall processing efficiency is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a software platform for real-time processing and intelligent analysis of UAV image data. Background Art

[0002] In view of the increasing expansion and in-depth development of the current UAV industry in China, as an extended platform for remote sensing applications, UAVs play an increasingly important role in all walks of life. Analyzing from the current integrated use mode of UAVs, the research and development of UAVs and the research and development of UAV payloads are separate from each other and have quite strong independence. As a result, the coupling between the UAV itself and the UAV effective payload in other aspects except for the mechanical structure is reduced, especially the non-openness of some aircraft telemetry information and the spatio-temporal asynchrony of payload data. In UAV operations, efficiently acquiring and processing target image data is one of the core challenges. Traditional UAV image acquisition systems usually adopt a fixed task execution mode and lack the ability to adapt to dynamic environmental changes, resulting in low overall work efficiency when the UAV takes separate or collaborative images. Therefore, we propose a software platform for real-time processing and intelligent analysis of UAV image data. Summary of the Invention

[0003] The purpose of the present invention is to provide a software platform for real-time processing and intelligent analysis of UAV image data.

[0004] To achieve the above purpose, the present invention provides the following technical solution: A software platform for real-time processing and intelligent analysis of UAV image data, the software platform includes: A task objective module, used to establish a task objective database and a three-dimensional virtual model library. After the task objective module receives task objective data, it transmits the task objective data to the target database, analyzes the original image feature points of the task objective data and the multi-source image data associated with the task objective data, marks the multi-source image data associated with the task objective data as type A data, and marks the original image feature points of the task objective data as type B data. The three-dimensional virtual model library is used to store the three-dimensional model data of the map and the marking data of the feature points in the three-dimensional model data; A communication module, used to encrypt and transmit the image feature points and related image feature data of the task objective module by using the 128-bit AES encryption algorithm; An image acquisition module, used to collect original images and timestamps through multiple sources; A dynamic adjustment module, used to dynamically adjust the weight coefficient and processing method between type B data and type A data by identifying the relationship between type B data and type A data; The image processing module has two processing methods. When the dynamic adjustment module recognizes that type A data is in type B data, it dynamically adjusts the clarity of type B data according to the weight coefficient recognized by the dynamic adjustment module, cuts the image of type B data in the multi-source acquisition original image, then fuses the cut image data, and then identifies the original image feature points of type A data. When the dynamic adjustment module recognizes that type A data is in contact with type B data, it retrieves the data associated with type A data in other image acquisition modules by timestamp, splices the associated contact data, and then the software platform synthesizes the multi-source acquisition original image to improve the clarity of the image associated with type A data.

[0005] As a further solution of the present invention: The specific working steps of the task target database are as follows: S100. A semantic recognition unit is set in the task target module. After receiving the task target data, the semantic recognition unit retrieves type A data in the task target database; S110. Image feature point analysis. Transmit type A data to the feature extraction unit. The feature extraction unit analyzes and processes the multi-source image data to obtain type A data from different data sources; S120. After the feature extraction unit analyzes different type A data, transmit the different type A data, and the task target module generates an instruction for analyzing the original image feature points and transmits the original image feature point instruction to the task target database. Then the task target database analyzes type B data, and after the analysis is completed, it is transmitted and the current task target database task is terminated.

[0006] As a further solution of the present invention: An orthophoto image model and an oblique photo image model are set in the three-dimensional virtual model library. The orthophoto image model and the oblique photo image model are used to plan and mark the data in the stored map. The oblique photo image model is used to generate a map model for planning and management.

[0007] As a further solution of the present invention: The communication module encrypts the data in the task target module through the following steps. The specific steps are as follows: S210. After the communication module transmits type A data, it establishes a temporary storage area for storing the 128-bit AES encryption algorithm key of type A data; S220. When the type B data related to type A data is transmitted to the communication module, it retrieves the 128-bit AES encryption algorithm key in the temporary storage area to encrypt the type B data.

[0008] As a further solution of the present invention: a camera, a multispectral camera, and an infrared camera are provided in the image acquisition module. After images are captured by the high-definition camera, the multispectral camera, and the infrared camera on different drones, the timestamp of the current image is obtained.

[0009] As a further solution of the present invention: the dynamic adjustment module sets a management threshold ; When holds, the image processing module stitches multiple Class A data according to the timestamp; When holds, the image processing module dynamically adjusts the clarity of Class B data; Wherein, and are respectively the weight coefficients of the height and the shooting wide angle of the image acquisition module, is the height data of the image acquisition module, is the shooting wide angle data of the image acquisition module.

[0010] As a further solution of the present invention: when the image processing module stitches multiple Class A data according to the timestamp, the software platform stitches the Class A data with the same timestamp through the random sample consensus algorithm, and then the image processing module obtains the image edge data of the Class A data according to the stitching of the Class B data.

[0011] As a further solution of the present invention: when the image processing module dynamically adjusts the clarity of Class A data, the priority of different regions in the Class A data is adjusted through the formula. The specific formula is as follows: ; Wherein, is the historical proportion of the task target data, and the value range is 0 ≤ ≤ 1, is the number of pixels, is the weight system, and the value range is > 0.

[0012] Adopting the above technical solutions, compared with the prior art, the beneficial effects of the present invention are as follows: 1. In the present invention, the task target module and the dynamic adjustment module cooperate with each other. After the task target database obtains the image data of the task target, the image processing module uses the processing state selected by the dynamic adjustment module to perform real-time processing and analysis on the task target according to the processing state, which is convenient for selecting whether to perform collaborative work on the captured images, thereby reducing the overall processing efficiency; 2. When the present invention works in a single image processing module, by adjusting the weight coefficients associated between different regions, and then judging before and after the image processing module recognizes the target task, thereby improving the recognition speed of the target task. 3. After the image processing module updates and completes the data, the data in the image processing module is transmitted to the three-dimensional model, and the three-dimensional model data is updated, which is convenient for subsequent retrieval of the three-dimensional model data to analyze the image content. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the software platform process in an embodiment of the present invention; Figure 2 It is a schematic diagram of the dynamic adjustment module process in an embodiment of the present invention; Figure 3 It is a schematic diagram of the steps of the image processing module in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] The following further describes the specific embodiments of the present invention with reference to the drawings. It should be noted here that the description of these embodiments is used to help understand the present invention, but does not limit the present invention.

[0015] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0016] Please refer to the attached Figure 1 - attached Figure 3 , a software platform for real-time processing and intelligent analysis of UAV image data according to the present invention, the software platform includes: A task target module, which is used to establish a task target database and a three-dimensional virtual model library. After the task target module receives the task target data, it transmits the task target data to the target database, analyzes the original image feature points of the task target data and the multi-source image data associated with the task target data, marks the multi-source image data associated with the task target data as type A data, and marks the original image feature points of the task target data as type B data. The three-dimensional virtual model library is used to store the three-dimensional model data of the map and the marking data of the feature points in the three-dimensional model data; A communication module, which is used to encrypt and transmit the image feature points and related image feature data of the task target module by using the 128-bit AES encryption algorithm; An image acquisition module, which is used to collect the original image and the time stamp through multiple sources; A dynamic adjustment module, which is used to dynamically adjust the weight coefficients and processing methods between type B data and type A data by identifying the relationship between type B data and type A data; The image processing module has two processing methods. When the dynamic adjustment module recognizes that type A data is in type B data, it dynamically adjusts the clarity of type B data according to the weight coefficient recognized by the dynamic adjustment module, cuts the image of type B data in the multi-source acquisition original image, then fuses the cut image data, and then recognizes the original image feature points of type A data. When the dynamic adjustment module recognizes that type A data is in contact with type B data, it retrieves the data associated with type A data in other image acquisition modules by timestamp, splices the associated contact data, and then the software platform synthesizes the multi-source acquisition original image to improve the clarity of the image associated with type A data.

[0017] In an embodiment of the present invention: The specific working steps of the task target database are as follows: S100. Set a semantic recognition unit in the task target module. After receiving the task target data, the semantic recognition unit retrieves type A data in the task target database; S110. Image feature point analysis. Transmit type A data to the feature extraction unit, and the feature extraction unit analyzes and processes the multi-source image data to obtain type A data from different data sources; S120. After the feature extraction unit analyzes different type A data, transmit the different type A data, and the task target module generates an instruction to analyze the original image feature points and transmits the original image feature point instruction to the task target database. Then the task target database analyzes type B data, and after the analysis is completed, it transmits and terminates the current task target database task.

[0018] In an embodiment of the present invention: An orthophoto image model and an oblique photo image model are set in the three-dimensional virtual model library. The orthophoto image model and the oblique photo image model are used to plan and mark the data in the stored map, and the oblique photo image model is used to generate a map model for planning and management.

[0019] In an embodiment of the present invention: The communication module encrypts the data in the task target module through the following steps. The specific steps are as follows: S210. After the communication module transmits type A data, it establishes a temporary storage area to store the 128-bit AES encryption algorithm key of type A data; S220. When type B data related to type A data is transmitted to the communication module, it retrieves the 128-bit AES encryption algorithm key in the temporary storage area to encrypt type B data.

[0020] In an embodiment of the present invention: The dynamic adjustment module sets a management threshold ; When When this occurs, the image processing module splices multiple Class A data according to the time stamp; When this occurs, the image processing module dynamically adjusts the clarity of Class B data; Among them, and are the weight coefficients of the height of the image acquisition module and the shooting wide angle respectively, ( and The weight coefficients of are 0.6 and 0.4 respectively, and ) is the height data of the image acquisition module, is the shooting wide angle data of the image acquisition module.

[0021] In an embodiment of the present invention: when the image processing module splices multiple Class A data according to the time stamp, the software platform splices the Class A data with the same time stamp through the random sample consensus algorithm, and then the image processing module obtains the image edge data of the Class A data according to the splicing of the Class B data.

[0022] In an embodiment of the present invention: when the image processing module dynamically adjusts the clarity of Class A data, the priority of different regions in the Class A data is adjusted through the formula. The specific formula is as follows: ; Among them, is the historical proportion of the task target data, and the value range is 0 ≤ ≤ 1, is the number of pixels, is the weight system, and the value range is > 0.

[0023] Example 1: Single UAV processing scenario When the dynamic adjustment module recognizes that the height and shooting wide angle data of the image acquisition module in the UAV exceed the management threshold, the image processing module separately processes the image data collected by the image acquisition module to obtain the feature analysis of the same scene by different cameras; Use multi-band fusion to fuse the feature data captured by different cameras into the required image content; Analyze the task target data features in the fused image content through the matching unit.

[0024] Example 2: Multi-UAV coordinated processing scenario When the dynamic adjustment module recognizes that the height of the image acquisition module and the shooting wide-angle data in the drone are lower than the management threshold, the image acquisition module transmits the timestamp of the captured image to the image processing module. Then, the image processing module stitches the captured images according to the timestamp, and then plans according to the place where it contacts the B-class data to quickly clarify the scope of the A-class data.

[0025] Embodiment 3: Task Target Database Analysis Scenario When the target task is about the temperature direction, the weight ratio of fusion regarding the infrared camera is increased. When the target task is about the color chromatic aberration direction, the weight ratio of fusion regarding the multispectral camera is increased, and so on, thereby ensuring the accuracy of the software platform for the corresponding target task.

[0026] Multi-band fusion: Weighted superposition of data from different cameras (visible light, infrared, multispectral) according to bands, and the weights are assigned by the dynamic adjustment module; Robustness algorithm: The RANSAC (Random Sample Consensus) algorithm is used to eliminate stitching outliers.

[0027] Specifically, through the cooperation of the task target module according to the task target database and the dynamic adjustment module, after the task target database obtains the image data of the task target, the image processing module uses the processing state selected by the dynamic adjustment module, and then performs real-time processing and analysis on the task target according to the processing state. When the dynamic adjustment module determines that the multi-drone image overlap rate is lower than the threshold, the collaborative stitching mode is triggered.

[0028] Specifically, when working in a single image processing module, by adjusting the weight coefficients associated between different regions, before and after the image processing module recognizes the target task, the speed of recognizing the target task is improved.

[0029] Specifically, after the image processing module updates the data, the data in the image processing module is transmitted to the 3D model, and the 3D model data is updated to facilitate subsequent retrieval of the 3D model data for analyzing the image content.

[0030] Working principle: First, after the task target module obtains the task target, the task target database analyzes the image data of the task target and the associated image data. The communication module encrypts and transmits the image data and the associated image data, that is, A-class data and B-class data. At the same time, the image acquisition module captures images through different types of cameras and the corresponding timestamps of the captured images. Then, the dynamic adjustment module selects the processing mode of the image processing module, thereby facilitating the dynamic adjustment of the real-time processing and analysis of the images. Thus, the entire work process ends.

[0031] Although the present invention is disclosed above in preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and decorations made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention all fall within the protection scope defined by the claims of the present invention.

Claims

1. A real-time processing and intelligent analysis software platform for UAV image data, characterized in that: The software platform includes: A task target module is used to establish a task target database and a three-dimensional virtual model library. After receiving the task target data, the task target module transfers the task target data to the target database, analyzes the original image feature points of the task target data and the multi-source image data associated with the task target data, marks the multi-source image data associated with the task target data as Class A data, and marks the original image feature points of the task target data as Class B data. The three-dimensional virtual model library is used to store the three-dimensional model data of the map and the marking data of the feature points in the three-dimensional model data; A communication module, used to encrypt and transmit the image feature points of the task target module and the associated image feature data using a 128-bit AES encryption algorithm; An image acquisition module, used to acquire original images and timestamps through multiple sources; A dynamic adjustment module is used to dynamically adjust the weight coefficient and processing method between the class B data and the class A data by identifying the relationship between the class B data and the class A data; The image processing module has two processing methods. When the dynamic adjustment module recognizes that Class A data is in Class B data, the clarity of Class B data is dynamically adjusted according to the weight coefficient recognized by the dynamic adjustment module, and the image of Class B data in the original image acquired from multiple sources is cut, and the cut image data is fused, and then the original image feature points of Class A data are identified. When the dynamic adjustment module recognizes that Class A data is in contact with Class B data, the timestamp is used to retrieve the data associated with Class A data in other image acquisition modules, and the associated contacting data are spliced, and then the software platform synthesizes the original image acquired from multiple sources to improve the clarity of the image associated with Class A data.

2. The real-time processing and intelligent analysis software platform for UAV image data according to claim 1, characterized in that, The specific working steps of the task target database are as follows: S100, a semantic recognition unit is set in the task target module, and after receiving the task target data, the semantic recognition unit retrieves the Class A data in the task target database; S110, analyzing image feature points, transmitting the class A data to a feature extraction unit, wherein the feature extraction unit analyzes and processes the multi-source image data to obtain class A data from different data sources; S120. After the feature extraction unit analyzes different Class A data, the different Class A data are transmitted, and the task target module generates an instruction to analyze the feature points of the original image, and transmits the original image feature point instruction to the task target database. The task target database then analyzes the Class B data, transmits it after the analysis is completed, and terminates the current task target database task.

3. A real-time processing and intelligent analysis software platform for UAV image data according to claim 2, characterized in that: The three-dimensional virtual model library is provided with an orthophoto model and an oblique image model, the orthophoto model and the oblique image model are used for planning and marking data stored in the map, the oblique image model is used for generating a map model, and the map model is used for planning and management.

4. The real-time processing and intelligent analysis software platform for UAV image data according to claim 3, characterized in that: The communication module encrypts the data in the task target module through the following steps, and the specific steps are as follows: S210, after transmitting the Class A data, the communication module establishes a temporary storage interval for storing the key of the 128-bit AES encryption algorithm for the Class A data; S220. When the B - type data related to the A - type data is transmitted to the communication module, the key of the 128 - bit AES encryption algorithm in the temporary storage area is retrieved to encrypt the B - type data.

5. The real-time processing and intelligent analysis software platform for UAV image data according to claim 4, characterized in that: The image acquisition module is provided with a camera, a multispectral camera, and an infrared camera. After different drones capture images through the high - definition camera, the multispectral camera, and the infrared camera, the timestamp of the current image is obtained.

6. The real-time processing and intelligent analysis software platform for UAV image data according to claim 1, characterized in that: The dynamic adjustment module sets a management threshold ; When the image processing module splices multiple Class A data according to the time stamp; When the image processing module dynamically adjusts the clarity of Class B data; Among them, and are the height of the image acquisition module and the weight coefficient of the shooting wide angle respectively, is the height data of the image acquisition module, is the shooting wide angle data of the image acquisition module.

7. The real-time processing and intelligent analysis software platform for UAV image data according to claim 6, characterized in that: The image processing module stitches multiple A - type data according to the timestamp. Then, the software platform stitches the A - type data with the same timestamp through the random sample consensus algorithm. Furthermore, the image processing module obtains the image edge data of the A - type data according to the stitching of the B - type data.

8. The real-time processing and intelligent analysis software platform for UAV image data according to claim 6, characterized in that: When the image processing module dynamically adjusts the clarity of type A data, it adjusts the priorities of different regions in type A data through a formula. The specific formula is as follows: ; Among them, is the historical proportion of task target data, and the value range is 0 ≤ ≤ 1, is the number of pixels, is the weight system, and the value range is > 0.

Images