Intelligent Deployment Method and Device for Airborne Equipment Tasks Based on Airborne / Cloud Platform

Abstract

The present invention provides an intelligent deployment method and device for airborne device tasks based on an airborne / cloud platform. The method includes: annotating task targets in a very small number of captured images of the airborne platform, and sending the captured images to the cloud platform; extracting the task target features of the annotated captured images; based on the task target features, using a feature matching method to intelligently annotate the task targets in the unannotated captured images; performing manual cleaning on the intelligently annotated captured images to obtain training images; inputting the training images into a deep neural network model for iterative optimization, and outputting a file in the supported format of the airborne platform to complete the deployment of the airborne device for the task targets. The device includes an airborne platform and a cloud platform that are communicatively connected. The method and device designed by the present invention adopt the combination of the airborne platform and the cloud platform to achieve the rapid and intelligent deployment of airborne device tasks, and can meet the requirements in terms of performance, effect, convenience, etc.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to the cross - application of technologies such as deep learning, compilation optimization, and airborne computing in aviation. Specifically, it is an intelligent deployment method and device for airborne equipment tasks based on an airborne / cloud platform. Background Art

[0002] With the development of artificial intelligence technologies represented by neural networks, more and more industries use deep learning models to analyze high - level semantics in different environmental scenarios, such as image classification, object detection, image segmentation, speech recognition, etc.

[0003] Currently, in the field of aviation equipment for artificial intelligence application scenarios, deep learning models are deployed on the airborne platform. When deploying tasks for airborne equipment, artificial intelligence algorithms are usually used to detect and identify targets in images obtained by airborne equipment. It cannot quickly carry out efficient model iteration optimization on large - scale sample data, which will reduce the speed of model training and iterative upgrading, and thus affect the deployment speed and accuracy of airborne equipment tasks. Summary of the Invention

[0004] In order to solve the problem that when artificial intelligence algorithms detect and identify targets in images obtained by airborne equipment during the task deployment of airborne equipment, they cannot quickly carry out efficient model iteration optimization on large - scale sample data, thereby reducing the speed of model training and iterative upgrading, the present invention provides an intelligent deployment method and device for airborne equipment tasks based on an airborne / cloud platform.

[0005] The technical solutions to achieve the invention purpose are as follows:

[0006] In the first aspect, the present invention provides an intelligent deployment method for airborne equipment tasks based on an airborne / cloud platform, including the following steps:

[0007] S1. Based on M captured images obtained by the airborne platform, label the task targets in N captured images, and send the M captured images to the cloud platform, where M > 5 ≥ N ≥ 1;

[0008] S2. Extract the task target features of the labeled captured images;

[0009] S3. Based on the task target features, use the feature matching method to intelligently label the task targets in the unlabeled captured images;

[0010] S4. Manually clean the captured images intelligently labeled in S3 to obtain P training images, where M > P > 5;

[0011] S5. Input P training images into the deep neural network model for iterative optimization, and output a file in the format supported by the airborne platform to complete the deployment of the airborne equipment for the mission target.

[0012] The principle of the intelligent deployment method for the airborne equipment mission designed in the present invention is as follows: First, perform a small amount of manual annotation on the mission targets in the collected images of the airborne platform (when there is one mission target, one collected image can be annotated; when there are two or more mission targets, one or two or more images can be annotated). Second, send the annotated collected images and unannotated collected images to the cloud platform, and perform semi-intelligent annotation (intelligent annotation + manual cleaning) on the unannotated collected images on the cloud platform to obtain training images. Then, input the training images into the deep neural network model for iterative optimization, obtain a file in the format supported by the airborne platform for output to the airborne platform, and complete the deployment of the airborne equipment for the mission target. The intelligent deployment method for the airborne equipment mission designed in the present invention uses the combination of the airborne platform and the cloud platform to perform intelligent deployment on the airborne equipment mission, which can meet the requirements of mission deployment in terms of performance, effect, convenience, etc.

[0013] In an embodiment of the intelligent deployment method for the airborne equipment mission of the present invention, in the above step S1, the mission targets in the N collected images are annotated manually, and one mission target is annotated for each collected image.

[0014] In an embodiment of the intelligent deployment method for the airborne equipment mission of the present invention, in the above step S4, the method for manually cleaning the collected images intelligently annotated in S3 is: perform manual quality inspection on the collected images intelligently annotated and remove the collected images with inaccurate annotations.

[0015] In an improved embodiment of the intelligent deployment method for the airborne equipment mission of the present invention, before inputting the P training images into the deep neural network model for training, it further includes a process of enhancing the P training images, and Q new training images for input into the deep neural network model are obtained after the enhancement process.

[0016] Furthermore, the new training images are obtained by processing the training images with any one or more of the following enhancement methods: image flipping, or random cropping of the ROI region, or image scaling, or scale transformation, or time-frequency domain noise, or contrast transformation, or color perturbation, or random erasing of the ROI region.

[0017] In an embodiment of the intelligent deployment method for airborne equipment tasks of the present invention, in the above step S5, the deep neural network model includes a Pytroch framework model, a TensorFlow framework model, and a Caffe framework model, and the iterative optimization of the training images input into the deep neural network model includes a model lightweight processing process and a model conversion process. Among them, the model conversion process includes converting the.pth or.pth format file output by the Pytroch framework model into an ONNX format file and using it as the input of the hardware-supported file format.

[0018] Further, the above model conversion process also includes converting the.pb format file output by the TensorFlow framework and / or the.Caffemodel or.Caffetxt format file output by the Caffe framework into an ONNX format file and using it as the input of the hardware-supported file format.

[0019] In a second aspect, the present invention provides an intelligent deployment device for airborne equipment tasks based on an airborne / cloud platform. The intelligent deployment method provided in the first aspect is adopted, and the intelligent deployment device is combined to perform intelligent and rapid deployment of airborne equipment tasks. The intelligent deployment device includes an airborne platform located on the airborne equipment and a cloud platform located in the processing center, and the airborne platform is communicatively connected to the cloud platform.

[0020] Among them, the airborne platform is used to acquire the collected images and manually label the task targets in 1 to 5 collected images.

[0021] Among them, the cloud platform includes a data annotation module, a cleaning module, and a deep neural network module. The data annotation module is used to receive the collected images output by the airborne platform, extract the task target features in the already annotated collected images, and perform intelligent annotation on the unannotated collected images. The cleaning module is used to manually clean the annotated collected images output by the data annotation module and output the training images to the deep neural network module. The deep neural network module is used to perform iterative optimization on the training images and output a file in a format supported by the airborne platform.

[0022] In an embodiment of the intelligent deployment device for airborne equipment tasks of the present invention, an enhancement module is further connected to the output end of the above cleaning module. The enhancement module is used to perform enhancement processing on the training images to obtain new training images. The enhancement processing methods include any one or more of image flipping, or random cropping of the ROI region, or image scaling, or scale transformation, or time-frequency domain noise, or contrast transformation, or color perturbation, or random erasing of the ROI region on the training images.

[0023] In an embodiment of the intelligent deployment device for airborne equipment tasks of the present invention, the above-mentioned deep neural network module includes a lightweight function module, a Pytroch framework model, a TensorFlow framework, a Caffe framework, a model conversion module, and a deep learning compilation module. The lightweight function module is used for lightweight and training processing of training images, and the model conversion module is used to convert the.pth or.pth format file output by the Pytroch framework model into an ONNX format file and use it as the input of the hardware-supported file format.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: The intelligent deployment method for airborne equipment tasks designed by the present invention uses a combination of an airborne platform and a cloud platform to perform intelligent deployment of airborne equipment tasks, which can meet the task deployment requirements in terms of performance, effect, convenience, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only for the present invention to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0026] Figure 1 It is a flowchart of an intelligent deployment method for airborne equipment tasks based on an airborne / cloud platform in Embodiment 1;

[0027] Figure 2 It is another flowchart of an intelligent deployment method for airborne equipment tasks based on an airborne / cloud platform in Embodiment 1;

[0028] Figure 3 It is a schematic diagram of an intelligent deployment device for airborne equipment tasks based on an airborne / cloud platform in Embodiment 2;

[0029] Among them, 1. Airborne platform; 2. Cloud platform; 3. Data annotation module; 4. Cleaning module; 5. Deep neural network module; 51. Lightweight function module; 52. Pytroch framework model; 53. TensorFlow framework; 54. Caffe framework model; 55. Model conversion module; 56. Deep learning compilation module; 6. Enhancement module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are merely exemplary and do not constitute any limitation to the scope of the present invention. Those skilled in the art should understand that without departing from the spirit and scope of the present invention, modifications or substitutions can be made to the details and forms of the technical solutions of the present invention, but such modifications and substitutions all fall within the protection scope of the present invention.

[0031] In the description of this embodiment, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention.

[0032] In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0033] Embodiment 1:

[0034] This embodiment provides an intelligent deployment method for airborne device tasks based on an airborne / cloud platform, as Figure 1 shown. The intelligent deployment method for airborne device tasks includes the following steps:

[0035] S1. Based on M captured images obtained by the airborne platform, label the task targets in N captured images, and send the M captured images to the cloud platform, where M > 5 ≥ N ≥ 1.

[0036] In this step, the number of M captured images is very large, which can be dozens, hundreds, thousands, etc., and is captured according to actual needs. Only a very small number of captured images are selected for labeling, for example, 1 image can be selected, or 3 images can be selected, but it is preferably not more than 5 images.

[0037] In this step, the task targets in the N captured images are labeled manually, and in this specific embodiment, preferably only one task target is labeled for each captured image.

[0038] In this step and the following steps, each image is saved as a binary file in RAW format, and then the image is compressed. Using a neural network convolutional framework, scalar quantization technology and a file compression algorithm - Huffman coding - are employed to map the encoded features into a binary stream.

[0039] S2. Extract the task target features of the labeled acquisition images.

[0040] In this step, the extraction of task target features in the labeled acquisition images is carried out using existing general methods (such as unsupervised learning methods), which will not be elaborated here.

[0041] S3. Based on the task target features, use the feature matching method to intelligently label the task targets in the unlabeled acquisition images.

[0042] In this step, through a data annotation tool customized by the cloud platform, the task targets in the unlabeled acquisition images are intelligently labeled. Since there may be situations such as incorrect labeling and inaccurate labeling in the intelligent labeling, it is necessary to perform manual cleaning on the acquisition images input into the deep neural network model to improve the accuracy of the iterative optimization of the deep neural network model.

[0043] S4. Manually clean the acquisition images intelligently labeled in S3 to obtain P training images, where M > P > 5.

[0044] In this step, the method for manually cleaning the acquisition images intelligently labeled in S3 is as follows: conduct manual quality inspection on the acquisition images intelligently labeled and remove the acquisition images with inaccurate labeling. Steps S3 and S4 form a semi-supervised labeling method.

[0045] S5. Input the P training images into the deep neural network model for iterative optimization and output a format file supported by the airborne platform to complete the deployment of the task targets by the airborne equipment.

[0046] In this step, the deep neural network model is equipped with Pytroch framework model, TensorFlow framework model, and Caffe framework model, and the input of the training images into the deep neural network model for iterative optimization includes the model lightweight processing process and the model conversion process.

[0047] Since the hardware form of the airborne platform can be selected from two types: FPGA or NPU. Among them, the.pb format file output by the TensorFlow framework model and the.Caffemodel or.Caffetxt format file output by the Caffe framework model can be directly received by the airborne platform with FPGA or NPU. However, the.pth or.pth format file output by the Pytroch framework model cannot be directly received by the airborne platform with FPGA or NPU and needs to be converted before output.

[0048] In an example of this step, for the.pth or.pth format file output by the Pytroch framework model, the model conversion process includes converting the.pth or.pth format file output by the Pytroch framework model into an ONNX format file and using it as the input of the hardware-supported file format.

[0049] In another example of this step, although the.pb format file output by the TensorFlow framework model and the.Caffemodel or.Caffetxt format file output by the Caffe framework model can be directly output without conversion, for the sake of ensuring the unity of the output files, the above model conversion process also includes the conversion process of the output files of the TensorFlow framework model and the Caffe framework model. Specifically, it includes converting the.pb format file output by the TensorFlow framework and / or the.Caffemodel or.Caffetxt format file output by the Caffe framework into an ONNX format file and using it as the input of the hardware-supported file format.

[0050] In a preferred embodiment of this specific implementation manner, when the number of training images obtained after being processed in steps S1 to S4 is too small to meet the data for iterative optimization of the deep neural network model, it is necessary to perform enhancement processing on the training images to obtain new training images and increase the number input to the deep neural network model. As Figure 2 shown, before inputting P training images into the deep neural network model for training, it also includes the process of enhancing the P training images. After the enhancement process, Q new training images input to the deep neural network model are obtained. At this time, the training images input to step S are P training images plus Q new training images.

[0051] Specifically, the new training images are obtained by performing any one or more of the following enhancement methods on the training images: image flipping (randomly rotating the input image by [0, 360)), or randomly cropping the ROI region (randomly selecting a small square region from an image and setting the pixel values in this region to 0 or other unified values), or image scaling, or scale transformation, or time-frequency domain noise, or contrast transformation, or color perturbation, or randomly erasing the ROI region, or data mixing (randomly selecting two pictures from the dataset and fusing them according to a certain ratio). For example, an image flipping process can be performed on a training image to obtain a new training image; or the above 2, 3,..., or even 9 methods can be used for enhancement processing to obtain 2, 3,..., 9 new training images respectively.

[0052] The principle of the intelligent deployment method for the airborne equipment task designed in this specific embodiment is as follows: First, perform a small amount of manual annotation on the task targets in the captured images of the airborne platform (when there is one task target, one captured image can be annotated; when there are two or more task targets, one or two or more images can be annotated). Second, send the annotated captured images and unannotated captured images to the cloud platform, and perform semi-intelligent annotation (intelligent annotation + manual cleaning) on the unannotated captured images on the cloud platform to obtain training images. Then, input the training images into the deep neural network model for iterative optimization to obtain a file in the support format of the airborne platform output to the airborne platform, completing the deployment of the task targets by the airborne equipment. The intelligent deployment method for the airborne equipment task designed by the present invention uses the combination of the airborne platform and the cloud platform to perform intelligent deployment of the airborne equipment task, which can meet the requirements of task deployment in terms of performance, effect, convenience, etc.

[0053] Embodiment 2:

[0054] This embodiment provides an intelligent deployment device for the airborne equipment task based on the airborne / cloud platform. Using the intelligent deployment method for the airborne equipment task provided in Embodiment 1, the intelligent deployment device is combined to perform intelligent and rapid deployment of the airborne equipment task.

[0055] Among them, as Figure 3 shown, the intelligent deployment device includes the airborne platform 1 located on the airborne equipment and the cloud platform 2 located in the processing center. The airborne platform 1 is communicatively connected to the cloud platform 2, where the processing center can be a processing center set on the ground or a processing center set in an airborne flying device.

[0056] Among them, as Figure 3 shown, the airborne platform 1 is used to acquire captured images and manually annotate the task targets in 1 to 5 captured images.

[0057] Among them, asFigure 3 As shown, the cloud platform 2 includes a data annotation module 3, a cleaning module 4, and a deep neural network module 5. The data annotation module 3 is used to receive the collected images output by the airborne platform 1, extract the task target features in the labeled collected images, and perform intelligent annotation on the unlabeled collected images. The cleaning module 4 is used to manually clean the labeled collected images output by the data annotation module 3 and output training images to the deep neural network module 5. The deep neural network module 5 is used to iteratively optimize the training images and output format files supported by the airborne platform 1.

[0058] In an improved embodiment of the intelligent deployment device for the above airborne equipment tasks, as Figure 3 shown, the output end of the above cleaning module 4 is also connected to an enhancement module 6. The enhancement module 6 is used to perform enhancement processing on the training images to obtain new training images. The enhancement processing methods include any one or more of image flipping, or random cropping of the ROI region, or image scaling, or scale transformation, or time-frequency domain noise, or contrast transformation, or color perturbation, or random erasing of the ROI region on the training images. The enhancement method of the training images is the same as that in Embodiment 1 above.

[0059] In an embodiment of the intelligent deployment device for the airborne equipment tasks in this embodiment, as Figure 3 shown, the above deep neural network module 5 includes a lightweight function module 51, a Pytroch framework model 52, a TensorFlow framework model 53, a Caffe framework model 54, a model conversion module 55, and a deep learning compilation module 56. The lightweight function module 51 is used to perform lightweight and training processing on the training images. The model conversion module 55 is used to convert the.pth or.pth format files output by the Pytroch framework model 52 into ONNX format files and use them as the input of the hardware-supported file format.

[0060] In an improvement of the above deep neural network module 5, the model conversion module 55 is also used to convert the.pb format files output by the TensorFlow framework 53 and the.Caffemodel or.Caffetxt format files output by the Caffe framework 54, and convert them into ONNX format files and use them as the input of the hardware-supported file format.

[0061] In this embodiment, the methods for the model conversion module 55 and the deep learning compiler 56 to convert different format files are as follows: use the model conversion module 55 to convert files in formats such as.pt,.Caffemodel,.Caffetxt,.pb into ONNX format files; then the deep learning compiler 56 compiles them into binary files; finally, use the ONNX format as the input of the hardware-supported file format.

[0062] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0063] In addition, it should be understood that although this specification is described in accordance with the embodiments, not every embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An intelligent deployment method for airborne equipment tasks based on an airborne / cloud platform, characterized in that: It includes the following steps: S1. Based on M captured images obtained by an airborne platform, label the task targets in N captured images, and transport the M captured images to a cloud platform, where M > 5 ≥ N ≥ 1; S2. Extract the task target features of the labeled captured images; S3. Based on the task target features, use the feature matching method to intelligently label the task targets in the unlabeled captured images; S4. Manually clean the captured images intelligently labeled in S3 to obtain P training images, where M > P > 5; S5. Input the P training images into a deep neural network model for iterative optimization, and output a format file supported by the airborne platform to complete the deployment of the task targets by the airborne device; When the training images obtained after being processed by steps S1 to S4 cannot meet the data requirements for iterative optimization of the deep neural network model, it is necessary to perform enhancement processing on the training images to obtain new training images and increase the number input into the deep neural network model.

2. The intelligent deployment method for airborne equipment tasks according to claim 1, characterized in that: In step S1, the task targets in the N captured images are labeled manually, and one task target is labeled for each captured image.

3. The intelligent deployment method for airborne equipment tasks according to claim 1, wherein: In step S4, the method for manually cleaning the captured images intelligently labeled in S3 is: perform manual quality inspection on the captured images intelligently labeled and remove the captured images with inaccurate labels.

4. The intelligent deployment method for the airborne equipment tasks according to claim 1, characterized in that: The new training images are obtained by performing any one or more of the following enhancement methods on the training images: image flipping, or random cropping of the ROI region, or image scaling, or scale transformation, or time-frequency domain noise, or contrast transformation, or color perturbation, or random erasing of the ROI region.

5. The intelligent deployment method for airborne equipment tasks according to claim 1, characterized in that: In step S5, the deep neural network model includes a Pytroch framework model, a TensorFlow framework model, and a Caffe framework model, and inputting the training images into the deep neural network model for iterative optimization includes a model lightweight processing process and a model conversion process; Among them, the model conversion process includes converting the.pth or.pt format file output by the Pytroch framework model into an ONNX format file and using it as the input of the hardware-supported file format.

6. The intelligent deployment method for the airborne equipment tasks according to claim 5, wherein: The model conversion process also includes converting the.pb format file output by the TensorFlow framework and / or the.Caffemodel or.Caffetxt format file output by the Caffe framework into an ONNX format file and using it as the input of the hardware-supported file format.

7. An intelligent deployment device for airborne equipment tasks based on an airborne / cloud platform, characterized in that: Using the intelligent deployment method described in any one of claims 1 to 6 for airborne device task deployment, including an airborne platform located on the airborne device and a cloud platform located in the processing center, and the airborne platform is communicatively connected to the cloud platform; The airborne platform is used to obtain captured images and manually label the task targets in 1 to 5 captured images; The cloud platform includes a data annotation module, a cleaning module, and a deep neural network module. The data annotation module is used to receive the acquired images output by the airborne platform, extract the task target features in the annotated acquired images, and perform intelligent annotation on the unannotated acquired images; the cleaning module is used to manually clean the annotated acquired images output by the data annotation module and output training images to the deep neural network module; the deep neural network module is used to perform iterative optimization on the training images and output format files supported by the airborne platform.

8. The intelligent deployment device for airborne equipment tasks according to claim 7, characterized in that: An enhancement module is further connected to the output end of the cleaning module. The enhancement module is used to perform enhancement processing on the training images to obtain new training images. The enhancement processing methods include any one or more of image flipping, random cropping of ROI regions, image scaling, scale transformation, time-frequency domain noise, contrast transformation, color perturbation, and random erasing of ROI regions for the training images.

9. The intelligent deployment device for airborne equipment tasks according to claim 7, characterized in that: The deep neural network module includes a lightweight function module, a Pytroch framework model, a TensorFlow framework, a Caffe framework, a model conversion module, and a deep learning compilation module. The lightweight function module is used to perform lightweight and training processing on the training images. The model conversion module is used to convert the.pth or.pth format file output by the Pytroch framework model into an ONNX format file and use it as the input of the hardware-supported file format.

Images