Airborne high-real-time image processing architecture and method

Abstract

The invention belongs to the field of intelligent computing, and particularly relates to an airborne high-real-time image processing architecture and method.The airborne high-real-time image processing architecture comprises a software framework and a hardware framework, and the software framework comprises platform software and basic software; the hardware architecture is divided into a core component and peripheral components; the platform software comprises neural network runtime software, an embedded container, an OpenCV library and an OpenMP library; the core component comprises a data processing component and an intelligent processing component; decoupling of a hardware architecture and a software architecture is realized, and the two architectures have a calling relationship instead of a direct connection relationship, so that efficient processing of different tasks can be performed without mutual influence. In a parallel processing airborne image mode, when data is read from a dynamic random access memory to a buffer, three contents of image reading, sub-image splitting and intelligent processing are completed at the same time. And for each frame of sub-graph, only the time for reading the sub-graph by the DDR needs to be calculated, so that the efficiency is greatly improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent computing, and in particular relates to an airborne high-real-time image processing architecture and method. Background Art

[0002] In recent years, with the gradual increase in environmental complexity, intelligent air combat missions in the new era often have massive data inputs in multiple dimensions. These data are not only large in quantity but also highly real-time and concurrent. In order to ensure that the mission can be executed efficiently and accurately, an airborne image processing architecture with strong computing power and meeting the needs of the aviation field is needed to calculate and process the massive data generated, and to carry a large number of real-time and highly concurrent image processing tasks on the aircraft platform, and to more intelligently improve the processing capabilities of the platform under the condition of limited space resources, thereby supporting intelligent airborne applications.

[0003] Typical image processing requires research on the structural features, texture features, and color features of images. How to design a repeatable feature extraction method is the key to image processing. Traditional manually designed feature descriptors can extract features for image information in specific scenes, while image feature extraction methods based on deep learning (such as convolutional neural networks) can extract scale-invariant target features for images in different scenes. Therefore, they are widely used in image segmentation, splicing, and fusion scenes based on visual scenes. When processing ultra-high-precision SAR images, the original images need to be segmented, and a large number of image sets will be obtained after processing, and massive input data will be processed in parallel. Under the premise that the real-time requirements of the airborne environment remain unchanged, deep learning image processing and SAR image parallel processing have put forward higher requirements on the computing power of image processing. The traditional image processing architecture is constrained by computing power and inter-board bandwidth and is difficult to meet the real-time requirements.

[0004] Therefore, how to achieve high real-time image intelligent processing requirements is a problem that needs to be solved. Summary of the invention

[0005] The purpose of this application is to provide an airborne high-real-time image processing architecture and method to solve the problem that traditional image processing architecture is difficult to meet real-time requirements due to the constraints of computing power and inter-board bandwidth.

[0006] The technical solution of the present application is: an airborne high-real-time image processing architecture, including a software framework and a hardware framework, the software architecture includes platform software and basic software; the hardware architecture is divided into core components and peripheral components; the platform software includes neural network runtime software, embedded container, OpenCV library and OpenMP library; the neural network runtime software stores a neural network model, and the embedded container stores an embedded model; the OpenCV library and the OpenMP library respectively store different types of image processing tools; the core components include a data processing component and an intelligent processing component; the data processing component is used for image processing; the intelligent processing component is used for image recognition to obtain image features inside the image; the data processing component stores the address information of various image processing tools in the OpenCV library and the OpenMP library, and can call each image processing tool through the address; the intelligent processing component stores the address information of all neural network models and embedded models, and can call each model through the address information.

[0007] Preferably, the basic software includes an operating system and a hardware driver; the operating system is used for uploading, deleting, and modifying neural network models, embedded models, and data processing tools, and the hardware driver is used for operating drive control of the platform software.

[0008] Preferably, the peripheral components include a network interface component, a data storage component, a power supply component and a heat dissipation component; the network interface component is used to provide a language interface; the data storage component is used to store the operating data of the hardware architecture, and the power supply component and the heat dissipation component are used to supply power and dissipate heat for the data processing component and the intelligent processing component, respectively.

[0009] Preferably, a data pool is set up in the platform software. When uploading, each new neural network model, embedded model and data processing tool are uploaded to different space areas in the data pool and the corresponding addresses are output; the data processing components and the intelligent processing components call the corresponding models or tools through the addresses in the data pool.

[0010] Preferably, the data processing component includes a dynamic random access memory and a buffer. The dynamic random access memory can receive input image data and perform operations such as reading, writing and storing. The buffer can perform image reading, image splitting and intelligent processing operations.

[0011] Preferably, the intelligent processing component includes a model calling module, a data processing module and a data storage module; the model calling module is used to call the corresponding model according to the address and send it to the data processing module, the data processing module runs the corresponding model and performs image feature recognition calculations to obtain calculation results, and sends them to the data storage module and the dynamic random access memory respectively.

[0012] As a specific implementation, an airborne high real-time image processing method includes: inputting a neural network model, an embedded model and image processing tool information to be updated into a dynamic random access memory via Ethernet, and then the dynamic random access memory stores the neural network model and the embedded model information inside itself, and inputting the image processing tool information into a data storage module of an intelligent processing component for storage;

[0013] The network interface component receives the input video and sends it to the dynamic random access memory, the dynamic random access memory obtains the type of video, and determines the model type and image processing tool to be switched according to the type of video, forms a switching instruction, and sends it to the intelligent processing component and the buffer;

[0014] The buffer calls the corresponding image processing tool according to the switching instruction to perform image reading, sub-image splitting and intelligent processing operations, and sends each sub-image to the intelligent processing component after processing;

[0015] Every time the intelligent processing component receives a sub-image, it calls the corresponding new model to perform image detection according to the switching instruction, and outputs the detection result through the network interface component.

[0016] Preferably, the data processing component and the intelligent processing component are connected via an inter-board bus.

[0017] Preferably, when the new model is transmitted, the old model will continue to work; until the entire new model is transmitted, the new model is switched to perform subsequent target detection tasks.

[0018] The airborne high real-time image processing architecture and method of the present application realizes the decoupling of the hardware architecture and the software architecture. The two architectures have a calling relationship rather than a direct connection relationship, so that different tasks can be efficiently processed without affecting each other. In the way of parallel processing of airborne images, when the data is read from the dynamic random access memory to the buffer, the three contents of image reading, sub-image splitting, and intelligent processing are completed at the same time. For each frame of sub-image, it is only necessary to calculate the time for DDR to read the sub-image, which greatly improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solution provided by the present application, the following is a brief introduction to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present application.

[0020] Figure 1 This is a schematic diagram of the overall structure of this application;

[0021] Figure 2 This is the video processing flow chart for this application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] An airborne high-real-time image processing architecture provides an image processing architecture with decoupled software and hardware. When different mission requirements are met, new mission requirements can be met by loading different software components.

[0024] like Figure 1 , including software framework and hardware framework. The software architecture includes platform software and basic software; the hardware architecture is divided into core components and peripheral components. The platform software includes neural network runtime software, embedded container, OpenCV library and OpenMP library. The neural network runtime software stores the neural network model, and the embedded container stores the embedded model; the OpenCV library and OpenMP library store different types of image processing tools respectively.

[0025] The basic software includes operating system and hardware driver. The operating system is used for uploading, deleting, modifying and other operations of neural network models, embedded models and data processing tools, and the hardware driver is used for operating and driving control of platform software.

[0026] The core components include data processing components and intelligent processing components. The data processing components are used for image processing, including cutting, scaling, rotation, etc.; the intelligent processing components are used for image recognition to obtain the image features inside the image.

[0027] Peripheral components include network interface components, data storage components, power supply components and heat dissipation components. The network interface component is used to provide interfaces for languages ​​such as Python, Ruby, and MATLAB; the data storage component is used to store the running data of the hardware architecture; the power supply component and heat dissipation component are used to supply power and dissipate heat for the data processing component and the intelligent processing component, respectively.

[0028] The data processing component stores the address information of various image processing tools in the OpenCV library and the OpenMP library, and can call each image processing tool through the address; the intelligent processing component stores the address information of all neural network models and embedded models, and can call each model through the address information.

[0029] Through the above design, the decoupling of the hardware architecture and the software architecture is achieved. The two architectures have a calling relationship rather than a direct connection relationship, so that different tasks can be processed efficiently without affecting each other.

[0030] Furthermore, a data pool can be set up in the platform software. When uploading each new neural network model, embedded model and data processing tool, it is uploaded to a different space area in the data pool and the corresponding address is output. The data processing component and the intelligent processing component call the corresponding model or tool through the address in the data pool, thereby effectively improving the high real-time image intelligent processing requirements.

[0031] Furthermore, the data processing component includes a dynamic random access memory and a buffer. The dynamic random access memory preferably adopts the FT-D2000 / 8 daughter card DDR type. The dynamic random access memory can receive input image data and perform operations such as reading, writing and storing. The buffer can perform operations such as image reading, image splitting, and intelligent processing.

[0032] Furthermore, the intelligent processing component includes a model calling module, a data processing module and a data storage module; the model calling module is used to call the corresponding model according to the address and send it to the data processing module, the data processing module runs the corresponding model and performs image feature recognition calculations to obtain the calculation results, and sends them to the data storage module and the dynamic random access memory respectively.

[0033] As a specific implementation, it also includes an airborne high real-time image processing method,

[0034] like Figure 2 , specifically including the following steps:

[0035] Step S100, the neural network model, embedded model and image processing tool information to be updated are input into the dynamic random access memory via Ethernet, and then the dynamic random access memory stores the neural network model and embedded model information inside itself, and inputs the image processing tool information into the data storage module of the intelligent processing component for storage.

[0036] Preferably, the data processing component and the intelligent processing component are connected via an inter-board bus.

[0037] Step S200, the network interface component receives the input video and sends it to the dynamic random access memory, the dynamic random access memory obtains the type of video, and determines the model type and image processing tool to be switched according to the type of video, forms a switching instruction, and sends it to the intelligent processing component and the buffer.

[0038] Step S300: the buffer calls the corresponding image processing tool according to the switching instruction to perform operations such as image reading, sub-image splitting and intelligent processing, and sends each processed sub-image to the intelligent processing component.

[0039] Step S400, each time the intelligent processing component receives a sub-image, it calls the corresponding new model for image detection according to the switching instruction, and outputs the detection result through the network interface component; then: for each frame of sub-image, it is only necessary to calculate the time of DDR reading the sub-image, which greatly improves the efficiency.

[0040] In particular, to ensure the continuity of detection tasks, the old model will continue to work while the new model is being transferred. After the entire new model is transferred, the new model will be switched to perform subsequent target detection tasks to ensure the continuity of image detection.

[0041] Through the above design, in a parallel processing mode of airborne images, when data is read from the dynamic random access memory to the buffer, the three tasks of image reading, sub-image splitting and intelligent processing are completed at the same time.

[0042] Assume that the sub-image is split into 1280*1280 size. Based on pipeline design implementation: the intelligent processing unit calculates the sub-image once, and sends the 1280x1280 sub-image from the buffer once. The calculation result is cached in the intelligent processing unit and returned to the DDR.

[0043] Finally, it should be noted that: the drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;

[0044] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An airborne high real-time image processing architecture, characterized in that: It includes software framework and hardware framework. The software architecture includes platform software and basic software. The hardware architecture is divided into core components and peripheral components. The platform software includes neural network runtime software, embedded container, OpenCV library and OpenMP library. The neural network runtime software stores neural network models, and the embedded container stores embedded models. The OpenCV library and OpenMP library store different types of image processing tools respectively. The core components include data processing components and intelligent processing components. The data processing components are used for image processing. The intelligent processing components are used for image recognition to obtain image features inside the image. The data processing component stores the address information of various image processing tools in the OpenCV library and the OpenMP library, and can call each image processing tool through the address; the intelligent processing component stores the address information of all neural network models and embedded models, and can call each model through the address information.

2. The airborne high real-time image processing architecture as claimed in claim 1, characterized in that: The basic software includes an operating system and a hardware driver; the operating system is used for uploading, deleting, and modifying neural network models, embedded models, and data processing tools, and the hardware driver is used for operating and driving control of the platform software.

3. The airborne high real-time image processing architecture as claimed in claim 1, characterized in that: The peripheral components include a network interface component, a data storage component, a power supply component and a heat dissipation component; the network interface component is used to provide a language interface; the data storage component is used to perform operational data storage of the hardware architecture, and the power supply component and the heat dissipation component are used to supply power and dissipate heat for the data processing component and the intelligent processing component, respectively.

4. The airborne high real-time image processing architecture as claimed in claim 1, characterized in that: A data pool is set up in the platform software. When uploading, each new neural network model, embedded model and data processing tool is uploaded to a different space area in the data pool and the corresponding address is output; the data processing component and the intelligent processing component both call the corresponding model or tool through the address in the data pool.

5. The airborne high real-time image processing architecture as claimed in claim 1, characterized in that: The data processing component includes a dynamic random access memory and a buffer. The dynamic random access memory can receive input image data and perform operations such as reading, writing and storing. The buffer can perform image reading, image splitting and intelligent processing operations.

6. The airborne high real-time image processing architecture as claimed in claim 1, characterized in that: The intelligent processing component includes a model calling module, a data processing module and a data storage module; the model calling module is used to call the corresponding model according to the address and send it to the data processing module, the data processing module runs the corresponding model and performs image feature recognition calculations to obtain calculation results, and sends them to the data storage module and the dynamic random access memory respectively.

7. An airborne high real-time image processing method, using the image processing architecture as described in any one of claims 1 to 6, characterized in that: include: Inputting the neural network model, embedded model and image processing tool information to be updated into the dynamic random access memory through Ethernet, and then the dynamic random access memory stores the neural network model and embedded model information inside itself, and inputting the image processing tool information into the data storage module of the intelligent processing component for storage; The network interface component receives the input video and sends it to the dynamic random access memory, the dynamic random access memory obtains the type of video, and determines the model type and image processing tool to be switched according to the type of video, forms a switching instruction, and sends it to the intelligent processing component and the buffer; The buffer calls the corresponding image processing tool according to the switching instruction to perform image reading, sub-image splitting and intelligent processing operations, and sends each sub-image to the intelligent processing component after processing; Every time the intelligent processing component receives a sub-image, it calls the corresponding new model to perform image detection according to the switching instruction, and outputs the detection result through the network interface component.

8. The airborne high real-time image processing architecture as claimed in claim 7, characterized in that: The data processing component and the intelligent processing component are connected via an inter-board bus.

9. The airborne high real-time image processing architecture of claim 7, wherein: While the new model is being transferred, the old model will continue to work; after the entire new model is transferred, the new model will be switched to perform subsequent target detection tasks.

Images