A real-time processing method for spaceborne optical images based on heterogeneous processing

Abstract

The present application discloses a real-time processing method for satellite optical images based on heterogeneous processing, which relates to the field of computer technology and is applied to an intelligent heterogeneous processing architecture, which includes an FPGA and an AI processing unit. The method includes: acquiring an optical image through an FPGA, performing image preprocessing on the optical image to obtain a preprocessed image, and transmitting the preprocessed image to the AI ​​processing unit through a high-speed serial bus; using a neural network acceleration core in the AI ​​processing unit, performing image detection on the preprocessed image based on a trained target detection model to obtain a target detection image containing a target detection frame; the target detection model is a model adapted to the underlying acceleration operator of an embedded AI chip obtained by building on Python and converting the model; and transmitting the target detection image to the FPGA through a high-speed serial bus through the AI ​​processing unit. A large number of optical images can be processed quickly.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a real-time processing method for satellite-borne optical images based on heterogeneous processing. Background Art

[0002] With the development of science and technology, optical sensor technology is changing with each passing day. Among them, optical satellites have received extensive attention in both military and civilian fields because they are not restricted by time and region. Traditional space-based optical platforms generally adopt the method of "sensing on the satellite and transmitting the original image, and receiving and processing on the ground to obtain the original image". However, with the development of advanced manufacturing technology, the spatial and temporal resolution of optical payloads has been continuously improved, the amount of data has increased dramatically, and the timeliness requirements for processing tasks have increased. Traditional processing methods can no longer meet future development needs.

[0003] At present, the image processing architecture of space-based optical platforms is generally DSP (Digital Signal Process) + Field-Programmable Gate Array (FPGA). Among them, FPGA is used for data interaction and image information preprocessing, and DSP receives the target candidate point information output by FPGA for judgment and trajectory association. Although FPGA has high parallel computing efficiency when processing images, its development and transplantation are difficult and consume a lot of resources. DSP is good at sequential processing and performs calculations based on pipelines, but the processing time of the detection algorithm based on DSP chips has reached the design upper limit, which is difficult to meet the needs of real-time processing of larger data volumes in the future. In addition, the signal processing algorithm software is large in scale, and frequent updates and iterations also put great pressure on the data transmission bandwidth between the satellite and the ground.

[0004] In summary, how to process a large number of optical images is a problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a real-time processing method for satellite-borne optical images based on heterogeneous processing, which can quickly process a large number of optical images. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a method for real-time processing of satellite-borne optical images based on heterogeneous processing, which is applied to an intelligent heterogeneous processing architecture, wherein the intelligent heterogeneous processing architecture includes an FPGA and an AI processing unit, and the method includes:

[0007] Acquire an optical image through the FPGA, perform image preprocessing on the optical image to obtain a preprocessed image, and transmit the preprocessed image to the AI ​​processing unit through a high-speed serial bus;

[0008] Using the neural network acceleration core in the AI ​​processing unit, image detection is performed on the preprocessed image based on the trained target detection model to obtain a target detection image containing a target detection frame; the target detection model is a model adapted to the underlying acceleration operator of the embedded AI chip that is built based on Python and obtained through model conversion;

[0009] The AI ​​processing unit transmits the detected target image to the FPGA via a high-speed serial bus.

[0010] The method of using the neural network acceleration core in the AI ​​processing unit to perform image detection on the preprocessed image based on the trained target detection model to obtain a target detected image containing a target detection frame includes:

[0011] Using the neural network acceleration core in the AI ​​processing unit, the pre-processed image is initially detected by a one-stage algorithm based on the trained target detection model to obtain an initial detected image containing image features;

[0012] Utilizing the neural network acceleration core in the AI ​​processing unit, a final detection result is extracted from the image after initial detection through a post-processing algorithm based on the target detection model; the final detection result includes a final target detection frame and a target type corresponding to the final target detection frame.

[0013] Among them, using the neural network acceleration core in the AI ​​processing unit, performing initial detection processing on the pre-processed image through a One-stage algorithm based on a trained target detection model to obtain an initial detected image containing image features, it also includes:

[0014] Export the model built based on Python on the GPU server to onnx format, and use the conversion tool to convert the onnx model to an edge model adapted to the embedded AI chip;

[0015] The post-processing algorithm is added to the edge model, and the threshold filtering algorithm in the post-processing algorithm is developed and compiled into a neural network node through the underlying instruction set to obtain the target detection model.

[0016] The method of using the neural network acceleration core in the AI ​​processing unit to extract the final detection result from the image after the initial detection through a post-processing algorithm based on the target detection model includes:

[0017] Using the neural network acceleration core in the AI ​​processing unit, based on the target detection model, an initial prediction box in the image after the initial detection is determined, and the confidence of each of the initial prediction boxes is determined, and a temporary detection box with a confidence greater than a preset threshold is determined from the initial prediction box through a post-processing algorithm;

[0018] Determine the area of ​​each temporary detection frame and the overlapping area between each temporary detection frame and each other temporary detection frame, and calculate the ratio of the overlapping area to the total area of ​​the two frames to obtain the area intersection and union ratio of the temporary detection frame;

[0019] A non-maximum suppression algorithm is used to remove overlapping redundant frames from the temporary detection frame based on the area intersection-union ratio, and a target detection frame is determined to obtain a target detection result including the target detection frame.

[0020] The method of removing overlapping redundant frames from the temporary detection frame based on the area intersection-to-union ratio by using a non-maximum suppression algorithm to determine a target detection frame to obtain a target detection result including the target detection frame includes:

[0021] The temporary detection frame with the largest area intersection-union ratio in the temporary detection frame set is used as the detection frame to be processed;

[0022] Removing other temporary detection frames in the temporary detection frame set that overlap with the detection frame to be processed, and using the detection frame to be processed as the target detection frame;

[0023] In the remaining temporary detection frames, jump to the step of taking the temporary detection frame with the largest area intersection-union ratio in the temporary detection frame set as the detection frame to be processed, until the target detection frame is determined to obtain the target detection result containing the target detection frame.

[0024] Wherein, performing image preprocessing on the optical image to obtain a preprocessed image includes:

[0025] The bit width of the optical image is converted into the input bit width of the target detection model by using a bit width conversion algorithm, and the image after the bit width adjustment is preprocessed to obtain a preprocessed image; the image preprocessing includes cropping and normalization.

[0026] The step of transmitting the pre-processed image to the AI ​​processing unit via a high-speed serial bus includes:

[0027] The preprocessed image is transmitted to the AI ​​processing unit via a high-speed serial bus, and the preprocessed image is stored in the memory space of the AI ​​chip.

[0028] Wherein, the real-time processing method of spaceborne optical images based on heterogeneous processing further includes:

[0029] The weight incremental update algorithm is used to update some model weights of the previous target detection model to obtain a new target detection model.

[0030] The method of using a weight incremental update algorithm to update some model weights of a previous target detection model to obtain a new target detection model includes:

[0031] The model weights of the previous round of target detection models are updated based on the gradient optimization algorithm, and the model structure of the previous round of target detection models is adjusted using the transfer learning technology to obtain a new round of models;

[0032] The influence of different model weights on the model accuracy is tested, and the partial model weights with the greatest influence on the model accuracy are determined based on a saturation test method, so as to obtain a new target detection model based on the model weight update.

[0033] Among them, the update formula of the weight incremental update algorithm is:

[0034] ;

[0035] in, represents the loss function, represents the weight, Indicates the update round. Indicates The weight of the round compared to the previous round, Indicates The training data set, represents the update rate, which is related to the communication requirements. represents the total number of model weights, The full name is subject to, which means subject to constraints; Represents the norm.

[0036] It can be seen that the present application acquires an optical image through the FPGA, performs image preprocessing on the optical image to obtain a preprocessed image, and transmits the preprocessed image to the AI ​​processing unit through a high-speed serial bus; utilizes the neural network acceleration core in the AI ​​processing unit to perform image detection on the preprocessed image based on the trained target detection model to obtain a target detection image containing a target detection frame; the target detection model is a model adapted to the underlying acceleration operator of the embedded AI chip obtained by model conversion based on Python; and transmits the target detection image to the FPGA through a high-speed serial bus through the AI ​​processing unit. It can be seen that the present application designs an intelligent heterogeneous processing architecture including FPGA and AI processing unit, utilizes the parallel acceleration computing capability of the neural network acceleration core in the AI ​​processing unit, improves image processing efficiency, and realizes real-time high-precision target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of a method for real-time processing of satellite-borne optical images based on heterogeneous processing disclosed in this application;

[0039] Figure 2 A schematic diagram of an intelligent heterogeneous processing architecture disclosed in this application;

[0040] Figure 3 A schematic diagram of a weight incremental update process disclosed in this application;

[0041] Figure 4 A flowchart of a specific method for real-time processing of satellite-borne optical images based on heterogeneous processing disclosed in this application;

[0042] Figure 5 This is a schematic diagram of a multi-core parallel post-processing algorithm process disclosed in this application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] At present, the image processing architecture of space-based optical platforms is generally DSP (Digital Signal Process) + Field-Programmable Gate Array (FPGA). Among them, FPGA is used for data interaction and image information preprocessing, and DSP receives the target candidate point information output by FPGA for judgment and trajectory association. Although FPGA has high parallel computing efficiency when processing images, its development and transplantation are difficult and consume a lot of resources. DSP is good at sequential processing and performs calculations based on pipelines, but the processing time of the detection algorithm based on DSP chips has reached the design upper limit, which is difficult to meet the needs of real-time processing of larger data volumes in the future. In addition, the signal processing algorithm software is large in scale, and frequent updates and iterations also put great pressure on the data transmission bandwidth between the satellite and the ground.

[0045] To this end, an embodiment of the present application proposes an optical image processing solution that can quickly process a large number of optical images.

[0046] The present application embodiment discloses a real-time processing method for satellite-borne optical images based on heterogeneous processing, which is applied to an intelligent heterogeneous processing architecture, wherein the intelligent heterogeneous processing architecture includes an FPGA and an AI processing unit, see Figure 1 As shown, the method includes:

[0047] Step S11: acquiring an optical image through the FPGA, performing image preprocessing on the optical image to obtain a preprocessed image, and transmitting the preprocessed image to the AI ​​processing unit through a high-speed serial bus.

[0048] See also Figure 2As shown in the figure, it is a schematic diagram of an intelligent heterogeneous processing architecture; in the figure, the architecture is composed of FPGA+AI, in which the FPGA chip is a data transfer station, which receives images and auxiliary data from the interface module, performs image preprocessing, and forwards the processing results to the AI ​​signal processing unit through a high-speed interface. The AI ​​signal processing unit is mainly composed of a domestic high-performance AI chip, including an NPU acceleration core and an ARM core (not shown), which is responsible for data analysis and processing, and performs subsequent target detection and classification recognition on the image through an intelligent algorithm, and packages the processing results and returns them to the FPGA through a high-speed serial bus. The present invention makes full use of the rich BRAM (bipolar random access memory, bipolar random access memory) (MRAM, Magnetic Random Access Memory, non-volatile magnetic random access memory support implementation) and DSPSlice resources of FPGA and the parallel acceleration computing capability of AI chips, and adopts "pipeline" computing to achieve real-time high-precision target detection and recognition. It should be pointed out that the FPGA obtains optical images from the GTH master / standby interface, and power management is used to provide power for the intelligent heterogeneous processing architecture. EMIF is an external memory interface, and SRIO is for A new generation of high-speed interconnection technology with high reliability, high performance and packet switching is proposed. UART is a universal asynchronous receiver and transmitter.

[0049] In this embodiment, the FPGA acquires an optical image including auxiliary data.

[0050] In this embodiment, the image preprocessing of the optical image to obtain the preprocessed image includes: using a bit width conversion algorithm to convert the bit width of the optical image into the input bit width of the target detection model, and performing image preprocessing on the bit width adjusted image to obtain the preprocessed image; the image preprocessing includes cropping and normalization.

[0051] It should be pointed out that the input bit width of the target detection model is set to 8 bits, so the image data needs to be parsed and the bit width conversion algorithm is used to convert the bit width to 8 bits. Then, the image can be preprocessed on this basis, such as cropping and normalization, to improve the target reasoning accuracy.

[0052] It should be pointed out that before the present application uses a bit width conversion algorithm to convert the bit width of the optical image into the input bit width of the target detection model, it also includes: parsing the image and auxiliary data according to the data transmission protocol, verifying the correctness of the image and data, and discarding erroneous images and data.

[0053] In this embodiment, transmitting the preprocessed image to the AI ​​processing unit via a high-speed serial bus includes: transmitting the preprocessed image to the AI ​​processing unit via a high-speed serial bus, and storing the preprocessed image in the memory space of the AI ​​chip.

[0054] It should be noted that storing the pre-processed image in the AI ​​chip memory space can improve the image reading and writing efficiency so as to improve the subsequent image detection efficiency. It should be noted that in order to avoid the unprocessed image being overwritten, a "multi-level ping-pong cache" design is used for storage.

[0055] Step S12: Using the neural network acceleration core in the AI ​​processing unit, perform image detection on the preprocessed image based on the trained target detection model to obtain a target detection image containing a target detection frame; the target detection model is a model adapted to the underlying acceleration operator of the embedded AI chip that is built based on Python and obtained through model conversion.

[0056] In this embodiment, the model can be updated. The weight incremental update algorithm can be used to update part of the model of the previous round of target detection model to obtain a new target detection model. That is, when determining the weight of the r-th round model, only the weight of the r-1 round is updated. The specific weight incremental update process is as follows: Figure 3 FIG. 1 is a schematic diagram of a weight incremental update process. In a specific embodiment, the weight incremental update process is as follows:

[0057] Step 1: Model Analysis:

[0058] Model analysis is the basis of incremental update. Through comprehensive evaluation of the model structure, weight, performance, etc. of the r-1 round, the availability and reliability of the model are determined. In addition, the model is diagnosed for errors to identify problems and deficiencies in the model, providing a basis for subsequent incremental design.

[0059] Step 2: Incremental Design:

[0060] According to the new task requirements, update the weight of the r-round model and determine the interface with the r-1-round model.

[0061] Step 3: Incremental Update:

[0062] The model weights are updated using a gradient-based optimization algorithm, and the model structure is adjusted using transfer learning technology. The impact of different weights and structure updates on model accuracy is tested. The weights with the greatest impact on accuracy are selected according to the update rate to upgrade the model. The update formula can be expressed as:

[0063] ;

[0064] in represents the loss function, represents the weight, Indicates the update round. Indicates The weight of the round compared to the previous round, Indicates The training dataset is called “subject to”, which means it is subject to constraints. represents the norm, k represents the update rate, which is related to the communication requirements, and I is the number of model weights. Therefore, when updating the model, only The partial weights and corresponding index information are transmitted to the onboard equipment.

[0065] Step 4: Performance Evaluation:

[0066] The updated model is evaluated and tested for performance, including indicators such as detection rate, false alarm rate, precision, recall rate, etc. If the requirements are not met, the model needs to be optimized and adjusted, and incremental updates and performance evaluations are performed again.

[0067] In this embodiment, the model weights of the previous round of target detection models are updated based on the gradient optimization algorithm, and the model structure of the previous round of target detection models is adjusted using the transfer learning technology to obtain a new round of models; the degree of influence of different model weights on the model accuracy is tested, and the part of the model weights that has the greatest influence on the model accuracy is determined based on the saturation test method, so as to obtain a new target detection model based on the model weight update.

[0068] It should be pointed out that during the development process, the update and iteration speed of the AI ​​model is relatively fast, but due to the limitations of the transmission bandwidth and distance of the onboard equipment, large-scale AI models are difficult to update in time. In order to solve this problem, this project intends to adopt a weight incremental update algorithm. First, a comprehensive evaluation of the r-1 round model is carried out, including model parameters, structure, performance, etc., to find out the problems and deficiencies of the model and provide a basis for subsequent incremental design. Then, the structure of the r-th round model is designed according to the new task requirements, and the interface with the r-1 round model is determined. The r-th round model is trained and tested, the model parameters are updated using a gradient-based optimization algorithm, and the model structure is adjusted using transfer learning technology. And test the impact of different parameter and structure updates on model accuracy. Through saturation testing, the weights that have the greatest impact on accuracy are found, and then when updating the r-th round weights, only the weights that have the greatest impact on accuracy are updated.

[0069] Step S13: The AI ​​processing unit transmits the detected target image to the FPGA via a high-speed serial bus.

[0070] It can be seen that the present application acquires an optical image through the FPGA, performs image preprocessing on the optical image to obtain a preprocessed image, and transmits the preprocessed image to the AI ​​processing unit through a high-speed serial bus; utilizes the neural network acceleration core in the AI ​​processing unit to perform image detection on the preprocessed image based on the trained target detection model to obtain a target detection image containing a target detection frame; the target detection model is a model adapted to the underlying acceleration operator of the embedded AI chip obtained by model conversion based on Python; and transmits the target detection image to the FPGA through a high-speed serial bus through the AI ​​processing unit. It can be seen that the present application designs an intelligent heterogeneous processing architecture including FPGA and AI processing unit, utilizes the parallel acceleration computing capability of the neural network acceleration core in the AI ​​processing unit, improves image processing efficiency, and realizes real-time high-precision target detection.

[0071] The embodiment of the present application discloses a specific method for real-time processing of satellite-borne optical images based on heterogeneous processing. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Figure 4 As shown, specifically including:

[0072] Step S21: acquiring an optical image through the FPGA, performing image preprocessing on the optical image to obtain a preprocessed image, and transmitting the preprocessed image to the AI ​​processing unit through a high-speed serial bus.

[0073] Step S22: Using the neural network acceleration core in the AI ​​processing unit, perform initial detection processing on the preprocessed image through a One-stage algorithm based on the trained target detection model to obtain an initial detected image containing image features; the target detection model is a model that is built based on Python and obtained through model conversion to adapt to the underlying acceleration operator of the embedded AI chip.

[0074] In this embodiment, the development and training of the target detection algorithm are on the server GPU (Graphics Processing Unit), whose underlying architecture is x86, and the compilation framework is generally based on CUDA (parallel computing architecture). For domestic AI chips, their underlying architecture is far different from GPU, so the deployment of the algorithm needs to be customized to adapt to the NPU underlying acceleration operator. In addition to the aforementioned preprocessing of the image, the model also needs to be changed, as described below: The target detection model is the Shenjiang network model, and the target detection algorithm development language on the GPU is generally Python, which has low operating efficiency. In order to adapt to the embedded operating environment and fully mobilize the NPU processing efficiency, it is necessary to convert the Python algorithm into a C / C++ architecture. The AI ​​chip model used in this application is Yulong810A, which has 8 NPU acceleration cores and follows the OpenVX standard at the bottom. The model is converted into an .onnx file through the ONNX tool library, which not only stores the weights of the neural network model, but also stores the structural information of the model and the input and output of each layer in the network and other auxiliary information. Then, based on the YL-ACUITY tool, the input onnx network model is converted, and the model is further quantized and trimmed to generate C code (C language program) based on ovxlib. For operators in the network that are not supported by ACUITY, custom development is performed through the EVIS instruction set to generate instantiated Nodes to achieve NPU acceleration. The forward reasoning results are processed using the post-process algorithm to obtain the final detection results. In summary, this application fully exploits the parallel acceleration efficiency of NPU for detection algorithms, improves the target detection speed, and ensures real-time and efficient processing in multi-target scenarios. In summary, this application fully exploits the parallel acceleration efficiency of NPU for detection algorithms, improves the target detection speed, and ensures real-time and efficient processing in multi-target scenarios.

[0075] Specifically, the neural network acceleration core in the AI ​​processing unit is used to perform initial detection processing on the pre-processed image through a One-stage algorithm based on the trained target detection model to obtain an initial detected image containing image features. It also includes: exporting the model built based on Python on the GPU server into onnx format, and converting the onnx format model through a conversion tool to obtain an edge model adapted to the embedded AI chip; adding the post-processing algorithm to the edge model, and compiling the threshold filtering part of the post-processing algorithm into a neural network node through the underlying instruction set development to obtain the target detection model.

[0076] In a specific embodiment, the model conversion process is as follows:

[0077] Step 1: Model training:

[0078] Develop and train the target detection model on the GPU and export the model based on the onnx file.

[0079] Step 2: Model deployment:

[0080] The model based on the onnx file is input into the YL-ACUITY tool for model conversion, and further quantization and trimming are performed to generate C code. Post-processing code is added, and the threshold filtering part is developed through the EVIS instruction set to generate an instantiated Node as a forward process for NPU acceleration processing.

[0081] Step 3: System Initialization:

[0082] Initialize various interfaces, algorithms, drivers, etc.

[0083] Step S23: Utilizing the neural network acceleration core in the AI ​​processing unit, based on the target detection model, a final detection result is extracted from the image after the initial detection through a post-processing algorithm; the final detection result includes a final target detection frame and a target type corresponding to the final target detection frame.

[0084] In this embodiment, the neural network acceleration core (embedded neural network processor acceleration core) in the AI ​​processing unit is used to extract the final detection result from the image after the initial detection through a post-processing algorithm based on the target detection model; the final detection result includes a final target detection frame and a target type corresponding to the final target detection frame, including: using the neural network acceleration core in the AI ​​processing unit to determine an initial prediction frame in the image after the initial detection based on the target detection model, and determining the confidence of each initial prediction frame, and determining a temporary detection frame whose confidence is greater than a preset threshold from the initial prediction frame through a post-processing algorithm; determining the area of ​​each temporary detection frame and the overlapping area between each temporary detection frame and each other temporary detection frame, and calculating the ratio of the overlapping area to the total area of ​​the two frames to obtain the area intersection and union ratio of the temporary detection frame; using a non-maximum suppression algorithm to remove overlapping redundant frames from the temporary detection frame based on the area intersection and union ratio, and determining a target detection frame to obtain a target detection result including the target detection frame.

[0085] In this embodiment, the non-maximum suppression algorithm is used to remove overlapping redundant frames from the temporary detection frame based on the area intersection and union ratio to determine the target detection frame to obtain a target detection result containing the target detection frame, including: taking the temporary detection frame with the largest area intersection and union ratio in the temporary detection frame set as the detection frame to be processed; removing other temporary detection frames in the temporary detection frame set that have overlapping parts with the detection frame to be processed, and taking the detection frame to be processed as the target detection frame; in the remaining temporary detection frames, jumping to the step of taking the temporary detection frame with the largest area intersection and union ratio in the temporary detection frame set as the detection frame to be processed until the target detection frame is determined to obtain a target detection result containing the target detection frame.

[0086] In a specific embodiment, the multi-core parallel post-process algorithm is introduced as follows: Since the intelligent target detection algorithm cannot determine the number of targets in the input image, the number of inference results is often unknown. The number of these boxes is much larger than the number of annotations, and most of them are stacked together. Therefore, it is necessary to use a post-processing algorithm to select the best from the stacked boxes as the prediction result. However, the ordinary post-process processing flow can only be run through the ARM core, and the NPU acceleration core cannot be called, and the processing speed is very slow. Therefore, the present invention designs a post-process algorithm based on NPU hardware parallel acceleration, which splits the post-processing process into two parts: (box represents the detection box)

[0087] The first part is grid division and threshold filtering. The grid is divided according to the size of the feature-map, the three-dimensional image features are converted into two-dimensional boxes and confidence results, and the boxes greater than the threshold are retained. This process is usually processed through ARM, but due to the large amount of predicted boxes data, the ARM computing power is low, and the operation time cost is high. In addition, the data needs to be moved from the NPU to the ARM for ARM calculation, and the data communication cost is very high, resulting in a large processing delay. In order to call the NPU acceleration core, the present invention compiles the feature conversion into a neural network node, and inserts it into the model as a forward process for processing. Compared with ARM processing, the NPU has 8 cores for parallel processing, and saves a lot of data from the NPU to the ARM, greatly speeding up the post-process.

[0088] The second part is to calculate IoU (Intersection of Union) and non-maximum suppression (NMS). First, for each filtered box, calculate the intersection area and total area of ​​the box with the remaining boxes, and then get IoU = Intersection / Union. Then take the box with the highest intersection of union and use it as the target box, remove the boxes that overlap with it, and then continue to take the box with the highest intersection of union from the remaining boxes to get the second target box, and repeat the above steps until there are no overlapping boxes.

[0089] The specific multi-core parallel post-processing algorithm process is as follows Figure 5 As shown in the figure, the prediction box (preset detection box) is first obtained, and then 8 NPUs are used to perform threshold filtering in parallel to obtain the filtered candidate box (temporary detection box). After that, the data is moved to ARM, and the intersection-over-union (IOU) calculation and non-maximum suppression (NMS) operation are completed in ARM to obtain the final target detection box.

[0090] In summary, this application designs a post-process algorithm, which uses NPU to accelerate the parallel processing of the check box, greatly improving the timeliness of post-processing.

[0091] Step S24: The AI ​​processing unit transmits the detected target image to the FPGA via a high-speed serial bus.

[0092] It can be seen that the present application acquires an optical image through the FPGA, performs image preprocessing on the optical image to obtain a preprocessed image, and transmits the preprocessed image to the AI ​​processing unit through a high-speed serial bus; utilizes the neural network acceleration core in the AI ​​processing unit, and performs initial detection processing on the preprocessed image based on the trained target detection model through a One-stage algorithm to obtain an initial detected image containing image features; the target detection model is a model adapted to the underlying acceleration operator of the embedded AI chip obtained by building it based on Python and through model conversion; utilizes the neural network acceleration core in the AI ​​processing unit, and extracts the final detection result from the initial detected image through a post-processing algorithm based on the target detection model; the final detection result includes a final target detection box and the target type corresponding to the final target detection box; and transmits the target detection image to the FPGA through a high-speed serial bus through the AI ​​processing unit. It can be seen that the present application utilizes a pre-built target detection model to perform initial detection processing on the pre-processed image through a One-stage target detection algorithm, giving full play to the parallel acceleration efficiency of the NPU for the detection algorithm, improving the target detection speed, and ensuring real-time and efficient processing in multi-target scenarios. The present application utilizes the neural network acceleration core in the AI ​​processing unit to complete post-processing based on the pre-built target detection model through a post-processing algorithm, thereby improving the timeliness of post-processing.

[0093] In summary, this application proposes a method for real-time processing of massive satellite image data based on AI heterogeneous processors. The method mainly includes three parts: first, design an intelligent heterogeneous processing architecture of high-computing power domestic AI chip + FPGA, in which FPGA receives serial optical image data and auxiliary data through a high-speed serial bus, and the AI ​​chip performs deployment reasoning calculation of the intelligent model and outputs the target detection result; second, based on the NPU operator design, deploy a high-precision one-stage target detection algorithm to improve the algorithm's reasoning efficiency on the embedded platform; in addition, design a post-process algorithm accelerated by NPU hardware to improve the running speed of the post-processing process; finally, design a weight incremental update algorithm to reduce the bandwidth pressure of satellite-to-ground data transmission. The following effects are achieved: 1. The AI+FPGA heterogeneous processing architecture is designed, and the rich BRAM and DSPSlice resources of FPGA and the parallel acceleration computing capabilities of AI are utilized. "Pipeline" computing is adopted to achieve real-time high-precision target detection; 2. The NPU-accelerated one-stage target detection algorithm is designed. Fully exploit the parallel acceleration efficiency of NPU for the detection algorithm, improve the target detection speed, and ensure real-time and efficient processing in multi-target scenarios. 3. Designed a multi-core parallel post-process algorithm. Converted the process of filtering the box traversal threshold in post-processing into a node as part of the forward process, and used the NPU to accelerate the parallel processing, which greatly improved the timeliness of post-processing. 4. Designed a weight incremental update algorithm. When determining the weight of the r-th round model, only the partial weight of the r-1 round is updated, which greatly reduces the pressure of satellite-to-ground data transmission.

[0094] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0095] The above is a detailed introduction to a real-time processing method for satellite optical images based on heterogeneous processing provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for real-time processing of spaceborne optical images based on heterogeneous processing, characterized in that: Applied to an intelligent heterogeneous processing architecture, the intelligent heterogeneous processing architecture includes an FPGA and an AI processing unit, and the method includes: Acquire an optical image through the FPGA, perform image preprocessing on the optical image to obtain a preprocessed image, and transmit the preprocessed image to the AI ​​processing unit through a high-speed serial bus; Using the neural network acceleration core in the AI ​​processing unit, image detection is performed on the preprocessed image based on the trained target detection model to obtain a target detection image containing a target detection frame; the target detection model is a model adapted to the underlying acceleration operator of the embedded AI chip that is built based on Python and obtained through model conversion; The AI ​​processing unit transmits the detected target image to the FPGA via a high-speed serial bus; Wherein, the real-time processing method of spaceborne optical images based on heterogeneous processing further includes: The weight incremental update algorithm is used to update some model weights of the previous target detection model to obtain a new target detection model; Among them, the update formula of the weight incremental update algorithm is: ； in, represents the loss function; represents weight; Indicates the round of this update; Indicates The weight of the round compared to the previous round; Indicates Dataset for round training; represents the update rate, which is related to the communication demand; represents the total number of model weights; The full name is subject to, which means subject to constraints; Represents the norm.

2. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 1, characterized in that: The using the neural network acceleration core in the AI ​​processing unit to perform image detection on the preprocessed image based on the trained target detection model to obtain a target detected image containing a target detection frame includes: Using the neural network acceleration core in the AI ​​processing unit, based on the trained target detection model, the pre-processed image is initially detected by using a one-stage algorithm to obtain an initial detected image containing image features; Utilizing the neural network acceleration core in the AI ​​processing unit, a final detection result is extracted from the image after initial detection through a post-processing algorithm based on the target detection model; the final detection result includes a final target detection frame and a target type corresponding to the final target detection frame.

3. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 2, characterized in that: Before the neural network acceleration core in the AI ​​processing unit is used to perform initial detection processing on the preprocessed image through a One-stage algorithm based on a trained target detection model to obtain an initial detected image containing image features, the method further includes: Export the model built based on Python on the GPU server to onnx format, and use the conversion tool to convert the onnx model to an edge model adapted to the embedded AI chip; The post-processing algorithm is added to the edge model, and the threshold filtering part of the post-processing algorithm is developed and compiled into a neural network node through the underlying instruction set to obtain the target detection model.

4. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 2, characterized in that: The neural network acceleration core in the AI ​​processing unit is used to extract the final detection result from the image after the initial detection through a post-processing algorithm based on the target detection model; The final detection result includes a final target detection frame and a target type corresponding to the final target detection frame, including: Using the neural network acceleration core in the AI ​​processing unit, based on the target detection model, an initial prediction box in the image after the initial detection is determined, and the confidence of each of the initial prediction boxes is determined, and a temporary detection box whose confidence is greater than a preset threshold is determined from the initial prediction box through a post-processing algorithm; Determine the area of ​​each temporary detection frame and the overlapping area between each temporary detection frame and each other temporary detection frame, and calculate the ratio of the overlapping area to the total area of ​​the two frames to obtain the area intersection and union ratio of the temporary detection frame; A non-maximum suppression algorithm is used to remove overlapping redundant frames from the temporary detection frame based on the area intersection-union ratio, and a target detection frame is determined to obtain a target detection result including the target detection frame.

5. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 4, characterized in that: The method of removing overlapping redundant frames from the temporary detection frame based on the area intersection-to-union ratio by using a non-maximum suppression algorithm to determine a target detection frame to obtain a target detection result including the target detection frame includes: The temporary detection frame with the largest area intersection-and-union ratio in the temporary detection frame set is used as the detection frame to be processed; Removing other temporary detection frames in the temporary detection frame set that overlap with the detection frame to be processed, and using the detection frame to be processed as the target detection frame; Jump to the step of taking the temporary detection frame with the largest area intersection-union ratio in the temporary detection frame set as the detection frame to be processed, until the target detection frame is determined, so as to obtain the target detection result including the target detection frame.

6. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 1, characterized in that: The performing image preprocessing on the optical image to obtain a preprocessed image comprises: The bit width of the optical image is converted into the input bit width of the target detection model by using a bit width conversion algorithm, and the image after the bit width adjustment is preprocessed to obtain a preprocessed image; the image preprocessing includes cropping and normalization.

7. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 1, characterized in that: The step of transmitting the pre-processed image to the AI ​​processing unit via a high-speed serial bus includes: The preprocessed image is transmitted to the AI ​​processing unit via a high-speed serial bus, and the preprocessed image is stored in the memory space of the AI ​​chip.

8. The method for real-time processing of spaceborne optical images based on heterogeneous processing according to claim 1, characterized in that: The method of using a weight incremental update algorithm to update some model weights of a previous target detection model to obtain a new target detection model includes: The model weights of the previous round of target detection models are updated based on the gradient optimization algorithm, and the model structure of the previous round of target detection models is adjusted using the transfer learning technology to obtain a new round of models; The influence of different model weights on the model accuracy is tested, and the partial model weights with the greatest influence on the model accuracy are determined based on a saturation test method, so as to obtain a new target detection model based on the model weight update.

Images