Efficient remote sensing image processing method based on algorithm-hardware collaborative design

By building a hybrid heterogeneous computing platform in an edge computing environment, combining the lightweight neural network model after quantization and pruning and historical information fusion strategy, the problems of resource waste and performance bottlenecks are solved, and efficient, accurate and low-power are achieved. Flood detection.

CN118918132APending Publication Date: 2024-11-08TSINGHUA UNIVERSITY

Patent Information

Application Number
CN202410929342.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In edge computing environments with limited resources, traditional software algorithm design ignores hardware architecture details, resulting in resource waste and performance bottlenecks in flood detection, making it difficult to achieve efficient flood detection.

Method used

Using an algorithm-hardware collaborative design method, a hybrid heterogeneous computing platform for CPU, GPU and FPGA is constructed. The flooded areas are quickly extracted on the FPGA development board through quantization and pruning lightweight neural network model, and the CPU is used to utilize the CPU- GPU edge computing devices perform historical information fusion to optimize flood detection results.

Benefits of technology

Accurate, fast and low-power flood detection of multi-source remote sensing images is achieved, the hardware potential is fully utilized, and resource consumption and delay are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918132A_ABST
    Figure CN118918132A_ABST
Patent Text Reader

Abstract

The invention discloses an efficient remote sensing image processing method based on algorithm-hardware collaborative design, the method is applied to an FPGA development board and CPU-GPU edge computing equipment, a rapid flood area extraction module is arranged in a main chip of the FPGA development board, and the rapid flood area extraction module adopts a lightweight neural network model. A historical information fusion module is arranged in the CPU-GPU edge computing device; acquiring a slice image of the multi-source remote sensing image; inputting the slice image into a quantized and pruned lightweight neural network model for data processing so as to output a preliminary flood extraction result; and inputting the preliminary flood extraction result into a historical information fusion module, and optimizing a current image flood extraction result by using a historical image flood extraction result and a plurality of decision-level fusion strategies to output a final flood detection result. The method can be applied to a resource-limited edge computing environment, and accurate, rapid and low-power-consumption flood detection of the multi-source remote sensing image can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to remote sensing image processing technology and edge computing environment optimization technology, and in particular to an efficient remote sensing image processing method based on algorithm-hardware collaborative design. Background Art

[0002] Flood disasters are frequent natural disasters around the world, causing huge losses of life and property to human society. In order to conduct flood analysis and disaster assessment more effectively, with the continuous advancement of satellite remote sensing technology, multi-source satellite remote sensing images have become an important tool in the field of flood detection with their high resolution, high revisit rate and wide coverage.

[0003] In the evolution of flood detection technology for multi-source satellite images, a variety of methods have been explored and developed. For example, some scholars have combined three different traditional threshold-based algorithms to more accurately depict flood areas in multi-source remote sensing images. Compared with methods that rely solely on a single image source, this method shows significant advantages. However, it cannot be ignored that these threshold-based methods are often interfered by noise factors such as terrain shadows and cloud occlusion during processing, which to a certain extent affects the accuracy of flood detection. In recent years, the application of convolutional neural networks (CNNs) in the field of flood detection has received increasing attention, thanks to CNN's excellent feature extraction and self-learning capabilities, as well as its adaptability to various complex environments. Some scholars have proposed a multi-source remote sensing image change detection method based on the classic Unet architecture. At the same time, in order to improve the accuracy of flood detection, information fusion technology has also been widely used. For example, some scholars have innovatively proposed an improved version of the ResNet-50 model that uses the joint features of multi-source satellite images in combination with multi-source satellite image time series; other studies have adopted a strategy that combines a lightweight Unet architecture with decision-level data fusion to achieve fast and accurate flood detection, thereby optimizing overall performance.

[0004] With the surge in the amount of satellite remote sensing data, traditional ground data processing centers are facing huge data transmission and processing pressure, which greatly limits the real-time nature of flood detection results. The rise of on-orbit computing technology has provided a new solution for real-time flood detection, in which satellite-borne equipment can directly process satellite data and only transmit a small amount of processing results back to the ground, significantly reducing the pressure on data transmission and ground processing systems. However, the computing power and power resources of satellite-borne equipment are limited, making it difficult to meet the needs of complex flood detection tasks. Although some studies have achieved fast water body extraction and identification on high-performance edge computing devices, these methods mainly focus on the application of high-performance hardware and ignore the collaborative design of algorithms and hardware.

[0005] In modern high-performance general-purpose processor environments, software algorithm design usually does not consider hardware architecture details, which results in the algorithm being unable to fully utilize the hardware potential, resulting in resource waste and performance bottlenecks. Therefore, in resource-limited edge computing environments, how to efficiently perform flood detection remains an important technical challenge. Summary of the invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, the present invention proposes an efficient remote sensing image processing method based on algorithm-hardware co-design. By constructing a hybrid heterogeneous computing platform that integrates central processing unit (CPU), graphics processing unit (GPU) and field programmable gate array (FPGA) hardware, and performing multiple algorithm optimization strategies based on the characteristics of the hardware platform, the performance of flood detection is significantly improved.

[0008] To achieve the above-mentioned purpose, the efficient remote sensing image processing method based on algorithm-hardware co-design of the present invention is applied to an FPGA development board and a CPU-GPU edge computing device, wherein a fast flood area extraction module is set in the main chip of the FPGA development board, and the fast flood area extraction module adopts a lightweight neural network model, and a historical information fusion module is set in the CPU-GPU edge computing device, and the method comprises:

[0009] Acquire slice images of multi-source remote sensing images;

[0010] Inputting the slice image into a quantized and pruned lightweight neural network model for data processing to output a preliminary flood extraction result; wherein the flood extraction result includes identifying flood-inundated and non-flood-inundated pixel areas from the slice image;

[0011] The preliminary flood extraction results are input into the historical information fusion module, and the historical image flood extraction results and multiple decision-level fusion strategies are used to optimize the current image flood extraction results to output the final flood detection results.

[0012] The efficient remote sensing image processing method based on algorithm-hardware co-design in the embodiment of the present invention may also have the following additional technical features:

[0013] In one embodiment of the present invention, obtaining a slice image of a multi-source remote sensing image includes:

[0014] The multi-source remote sensing images are collected by front-end sensors, and the multi-source remote sensing images are sliced ​​to obtain slice images; wherein the multi-source remote sensing images include current remote sensing images and historical remote sensing images;

[0015] The slice image is received through a preset network port using an FPGA development board.

[0016] In one embodiment of the present invention, the slice image is input into a quantized and pruned lightweight neural network model for data processing to output a preliminary flood extraction result, including:

[0017] Performing a lightweight operation on the neural network model to obtain a lightweight neural network model; wherein the lightweight operation includes model parameter quantization and model pruning operations;

[0018] The slice images of the multi-source remote sensing images are input into the lightweight neural network model to output the probability value of a certain area being flooded, and the probability value is processed by threshold segmentation to output a flood extraction result; wherein the result value of the flood extraction result indicates whether the current pixel is a flood area.

[0019] In one embodiment of the present invention, the rapid flood area extraction module includes a first CBP network structure, a BP neural network structure and a second CBP network structure.

[0020] In one embodiment of the present invention, data fusion is performed on the convolutional layer and the batch normalization layer in the network of the rapid flood area extraction module, including:

[0021] The processing process of the convolution layer (Equation (1)) and the processing process of the batch normalization layer (Equation (2)) are combined into one process, including equations (3), (4), and (5):

[0022] y conv =w·x+b (1)

[0023]

[0024] y=w′·x+b' (3)

[0025]

[0026] Where x and y represent input and output respectively, w, b, γ, ε and β are all the parameters in the calculation, μ and σ 2 Represents the mean and variance of the input.

[0027] In one embodiment of the present invention, the historical information fusion module includes a GPU and a CPU; the preliminary flood extraction result is input into the historical information fusion module, and the current image flood extraction result is optimized by using the historical image flood extraction result and a plurality of decision-level fusion strategies to output the final flood detection result, including:

[0028] The processing unit of the CPU receives the binary classification result of the flood extraction result corresponding to the output of the lightweight neural network model, and determines the flood stage of the current image by counting pixels to determine the corresponding fusion strategy; wherein the binary classification result includes the flood extraction result of the current image and the flood extraction result of the historical image;

[0029] The binary classification result and the fusion strategy are input into the GPU to perform information fusion calculation to obtain an information fusion result, and the information fusion result is processed through data to obtain a final flood detection result.

[0030] In one embodiment of the present invention, after obtaining the final flood detection result, the method further includes:

[0031] The final flood detection result is sent to the user terminal after being processed by graphics.

[0032] In one embodiment of the present invention, a Gigabit Ethernet cable is used to connect the FPGA development board and the CPU-GPU edge computing device, and data is directly exchanged with external devices via Ethernet using the UDP / IP protocol.

[0033] In one embodiment of the present invention, the model parameters of the neural network model are quantized so that the original 64-bit floating point numbers are converted into 16-bit fixed point numbers.

[0034] In one embodiment of the present invention, performing lightweight operations on a neural network model includes:

[0035] The calculation channel and the maximum value index operations used to determine the non-flood area in the original neural network model are removed to modify the network model architecture to obtain a simplified single-channel model.

[0036] The efficient remote sensing image processing method based on algorithm-hardware co-design in the embodiment of the present invention, through the co-design of algorithm and hardware and based on the existing flood detection method lightweight historical information fusion network (LHIFN), constructs a hybrid heterogeneous computing platform integrating CPU, GPU and FPGA, thereby realizing accurate, fast and low-power flood detection of multi-source remote sensing images.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1is a flow chart of an efficient remote sensing image processing method based on algorithm-hardware co-design according to an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of the LHIFN algorithm structure of an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a system framework of a flood detection method according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of the FPGA module design of an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of the model pruning process of an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] The following describes an efficient remote sensing image processing method based on algorithm-hardware co-design according to an embodiment of the present invention with reference to the accompanying drawings.

[0047] Figure 1 1 is a flow chart of an efficient remote sensing image processing method based on algorithm-hardware co-design according to an embodiment of the present invention. Figure 1 As shown, the method is applied to an FPGA development board and a CPU-GPU edge computing device, wherein a fast flood area extraction module is set in the main chip of the FPGA development board, the fast flood area extraction module adopts a lightweight neural network model, and a historical information fusion module is set in the CPU-GPU edge computing device. The method includes:

[0048] S1, obtain slice images of multi-source remote sensing images;

[0049] S2, inputting the slice image into a quantized and pruned lightweight neural network model for data processing to output a preliminary flood extraction result; wherein the flood extraction result includes identifying flood-inundated and non-flood-inundated pixel areas from the slice image;

[0050] S3, input the preliminary flood extraction results into the historical information fusion module, and use the historical image flood extraction results and multiple decision-level fusion strategies to optimize the current image flood extraction results to output the final flood detection results.

[0051] Specifically, the LHIFN model structure on which the present invention is based includes a fast flood area extraction (SFE) module and a historical information fusion (HIF) module as its core. Among them, the SFE module adopts a lightweight neural network model (LNNM), which can quickly identify flooded and non-flooded pixel areas from time-series multi-source images. The HIF module optimizes the current image according to the current image flood status by integrating the flood extraction results of multi-source images and the binary classification results of the SFE module, using a variety of decision-level fusion strategies and image processing methods to improve the accuracy of flood detection. Finally, the flood detection results are output. The structure of LHIFN is as follows: Figure 2 As shown, (a, b) in the SFE module means: a represents the number of input channels, and b represents the number of output channels.

[0052] Based on the efficient hardware design of flood detection algorithm, the present invention can use MLK MK7325FC FPGA development board and Nvidia Jetson AGX Orin to deploy efficient flood detection method. System architecture is as follows Figure 3 As shown. The MLK MK7325FCFPGA development board is equipped with the Xilinx Kintex-7 main chip. Jetson AGX Orin is a compact edge computing device that uses the NVIDIA Ampere architecture GPU and Cortex-A78AE CPU. Table 1 and Table 2 show the configurations of the above two hardware components respectively.

[0053] In the flood detection architecture proposed in the present invention, the MLK MK7325FC FPGA development board and the Nvidia JetsonAGX Orin are connected using a Gigabit Ethernet cable. In addition, both hardware components can directly exchange data with external devices using the UDP / IP protocol via Ethernet.

[0054] Nvidia Jetson AGX ORIN is responsible for executing host software programs, data flow, and historical information fusion modules. Its operating system is Ubuntu 20.04. Nvidia Jetson AGX ORIN supports multiple optimized power budgets, such as 15W, 30W, 50W, and MAXN. In order to limit power consumption, the present invention uses the lowest 15W power budget. The process is implemented in Python. The fusion algorithm is implemented on the GPU using Nvidia's CUDA programming model.

[0055] The MLK MK7325FC FPGA development board is used to implement the LNNM-based SFE module. The processor is implemented in Verilog language. The development tool used is Xilinx Vivado DesignSuite 2023.1 configured for the target device XC7Z020-CLG484-1. After describing the processor in Verilog language, synthesis and simulation-based verification are performed to ensure the accuracy of the code. The module design of the FPGA is as follows Figure 4 shown.

[0056] In summary, the present invention achieves efficient operation and minimized power consumption of the flood detection system by reasonably allocating computing tasks to different hardware platforms.

[0057] Table 1

[0058]

[0059]

[0060] Table 2

[0061]

[0062] The present invention describes in detail the efficient remote sensing image processing method based on algorithm-hardware co-design according to the embodiment of the present invention in conjunction with the accompanying drawings.

[0063] First, the present invention collects multi-source remote sensing images through a front-end sensor, and slices the multi-source remote sensing images to obtain slice images; wherein the multi-source remote sensing images include current remote sensing images and historical remote sensing images; and the slice images are received through a preset network port using an FPGA development board.

[0064] As we know, the SFE module is implemented based on FPGA technology. The network reasoning in the algorithm has the characteristics of fixed calculation steps, sequential execution, and relatively few processing parameters involved. FPGA is an ideal hardware choice for performing such tasks. Specifically, FPGA is suitable for real-time image processing, which is mainly due to its excellent pipeline parallel computing capabilities and optimized control of block random access memory (BRAM). In addition, the pipeline parallel operation capability of FPGA can efficiently manage the data dependencies between network levels, ensure the stable flow of data flow, thereby shortening processing time and maintaining low latency. Secondly, the programmability of FPGA has significant advantages in power consumption. Through the close integration of algorithms and hardware, FPGA can reduce the exchange of instructions and data, thereby improving overall performance while reducing energy consumption.

[0065] Specifically, the FPGA development board receives remote sensing images sent from the front-end sensor through the equipped network port. The images have been pre-processed and segmented into image slices of size 512 × 512. The images include current remote sensing images and historical remote sensing images.

[0066] Secondly, the image slices are sent to the lightweight neural network model LNNM in the present invention. The inference module has undergone optimization operations such as model parameter quantization and model pruning. The model has been trained on a real remote sensing image training set in advance. The inference module outputs a detection result of size 512×512, and the result value indicates whether the current pixel is a flood area, which is used in the subsequent information fusion module.

[0067] like Figure 2 As shown, the rapid flood area extraction module includes a first CBP network structure, a BP neural network structure and a second CBP network structure.

[0068] It is understandable that for tasks that do not require high numerical precision, the present invention reduces storage requirements and calculation times through data compression technology.

[0069] Exemplarily, a lightweight neural network model is obtained by performing lightweight operations on the neural network model; wherein the lightweight operations include model parameter quantization and model pruning operations; slice images of multi-source remote sensing images are input into the lightweight neural network model to output a probability value of a certain area being flooded, and the probability value is processed by threshold segmentation to output a flood extraction result; wherein the result value (0 or 1) of the flood extraction result indicates whether the current pixel is a flood area.

[0070] The present invention quantizes the model parameters of the SFE module, converts the original 64-bit floating point number into a 16-bit fixed point number, and deeply explores the influence of different decimal part bit widths in fixed point representation on the detection accuracy. By analyzing the experimental results, when the decimal part bit width is set to 9 bits, the detection accuracy reaches the best, and the deviation is only 0.2% compared with the network using floating point operations. Therefore, in the present invention, 9 bits are selected as the decimal part bit width.

[0071] Among them, in the model pruning stage, the last step of the original LNNM model, the 'maximum index' operation is responsible for comparing the outputs of the upper dual channels and outputting the maximum index as the detection result. Since there are a large number of cases where the two output results of the upper channel are relatively close, the method of taking the maximum index value will seriously affect the confidence of the detection result. Figure 5 As shown in the figure, the network architecture is modified by removing the computational channel used to determine non-flooded areas and the 'max index' operation. Instead, a simplified single-channel model is constructed that directly outputs the probability of an area being flooded. These probability values ​​are processed through threshold segmentation to output the final flood detection results. This model pruning strategy successfully halves the number of inference calculations and parameters; the threshold is generated during training.

[0072] Therefore, it can be seen that, in view of the redundancy in the LNNM model, the present invention reduces unnecessary calculations and access operations through model pruning technology. By removing some calculation channels and operations, the network architecture is modified to construct a simplified single-channel model, which can directly output the probability of a certain area being flooded. These probability values ​​are processed by threshold segmentation to output the final flood detection results. This model pruning strategy successfully reduces the number of calculations and parameters.

[0073] Furthermore, by optimizing the algorithm flow, the present invention systematically eliminates redundant intermediate steps and repeated calculations, thereby significantly improving the overall computing efficiency. In the model reasoning process of the SFE module, the convolutional layer and the batch normalization layer are fused. This optimization strategy effectively reduces the amount of calculation and memory consumption.

[0074] Specifically, in the model inference process of the SFE module, the convolution layer and batch normalization layer in the network are fused. Specifically, the convolution layer (as shown in formula (1)) and the batch normalization layer (as shown in formula (2)) were originally two independent calculation processes, but now they can be merged into one process (as shown in formulas (3), (4), and (5)). This optimization strategy effectively reduces the amount of calculation and memory consumption.

[0075] y conv =w·x+b (1)

[0076]

[0077] y=w′·x+b' (3)

[0078]

[0079] Where x and y represent input and output respectively, w, b, γ, ε and β are all the parameters in the calculation, μ and σ 2 Represents the mean and variance of the input.

[0080] In the calculation process of the embodiment of the present invention, the mutual dependence between data often leads to significant delays between subtasks, which in turn affects parallelization and overall running speed. In order to eliminate this data dependence, the present invention adopts a numerical approximation method to reduce running time and resource waste. In LHIFN, some parameters originally need to be dynamically calculated based on the input data. The present invention determines and fixes these parameters by using a small data set in advance. Experimental results show that this method has little impact on the detection results while effectively reducing the number of calculations.

[0081] Therefore, in LHIFN, μ and σ in equation (2) are 2 Originally, it was necessary to dynamically calculate according to the input data. The present invention determines and fixes these parameters by using a small data set in advance. By analyzing the experimental results, the parameters are fixed by selecting the values ​​that have the least impact on the detection results.

[0082] It is known that the SFE module is implemented based on FPGA technology. The network reasoning in the algorithm has the characteristics of fixed calculation steps, sequential execution, and relatively few processing parameters involved. FPGA is an ideal hardware choice for performing such tasks. Specifically, FPGA is suitable for real-time image processing, which is mainly due to its excellent pipeline parallel computing capabilities and optimized control of block random access memory (BRAM). In addition, the pipeline parallel operation capability of FPGA can efficiently manage the data dependencies between network levels, ensure the stable flow of data flow, thereby shortening processing time and maintaining low latency. Compared with CPU and GPU, the programmability of FPGA has significant advantages in power consumption. Through the close integration of algorithms and hardware, FPGA can reduce the exchange of instructions and data, thereby improving overall performance while reducing energy consumption.

[0083] Furthermore, the CPU-based processing module of the present invention receives the flood area detection result output by the FPGA development board transmitted by the network cable, determines the flood stage of the current image by counting pixels, and selects the corresponding fusion strategy for fusion calculation. The fusion strategy and the image after CPU judgment and statistics are sent to the GPU for information fusion calculation. The fused result is processed by a simple model post-processing (can be linearly stretched and intercepted) to obtain the final flood detection result.

[0084] It is known that the CPU is the core of the host software program, and it is responsible for many tasks such as program flow control, logical operations, and data flow. In terms of processing serial instructions, the CPU performs particularly well, fully demonstrating its excellent instruction-level parallel processing capabilities.

[0085] As we know, the HIF module is processed by the GPU, which involves complex matrix calculations and the processing of a large amount of multi-source data. Given that the computational requirements of this module exceed the processing capabilities of the CPU and require the processing of intensive parallel computing tasks, the GPU is selected as the execution platform. The GPU can efficiently integrate all historical data when processing tasks with non-pipeline structures, which is an advantage that FPGAs do not have. At the same time, considering the limitations of FPGA resource capacity and the efficiency of data transmission between the CPU and FPGA, after comprehensive evaluation, it was determined that the GPU combined with NVIDIA's Compute Unified Device Architecture (CUDA) is the best solution to accelerate the processing of the HIF module.

[0086] Specifically, a processing unit of a CPU is used to receive a binary classification result of a flood extraction result corresponding to an output of a lightweight neural network model, and the flood stage of a current image is judged by counting pixels to determine a corresponding fusion strategy; wherein the binary classification result includes a flood extraction result of a current image and a flood extraction result of a historical image; the binary classification result and the fusion strategy are input into a GPU for information fusion calculation to obtain an information fusion result, and the information fusion result is processed through data to obtain a final flood detection result.

[0087] Afterwards, the final flood result after information fusion can be sent to other user terminals after graphics processing using the CPU.

[0088] Therefore, the present invention implements optimized algorithms on different hardware platforms to give full play to the advantages of each hardware and improve the performance and efficiency of the overall system.

[0089] Specifically, on the CPU, the present invention uses Python language to write the host program. Python language is concise and easy to understand, has rich libraries and powerful data processing capabilities, and is very suitable for writing codes for program flow control, logical calculations, and data flow. Through Python language, the present invention realizes flexible control of the entire system and efficient data processing.

[0090] On FPGA, the present invention uses Verilog language to implement the SFE module in detail. As a hardware description language, Verilog language can accurately describe the structure and behavior of digital circuits and is very suitable for FPGA programming. Through Verilog language, the present invention implements the logical function of the SFE module and fully utilizes the parallel processing capability of FPGA to improve the data processing speed.

[0091] On the GPU, the present invention combines Python language and CUDA framework to implement the HIF calculation process. CUDA framework is a parallel computing platform and API launched by NVIDIA, which allows developers to use GPU for high-performance computing. By calling the CUDA framework through Python language, the parallel computing capability of GPU is fully utilized, the HIF calculation process is accelerated, and the overall performance of the system is further improved.

[0092] The efficient remote sensing image processing method based on algorithm-hardware co-design proposed in the present invention optimizes the hardware to reduce the demand for computing resources and memory access, while ensuring that the algorithm structure is concise and orderly. At the same time, the present invention utilizes multiple strategies such as numerical quantization, model pruning, elimination of data dependence and process optimization to improve the execution efficiency of the algorithm on efficient hardware and ensure that the detection performance is not significantly affected. Ultimately, by reasonably allocating computing tasks to different hardware platforms, the efficient operation and power consumption of the flood detection system are minimized, and accurate, fast and low-power flood detection of multi-source remote sensing images is achieved.

[0093] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0094] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. An efficient remote sensing image processing method based on algorithm-hardware co-design, characterized in that: Applied to an FPGA development board and a CPU-GPU edge computing device, wherein a fast flood area extraction module is set in the main chip of the FPGA development board, the fast flood area extraction module adopts a lightweight neural network model, and a historical information fusion module is set in the CPU-GPU edge computing device. The method includes: Acquire slice images of multi-source remote sensing images; Inputting the slice image into a quantized and pruned lightweight neural network model for data processing to output a preliminary flood extraction result; wherein the flood extraction result includes identifying flood-inundated and non-flood-inundated pixel areas from the slice image; The preliminary flood extraction results are input into the historical information fusion module, and the historical image flood extraction results and multiple decision-level fusion strategies are used to optimize the current image flood extraction results to output the final flood detection results.

2. The method according to claim 1, characterized in that Acquire slice images of multi-source remote sensing images, including: The multi-source remote sensing images are collected by front-end sensors, and the multi-source remote sensing images are sliced ​​to obtain slice images; wherein the multi-source remote sensing images include current remote sensing images and historical remote sensing images; The slice image is received through a preset network port using an FPGA development board.

3. The method according to claim 1, characterized in that The slice image is input into the quantized and pruned lightweight neural network model for data processing to output a preliminary flood extraction result, including: Performing a lightweight operation on the neural network model to obtain a lightweight neural network model; wherein the lightweight operation includes model parameter quantization and model pruning operations; The slice images of the multi-source remote sensing images are input into the lightweight neural network model to output the probability value of a certain area being flooded, and the probability value is processed by threshold segmentation to output a flood extraction result; wherein the result value of the flood extraction result indicates whether the current pixel is a flood area.

4. The method according to claim 3, characterized in that The rapid flood area extraction module includes a first CBP network structure, a BP neural network structure and a second CBP network structure.

5. The method according to claim 4, characterized in that The convolutional layer and the batch normalization layer in the network of the rapid flood area extraction module are subjected to data fusion, including: The processing process of the convolution layer (Equation (1)) and the processing process of the batch normalization layer (Equation (2)) are combined into one process, including equations (3), (4), and (5): y conv =w·x+b (1) y=w′·x+b' (3) Where x and y represent input and output respectively, w, b, γ, ε and β are all the parameters in the calculation, μ and σ 2 Represents the mean and variance of the input.

6. The method according to claim 1, characterized in that The historical information fusion module includes a GPU and a CPU; the preliminary flood extraction result is input into the historical information fusion module, and the current image flood extraction result is optimized by using the historical image flood extraction result and a plurality of decision-level fusion strategies to output the final flood detection result, including: The processing unit of the CPU receives the binary classification result of the flood extraction result corresponding to the output of the lightweight neural network model, and determines the flood stage of the current image by counting pixels to determine the corresponding fusion strategy; wherein the binary classification result includes the flood extraction result of the current image and the flood extraction result of the historical image; The binary classification result and the fusion strategy are input into the GPU to perform information fusion calculation to obtain an information fusion result, and the information fusion result is processed through data to obtain a final flood detection result.

7. The method according to claim 1, characterized in that After obtaining the final flood detection result, the method further includes: The final flood detection result is sent to the user terminal after being processed by graphics.

8. The method according to claim 1, characterized in that The FPGA development board and the CPU-GPU edge computing device are connected using a Gigabit Ethernet cable, and data is directly exchanged with external devices using the UDP / IP protocol through Ethernet.

9. The method according to claim 3, characterized in that: The model parameters of the neural network model are quantized so that the original 64-bit floating-point numbers are converted into 16-bit fixed-point numbers.

10. The method according to claim 3, characterized in that Perform lightweight operations on neural network models, including: The calculation channel and the maximum value index operations used to determine the non-flood area in the original neural network model are removed to modify the network model architecture to obtain a simplified single-channel model.

Citation Information

Patent Citations

  • SAR flood image change detection method based on FPGA

    CN109886941A

  • Flood change detection method and device and storage medium

    CN115761504A

  • Lightweight flood detection method and device based on soft fusion strategy

    CN116310869A

  • Internet remote sensing satellite real-time service system for mobile terminal user

    CN117040594A

  • Remote sensing satellite in-orbit video coding method and system based on heterogeneous computing platform

    CN117294858A

Cited By

  • Low-resolution object target detection method and system based on heterogeneous calculation

    CN119625505A