A Remote Sensing Image Change Detection Method Based on FPGA

Abstract

A remote sensing image change detection method based on FPGA includes: S1, acquiring or selecting a dataset and constructing a remote sensing change detection network; S2, selecting dual-temporal images from the dataset and inputting them into the remote sensing change detection network to obtain change detection results, i.e., predicted values; S3, constructing a loss function, calculating the loss between the predicted values ​​and the true values ​​of the samples, and repeating S2 to S3 until the loss function converges, updating the weights of the remote sensing change detection network; S4, performing Batch Normalization (BN) layer fusion and network weight quantization on the trained remote sensing change detection network, deploying the quantized remote sensing change detection network on an FPGA and testing it to obtain the result image after change detection. The quantized remote sensing change detection network in this invention has a simple network structure, making it very suitable for FPGA deployment. While ensuring accuracy, it significantly reduces the number of parameters, computational load, and memory usage.

Description

Technical Field

[0001] This invention relates to the field of visual image processing technology, and in particular to a method for detecting changes in remote sensing images based on FPGA. Background Technology

[0002] Image change detection is used to analyze image differences of the same area at different times. With the continuous collection and development of high-resolution remote sensing images, change detection has been widely applied in various fields such as vegetation cover monitoring, urban planning, street view analysis, and medical diagnosis. Traditional change detection methods are mainly divided into pixel-based change detection and object-based change detection. However, with the development of deep learning technology, the limitations of traditional algorithms in feature extraction have been overcome, especially convolutional neural networks, which can automatically extract complex high-level features, thereby generating more robust change detection results. This makes change detection technology perform better in various application fields.

[0003] While current mainstream change detection algorithms perform well on various public datasets, most methods enhance change features by adding complex information interaction modules to neural networks. This significantly increases model complexity and the number of parameters, limiting their deployment in real-world scenarios. Therefore, developing lightweight change detection models has become a research focus. Furthermore, when deploying neural networks in embedded environments, resource constraints mean that most hardware cannot meet the low-power and low-latency application requirements. FPGAs, due to their low power consumption, low latency, and reconfigurability, are particularly suitable for the inference stage of neural networks. Compared to other hardware, FPGAs can flexibly change their hardware structure through configuration, allowing for customization for specific applications and adapting to the rapidly evolving needs of deep learning. However, existing FPGAs suffer from low development efficiency, insufficient parallelism, inefficient processor architecture, and poor accuracy and versatility, making it difficult to directly deploy existing remote sensing image change detection methods on FPGAs. Therefore, there is an urgent need for an FPGA-based remote sensing image change detection method. Summary of the Invention

[0004] This invention provides a remote sensing image change detection method based on FPGA to solve the technical problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] This invention provides a method for detecting changes in remote sensing images based on FPGA, comprising the following steps:

[0007] S1. Collect or select datasets and construct a remote sensing change detection network;

[0008] S2. Select dual-temporal images from the dataset and input them into the remote sensing change detection network. The efficient early fusion module in the remote sensing change detection network extracts, enhances, and overlays feature maps of the dual-temporal images. The output of the efficient early fusion module is then input into the decoding module to obtain the change detection result, i.e., the predicted value.

[0009] S3. Construct the total loss function, calculate the loss between the predicted value and the true value of the sample, and repeat S2 to S3 until the total loss function converges. Then, use the backpropagation algorithm to update the weights of the remote sensing change detection network to obtain the trained remote sensing change detection network.

[0010] S4. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network. Deploy the quantized remote sensing change detection network in FPGA and test it to obtain the result image after change detection.

[0011] Furthermore, the construction of the remote sensing change detection network in S1 includes the following steps:

[0012] S11. First, the ordinary units and downsampling units in the lightweight deep learning network are improved respectively to obtain the improved lightweight deep learning network.

[0013] S12, Construct an efficient early fusion module based on an improved lightweight deep learning network and the improved ordinary units in S11;

[0014] S13. A decoding module is constructed based on an improved lightweight deep learning network and connected to an efficient early fusion module. The efficient early fusion module and the decoding module constitute a remote sensing change detection network.

[0015] Furthermore, step S11 specifically includes the following steps:

[0016] S111: Remove the channel shuffling operation from the ordinary unit and the downsampling unit in the lightweight deep learning network, respectively; thus, the improved ordinary unit is obtained.

[0017] S112. Replace the channel-wise convolutional layer with a stride of 2 in the downsampling unit after the channel shuffling operation with a max pooling layer with a stride of 2×2, and then connect a channel-wise convolutional layer with a stride of 1 on the max pooling layer. Thus, the improved downsampling unit and the improved lightweight deep learning network are obtained.

[0018] Furthermore, the decoding module in S13 includes a lightweight backbone network, two lightweight feature pyramid structures, and a classification layer;

[0019] The lightweight backbone network is connected to two lightweight feature pyramid structures, and the two lightweight feature pyramid structures are connected to the classification layer.

[0020] The lightweight backbone network consists mainly of multiple sequentially connected composite unit blocks, each of which contains one improved downsampling unit and three improved ordinary units;

[0021] Both lightweight feature pyramid structures consist of upsampling operators, feature addition operators, pointwise convolutional layers, and channelwise convolutional layers, and are responsible for fusing and decoding features at multiple different scales.

[0022] Furthermore, step S2 specifically includes the following steps:

[0023] S21. Select dual-temporal images from the dataset. Dual-temporal images include two remote sensing images of the same area at different times. Then input the dual-temporal images into the remote sensing change detection network.

[0024] S22. The dual-temporal image first undergoes a pointwise convolutional layer, a channelwise convolutional layer, and a 2×2 max pooling layer within the efficient early fusion module to extract initial features, thus obtaining the initial features.

[0025] S23. After expanding the channels through pointwise convolution of the initial features, the features are enhanced by n improved ordinary units, and then compressed through pointwise convolution to obtain two enhanced features.

[0026] S24. Perform channel stacking operation on the two enhanced features, i.e., Concat operation, and then perform pointwise convolution to obtain the output of the efficient early fusion module;

[0027] S25. Input the output of the efficient early fusion module into the decoding module. The output of the efficient early fusion module is processed by a lightweight backbone network to generate multiple feature maps of different scales.

[0028] S26. Input multiple feature maps of different scales into two lightweight feature pyramid structures for fusion decoding. Finally, pass through the classification layer to obtain the change detection result, which is the output of the decoding module.

[0029] Furthermore, the total loss function in S3 is as follows:

[0030] Loss = αLoss1 + βLoss2

[0031] Where Loss represents the total loss function, Loss1 and Loss2 represent the supervised loss function and the binary cross-entropy loss function, respectively; a and β represent the scaling factors of the supervised loss function and the binary cross-entropy loss function, respectively.

[0032] The specific formula for calculating the supervised guided loss function Loss1 is as follows:

[0033] Loss1 = -[1 + GlogP - Glog(1 - P)]

[0034] Where P and G represent the predicted value and the actual value of the sample output by the efficient early fusion module, respectively;

[0035] The formula for calculating the binary cross-entropy loss function Loss2 is as follows:

[0036]

[0037] Where H and W are the height and width of the output feature map, respectively; h and w represent the row and column of the corresponding pixel, respectively; g h,w p represents the true value of the sample pixel corresponding to the h-th row and w-th column; h,w This represents the predicted value of the pixel corresponding to the h-th row and w-th column.

[0038] Furthermore, step S4 specifically includes the following steps:

[0039] S41. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, and export the quantized remote sensing change detection network and store it on the SD card in the FPGA development board.

[0040] S42. The HLS technology was used to design and optimize the operators in the quantized remote sensing change detection network in a highly parallel manner, and an FPGA accelerator system was built based on these operators.

[0041] S43. Acquire the image to be detected and input it into the FPGA accelerator system of the deployed quantized remote sensing change detection network. The FPGA accelerator system performs board-level inference verification and obtains the result image after change detection.

[0042] Furthermore, step S41 specifically includes the following steps:

[0043] S411. In the trained remote sensing change detection network, BN layer fusion optimization is performed on the convolution operation containing BN layers. After fusion optimization, a fused convolutional module is obtained. The specific convolution operation of the fused convolutional module is as follows:

[0044] Y′ conv =w′x+b′

[0045] Among them, Y′ convrepresents the output of the fused convolutional module; w′ represents the weights of the convolutional kernels of the fused convolutional module; x represents the input; b′ represents the bias term of the fused convolutional module, i.e., the output after passing through the BN layer;

[0046] In the formula, the weight w′ and the bias term b′ are calculated using the following formula:

[0047]

[0048]

[0049] Where Y represents the scaling factor, w represents the weights of the convolutional kernels of the convolutional module before fusion optimization; σ represents the standard deviation of the input data, ε represents a fixed value approaching zero to prevent the denominator from being 0; b represents the bias term of the convolutional module before fusion optimization, i.e., the output of the convolutional module before fusion optimization; μ represents the average value of the input data; β represents another bias term.

[0050] S412. Quantize the 32-bit floating-point arithmetic parameters in the fused and optimized remote sensing change detection network obtained in S411 into fixed-point integers to reduce the computational and storage space occupied by the remote sensing change detection network; the quantization calculation is as follows:

[0051] Y fixed =round(Y float ×2 i )

[0052] Among them, Y fixed Y represents a quantized fixed-point integer, where round is a rounding function. float This represents the floating-point number to be quantized, and i represents the quantization factor;

[0053] Thus, the quantified remote sensing change detection network is obtained;

[0054] S413. Export the quantized weights of the remote sensing change detection network obtained in S412 to the SD card on the FPGA development board for use in remote sensing change detection network inference.

[0055] Furthermore, step S42 specifically includes the following steps:

[0056] S421. Select an FPGA development board. The ZYNQ series development board includes an interconnected ARM terminal, an FPGA terminal, and related peripheral devices. The ARM terminal and the FPGA terminal communicate via the AXI bus. The ARM terminal is responsible for image input and decoding as well as task scheduling, while the FPGA terminal is responsible for accelerating computational inference in the remote sensing change detection network.

[0057] S422. On the FPGA side, use HLS to design the corresponding operator IPs for the image preprocessing unit, channel-wise convolution unit, point-wise convolution unit, max pooling unit, upsampling unit, feature addition unit and post-processing unit respectively.

[0058] S423. Construct a quantized remote sensing change detection network on the ARM side. By calling the image preprocessing IP, channel-wise convolution IP, point-wise convolution IP, max pooling IP, upsampling IP, feature summation IP and post-processing IP designed in S422, the FPGA accelerator system is obtained.

[0059] Furthermore, step S43 specifically includes the following steps:

[0060] S431. Collect the dual-temporal remote sensing images to be detected and store them on the same SD card as the weight file of the quantized remote sensing change detection network;

[0061] The S432 FPGA accelerator system loads the dual-temporal remote sensing image to be detected and the weight file of the quantized remote sensing change detection network into DDR, starts the constructed FPGA accelerator system to obtain the final change detection result map, and stores it in SD card.

[0062] The beneficial effects of this invention are:

[0063] Compared with the prior art, the significant advantages of this invention are:

[0064] 1. The remote sensing change detection network proposed in this invention includes a lightweight, efficient early fusion module and a lightweight decoding module. Therefore, the entire remote sensing change detection network has a simple structure and is very lightweight, making it very suitable for FPGA deployment. While ensuring accuracy, the number of parameters, computational load and memory usage are greatly reduced.

[0065] 2. While ensuring the accuracy of change detection, this invention performs BN layer fusion and network quantization on the trained remote sensing change detection network, which greatly reduces the resource consumption of FPGA.

[0066] 3. This invention also constructs an FPGA accelerator system to deploy and infer the proposed remote sensing change detection network. HLS is used to design and optimize the operators in the remote sensing change detection network in a highly parallel manner, verifying the feasibility of the proposed lightweight remote sensing change detection network running efficiently on FPGA, and solving the technical problems of the existing technology where the network model is too large and it is difficult to deploy on the device. Attached Figure Description

[0067] Figure 1 This is a flowchart of the present invention;

[0068] Figure 2These are schematic diagrams of the improved ordinary unit and the improved downsampling unit in this invention; wherein (a) is a schematic diagram of the ordinary unit and (b) is a schematic diagram of the downsampling unit.

[0069] Figure 3 This is a schematic diagram of the efficient early fusion module in this invention;

[0070] Figure 4 This is a schematic diagram of the decoding module in this invention;

[0071] Figure 5 This is a schematic diagram of the FPGA accelerator system in this invention;

[0072] Figure 6 The results are shown in the figure obtained by the method of the present invention, wherein (a) is a real remote sensing image of a certain location that changes with the seasons obtained by Google Earth, (b) is a real remote sensing image of the same location at another time point that changes with the seasons obtained by Google Earth, (c) is the ground truth of dataset change detection, and (d) is the board-level inference result figure obtained by the method of the present invention. Detailed Implementation

[0073] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0074] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0076] It should also be noted that in the embodiments of this application, the same reference numerals are used to represent the same component or part. For the same part in the embodiments of this application, the reference numerals may only be used to mark one part or component as an example. It should be understood that the reference numerals are also applicable to other identical parts or components.

[0077] Reference Figure 1 This application provides an FPGA-based remote sensing image change detection method, comprising the following steps:

[0078] S1. Collect or select datasets and construct a remote sensing change detection network;

[0079] S2. Select dual-temporal images from the dataset and input them into the remote sensing change detection network. The efficient early fusion module in the remote sensing change detection network extracts, enhances, and overlays feature maps of the dual-temporal images. The output of the efficient early fusion module is then input into the decoding module to obtain the change detection result, i.e., the predicted value.

[0080] S3. Construct the total loss function, calculate the loss between the predicted value and the true value of the sample, and repeat S2 to S3 until the total loss function converges. Then, use the backpropagation algorithm to update the weights of the remote sensing change detection network to obtain the trained remote sensing change detection network.

[0081] S4. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network. Deploy the quantized remote sensing change detection network in FPGA and test it to obtain the result image after change detection.

[0082] In some embodiments, constructing the remote sensing change detection network in S1 includes the following steps:

[0083] S11. First, the ordinary units and downsampling units in the lightweight deep learning network are improved respectively to obtain the improved lightweight deep learning network.

[0084] S12, Construct an efficient early fusion module based on an improved lightweight deep learning network and the improved ordinary units in S11;

[0085] S13. A decoding module is constructed based on an improved lightweight deep learning network and connected to an efficient early fusion module. The efficient early fusion module and the decoding module constitute the remote sensing change detection network. The remote sensing change detection network does not contain any complex information interaction modules, has few parameters, and is easy to deploy.

[0086] In some embodiments, the structure of the efficient early fusion module is as follows: Figure 3 As shown, it mainly includes multiple pointwise convolutional layers, max pooling layers, multiple improved ordinary units, and an FCN head. The pointwise convolutional layers in the same layer share weights. The FCN head is used to calculate the supervised guided loss function Loss1 mentioned below to supervise the fusion of features by the efficient early fusion module.

[0087] In some embodiments, S11 specifically includes the following steps:

[0088] S111. Remove the channel shuffling operation from the ordinary unit and the downsampling unit in the lightweight deep learning network (ShuffleNetV2 Unit); thus, the improved ordinary unit is obtained. The channel shuffling operation is removed mainly because it involves a large amount of data transfer, which is very inefficient to implement on FPGA.

[0089] S112. Replace the channel-wise convolutional layer with a stride of 2 in the downsampling unit after removing the channel shuffling operation with a max pooling layer with a stride of 2×2. Then, connect a channel-wise convolutional layer with a stride of 1 on top of this max pooling layer. This yields the improved downsampling unit and the improved lightweight deep learning network. The improved downsampling unit saves more FPGA resources in operator design, providing more room for subsequent optimization.

[0090] Specifically, refer to Figure 2 (a) is a block diagram of the improved ordinary unit; (b) is a block diagram of the improved downsampling unit.

[0091] The improved ordinary unit mainly includes channel rearrangement, multiple pointwise convolutional layers, channel-wise convolutional layers, and the Concat operation; see [link to details] for the specific connection relationships of each layer. Figure 2 As shown in (a) in the text;

[0092] The improved downsampling unit mainly includes multiple pointwise convolutional layers, multiple channel-wise convolutional layers, multiple max pooling layers, and a concat operation. The specific structure of the improved downsampling unit is shown in [link to documentation]. Figure 2 As shown in (b) of the diagram.

[0093] In some embodiments, the decoding module in S13 includes a lightweight backbone network, two lightweight feature pyramid structures, and a classification layer;

[0094] The lightweight backbone network is connected to two lightweight feature pyramid structures, and the two lightweight feature pyramid structures are connected to the classification layer.

[0095] The lightweight backbone network mainly consists of multiple sequentially connected combination unit blocks, each combination unit block containing one improved downsampling unit and three improved ordinary units; preferably, the lightweight backbone network has three combination unit blocks, namely... Figure 4 Combined unit blocks 1 to 3 in the middle.

[0096] Both lightweight feature pyramid structures consist of upsampling operators, feature addition operators, pointwise convolutional layers, and channelwise convolutional layers, and are responsible for fusing and decoding features at multiple different scales.

[0097] In some embodiments, S2 specifically includes the following steps:

[0098] S21. Select dual-temporal images from the dataset. Dual-temporal images include two remote sensing images of the same area at different times. Then input the dual-temporal images into the remote sensing change detection network.

[0099] S22. The dual-temporal image first undergoes a pointwise convolutional layer, a channelwise convolutional layer, and a 2×2 max pooling layer within the efficient early fusion module to extract initial features, thus obtaining the initial features.

[0100] S23. After expanding the channels through pointwise convolution of the initial features, the features are enhanced by four improved ordinary units, and then compressed through pointwise convolution to obtain two enhanced features.

[0101] S24. Perform channel stacking operation on the two enhanced features, i.e., Concat operation, and then perform pointwise convolution to obtain the output of the efficient early fusion module;

[0102] S25. Input the output of the efficient early fusion module into the decoding module. The output of the efficient early fusion module is processed by a lightweight backbone network to generate multiple feature maps of different scales.

[0103] Specifically, refer to Figure 4 The output of the efficient early fusion module is input into a lightweight backbone network. Three sequentially connected combined unit blocks within the lightweight backbone network extract features, resulting in features at multiple different scales. Figure 4 Features 1 to 3.

[0104] S26. Input multiple feature maps of different scales into two lightweight feature pyramid structures for fusion decoding. Finally, pass through the classification layer to obtain the change detection result, which is the output of the decoding module.

[0105] In some embodiments, the total loss function in S3 is specifically as follows:

[0106] Loss = αLoss1 + βLoss2

[0107] Where Loss represents the total loss function, Loss1 and Loss2 represent the supervised loss function and the binary cross-entropy loss function, respectively; a and β represent the scaling factors of the supervised loss function and the binary cross-entropy loss function, respectively.

[0108] The specific formula for calculating the supervised guided loss function Loss1 is as follows:

[0109] Loss1 = -[1 + GlogP - Glog(1 - P)]

[0110] Where P and G represent the predicted value and the actual value of the sample output by the efficient early fusion module, respectively;

[0111] The formula for calculating the binary cross-entropy loss function Loss2 is as follows:

[0112]

[0113] Where H and W are the height and width of the output feature map, respectively; h and w represent the row and column of the corresponding pixel, respectively; g h,w p represents the true value of the sample pixel corresponding to the h-th row and w-th column; h,w This represents the predicted value of the pixel corresponding to the h-th row and w-th column.

[0114] In some embodiments, S4 specifically includes the following steps:

[0115] S41. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, and export the quantized remote sensing change detection network and store it on the SD card in the FPGA development board.

[0116] S42. Highly parallel design and optimization of operators in the quantized remote sensing change detection network were performed using HLS technology, and an FPGA accelerator system was built based on these operators; the specific structure of the FPGA accelerator system is shown in [link to S42]. Figure 5 As shown; the FPGA accelerator system specifically includes an FPGA development board and memory electrically connected to the FPGA development board, wherein the memory includes DDR memory and a DDR controller electrically connected to the DDR memory; the FPGA development board mainly includes Block RAM, a data input module, a weight cache module, a PS terminal, and a data output module; wherein, the Block RAM is electrically connected to the data input module; the data input module, the weight cache module, and the data output module are respectively electrically connected to the PS terminal.

[0117] S43. Acquire the image to be detected and input it into the FPGA accelerator system of the deployed quantized remote sensing change detection network. The FPGA accelerator system performs board-level inference verification and obtains the result image after change detection.

[0118] In some embodiments, S41 specifically includes the following steps:

[0119] S411. In the trained remote sensing change detection network, BN layer fusion optimization is performed on the convolution operations containing BN layers. The fusion of BN layers and convolution operations is lossless, without causing accuracy loss, reducing the number of parameters, and improving the computational efficiency of the remote sensing change detection network. After fusion optimization, a fused convolutional module is obtained. The specific convolution operations of the fused convolutional module are as follows:

[0120] Y′ conv =w′x+b′

[0121] Among them, Y′ conv represents the output of the fused convolutional module; w′ represents the weights of the convolutional kernels of the fused convolutional module; x represents the input; b′ represents the bias term of the fused convolutional module, i.e., the output after passing through the BN layer;

[0122] In the formula, the weight w′ and the bias term b′ are calculated using the following formula:

[0123]

[0124]

[0125] Where Y represents the scaling factor, w represents the weights of the convolutional kernels of the convolutional module before fusion optimization; σ represents the standard deviation of the input data, ε represents a fixed value approaching zero to prevent the denominator from being 0; b represents the bias term of the convolutional module before fusion optimization, i.e., the output of the convolutional module before fusion optimization; μ represents the average value of the input data; β represents another bias term.

[0126] S412. Quantize the 32-bit floating-point arithmetic parameters in the fused and optimized remote sensing change detection network obtained in S411 into fixed-point integers to reduce the computational and storage space occupied by the remote sensing change detection network; in this embodiment, the quantized fixed-point integer is 16 bits; the quantization calculation is as follows:

[0127] Y fixed =round(Y float ×2 i )

[0128] Among them, Y fixed Y represents a quantized fixed-point integer, where round is a rounding function. float This represents the floating-point number to be quantized, and i represents the quantization factor;

[0129] Thus, the quantified remote sensing change detection network is obtained;

[0130] S413. Export the quantized weights of the remote sensing change detection network obtained in S412 to the SD card on the FPGA development board for use in remote sensing change detection network inference.

[0131] In some embodiments, S42 specifically includes the following steps:

[0132] S421. Select an FPGA development board. The ZYNQ series development board includes an interconnected ARM terminal, an FPGA terminal, and related peripheral devices. The ARM terminal and the FPGA terminal communicate via the AXI bus. The ARM terminal is responsible for image input and decoding as well as task scheduling, while the FPGA terminal is responsible for accelerating computational inference in the remote sensing change detection network.

[0133] S422. On the FPGA side, use HLS to design the corresponding operator IPs for the image preprocessing unit, channel-wise convolution unit, point-wise convolution unit, max pooling unit, upsampling unit, feature addition unit and post-processing unit respectively.

[0134] S423. Construct a quantized remote sensing change detection network on the ARM side. By calling the image preprocessing IP, channel-wise convolution IP, point-wise convolution IP, max pooling IP, upsampling IP, feature summation IP and post-processing IP designed in S422, the FPGA accelerator system is obtained.

[0135] In some embodiments, S43 specifically includes the following steps:

[0136] S431. Collect the dual-temporal remote sensing images to be detected and store them on the same SD card as the weight file of the quantized remote sensing change detection network;

[0137] The S432 FPGA accelerator system loads the dual-temporal remote sensing image to be detected and the weight file of the quantized remote sensing change detection network into DDR, starts the constructed FPGA accelerator system to obtain the final change detection result map, and stores it in SD card.

[0138] In this invention, experiments were conducted using the LEVIR-CD dataset, which is widely used in the field of change detection. LEVIR-CD is a new large-scale remote sensing dataset for building change detection, consisting of 637 pairs of very high-resolution (0.5 m / pixel) Google Earth image patches, with a size of 1024 × 1024 pixels.

[0139] To quantitatively evaluate the accuracy of the designed network, we used four commonly used metrics in change detection research, including precision, recall, F1 score, and inter-overlap ratio (IoU), for comparative experiments. We also used model parameter count and multiplication operations (MACs) as evaluation metrics for model size and efficiency.

[0140] Currently, among existing technologies, the lightest remote sensing image change detection network is TINYCD from "TINYCD: A (not so) deep learning model for change detection," with a model size of only 0.28M. The change detection metrics of the method of this invention and existing methods are compared in the table below.

[0141] Table 1 Comparison of Detection Indicators for Changes in the Method of the Present Invention and the Existing Technical Method

[0142] Method type Param(M) MACs(G) Precision Recall F1-Score IoU Existing technology 0.28 1.54 92.68 89.47 91.05 83.57 This invention 0.18 1.15 92.74 90.44 91.58 84.46

[0143] As shown in Table 1, compared with the existing technology "TINYCD: A (not so) deep learning model forchange detection", the method of this invention outperforms the existing technology in six aspects: model parameter quantity, product operation, precision, accuracy, recall, F1 score, and mean crossover ratio. The method of this invention can effectively improve the detection efficiency of remote sensing image change detection networks while ensuring detection accuracy.

[0144] Reference Figure 6 , Figure 6 This is a diagram showing the board-level inference result obtained using the method of the present invention in one embodiment of the present invention. (a) is a real remote sensing image of a location that changes seasonally, obtained from Google Earth; (b) is a real remote sensing image of the same location at another time point that changes seasonally, obtained from Google Earth; (c) is the ground truth for dataset change detection; and (d) is the diagram showing the board-level inference result obtained using the method of the present invention.

[0145] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting changes in remote sensing images based on FPGA, characterized in that, Includes the following steps: S1. Collect or select datasets and construct a remote sensing change detection network; S2. Select dual-temporal images from the dataset and input them into the remote sensing change detection network. The efficient early fusion module in the remote sensing change detection network extracts, enhances, and overlays feature maps of the dual-temporal images. The output of the efficient early fusion module is then input into the decoding module to obtain the change detection result, i.e., the predicted value. S3. Construct the total loss function, calculate the loss between the predicted value and the true value of the sample, and repeat S2 to S3 until the total loss function converges. Then, use the backpropagation algorithm to update the weights of the remote sensing change detection network to obtain the trained remote sensing change detection network. S4. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, deploy the quantized remote sensing change detection network in FPGA and test it to obtain the result image after change detection. The construction of the remote sensing change detection network in S1 includes the following steps: S11. First, the ordinary units and downsampling units in the lightweight deep learning network are improved respectively to obtain the improved lightweight deep learning network. S12, Construct an efficient early fusion module based on an improved lightweight deep learning network and the improved ordinary units in S11; S13. A decoding module is constructed based on an improved lightweight deep learning network and connected to an efficient early fusion module. The efficient early fusion module and the decoding module constitute a remote sensing change detection network. S11 specifically includes the following steps: S111: Remove the channel shuffling operation from the ordinary unit and the downsampling unit in the lightweight deep learning network, respectively; thus, the improved ordinary unit is obtained. S112. Replace the channel-wise convolutional layer with a stride of 2 in the downsampling unit after removing the channel shuffle operation with a max pooling layer with a stride of 2×2, and then connect a channel-wise convolutional layer with a stride of 1 on the max pooling layer. Thus, the improved downsampling unit and the improved lightweight deep learning network are obtained. The decoding module in S13 includes a lightweight backbone network, two lightweight feature pyramid structures, and a classification layer; The lightweight backbone network is connected to two lightweight feature pyramid structures, and the two lightweight feature pyramid structures are connected to the classification layer. The lightweight backbone network consists of multiple sequentially connected composite unit blocks, each of which contains one improved downsampling unit and three improved ordinary units; Both lightweight feature pyramid structures consist of upsampling operators, feature addition operators, pointwise convolutional layers, and channelwise convolutional layers, which are responsible for fusing and decoding features at multiple different scales. S4 specifically includes the following steps: S41. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, and export the quantized remote sensing change detection network and store it on the SD card in the FPGA development board. S42. HLS technology was used to design and optimize the operators in the quantized remote sensing change detection network in parallel, and an FPGA accelerator system was built based on these operators. S43. Acquire the image to be detected and input it into the FPGA accelerator system of the deployed quantized remote sensing change detection network. The FPGA accelerator system performs board-level inference verification and obtains the result image after change detection.

2. The remote sensing image change detection method according to claim 1, characterized in that, S2 specifically includes the following steps: S21. Select dual-temporal images from the dataset. Dual-temporal images include two remote sensing images of the same area at different times. Then input the dual-temporal images into the remote sensing change detection network. S22. The dual-temporal image first undergoes a pointwise convolutional layer, a channelwise convolutional layer, and a 2×2 max pooling layer within the efficient early fusion module to extract initial features, thus obtaining the initial features. S23. After expanding the channels through pointwise convolution of the initial features, the features are enhanced by n improved ordinary units, and then compressed through pointwise convolution to obtain two enhanced features. S24. Perform channel stacking operation on the two enhanced features, i.e., Concat operation, and then perform pointwise convolution to obtain the output of the efficient early fusion module; S25. Input the output of the efficient early fusion module into the decoding module. The output of the efficient early fusion module is processed by a lightweight backbone network to generate multiple feature maps of different scales. S26. Input multiple feature maps of different scales into two lightweight feature pyramid structures for fusion decoding. Finally, pass through the classification layer to obtain the change detection result, which is the output of the decoding module.

3. The remote sensing image change detection method according to claim 1, characterized in that, The total loss function in S3 is as follows: Loss = αLoss1 + βLoss2 Where Loss represents the total loss function, Loss1 and Loss2 represent the supervised guidance loss function and the binary cross-entropy loss function, respectively; α and β represent the scaling factors of the supervised guidance loss function and the binary cross-entropy loss function, respectively. The specific formula for calculating the supervised guided loss function Loss1 is as follows: Loss1 = -[1 + GlogP - Glog(1 - P)] Where P and G represent the predicted value and the actual value of the sample output by the efficient early fusion module, respectively; The formula for calculating the binary cross-entropy loss function Loss2 is as follows: Where H and W are the height and width of the output feature map, respectively; h and w represent the row and column of the corresponding pixel, respectively; g h,w p represents the true value of the sample pixel corresponding to the h-th row and w-th column; h，w This represents the predicted value of the pixel corresponding to the h-th row and w-th column.

4. The remote sensing image change detection method according to claim 3, characterized in that, S41 specifically includes the following steps: S411. In the trained remote sensing change detection network, BN layer fusion optimization is performed on the convolution operation containing BN layers. After fusion optimization, a fused convolutional module is obtained. The specific convolution operation of the fused convolutional module is as follows: Y′ conv =w′x+b′ Among them, Y′ conv represents the output of the fused convolutional module; w′ represents the weights of the convolutional kernels of the fused convolutional module; x represents the input; b′ represents the bias term of the fused convolutional module, i.e., the output after passing through the BN layer; In the formula, the weight w′ and the bias term b′ are calculated using the following formula: Where Y represents the scaling factor, w represents the weights of the convolutional kernels of the convolutional module before fusion optimization; σ represents the standard deviation of the input data, ε represents a fixed value approaching zero to prevent the denominator from being 0; b represents the bias term of the convolutional module before fusion optimization, i.e., the output of the convolutional module before fusion optimization; μ represents the average value of the input data; β represents another bias term. S412. Quantize the 32-bit floating-point arithmetic parameters in the fused and optimized remote sensing change detection network obtained in S411 into fixed-point integers to reduce the computational and storage space occupied by the remote sensing change detection network; the quantization calculation is as follows: Y fixed =round(Y float ×2 i ) Among them, Y fixed Y represents a quantized fixed-point integer, where round is a rounding function. float This represents the floating-point number to be quantized, and i represents the quantization factor; Thus, the quantified remote sensing change detection network is obtained; S413. Export the quantized weights of the remote sensing change detection network obtained in S412 to the SD card on the FPGA development board for use in remote sensing change detection network inference.

5. The remote sensing image change detection method according to claim 1, characterized in that, S42 specifically includes the following steps: S421. Select an FPGA development board. The ZYNQ series development board includes an interconnected ARM terminal, an FPGA terminal, and related peripheral devices. The ARM terminal and the FPGA terminal communicate via the AXI bus. The ARM terminal is responsible for image input and decoding as well as task scheduling, while the FPGA terminal is responsible for accelerating computational inference in the remote sensing change detection network. S422. On the FPGA side, use HLS to design the corresponding operator IPs for the image preprocessing unit, channel-wise convolution unit, point-wise convolution unit, max pooling unit, upsampling unit, feature addition unit and post-processing unit respectively. S423. Construct a quantized remote sensing change detection network on the ARM side. By calling the image preprocessing IP, channel-wise convolution IP, point-wise convolution IP, max pooling IP, upsampling IP, feature summation IP and post-processing IP designed in S422, the FPGA accelerator system is obtained.

6. The remote sensing image change detection method according to any one of claims 1 to 5, characterized in that, S43 specifically includes the following steps: S431. Collect the dual-temporal remote sensing images to be detected and store them on the same SD card as the weight file of the quantized remote sensing change detection network; The S432 FPGA accelerator system loads the dual-temporal remote sensing image to be detected and the weight file of the quantized remote sensing change detection network into DDR, starts the constructed FPGA accelerator system to obtain the final change detection result map, and stores it in SD card.

Images