Remote sensing image change detection method based on FPGA

Abstract

A remote sensing image change detection method based on an FPGA comprises the steps that S1, a data set is collected or selected, and a remote sensing change detection network is constructed; s2, selecting a dual-time-phase image from the data set, and inputting the dual-time-phase image into a remote sensing change detection network to obtain a change detection result, namely a predicted value; s3, constructing a loss function, calculating the loss between a predicted value and a sample true value, circulating the steps S2 to S3 until the loss function converges, and updating the weight of the remote sensing change detection network; and S4, carrying out BN layer fusion and network weight quantization on the trained remote sensing change detection network, deploying the quantized remote sensing change detection network in the FPGA, and carrying out testing to obtain a result graph after change detection. The quantized remote sensing change detection network is simple in network structure and very suitable for FPGA deployment, and the parameter quantity, the calculation quantity and the memory occupation are greatly reduced on the premise that the accuracy is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image processing, and in particular to a remote sensing image change detection method based on FPGA. Background Art

[0002] Image change detection is used to analyze the image differences of the same area at different times. With the continuous collection and development of high-resolution remote sensing images, change detection has been widely used in various fields such as vegetation coverage monitoring, urban planning, street view analysis, and medical diagnosis. Traditional change detection methods are mainly divided into pixel-based change detection and object-based change detection. However, with the development of deep learning technology, the limitations of traditional algorithms in feature extraction have been broken, especially convolutional neural networks, which can automatically extract complex high-level features to generate more robust change detection results. This makes change detection technology perform better in various application fields.

[0003] Although the current mainstream change detection algorithms perform well on various public data sets, most methods enhance the change features by adding complex information interaction modules to the neural network, which leads to a significant increase in model complexity and the number of parameters, thus limiting its deployment in practical scenarios. Therefore, the development of lightweight change detection models has become a research focus. At the same time, when deploying neural networks in embedded environments, most hardware does not meet the application requirements of low power consumption and low latency due to resource constraints in embedded environments. FPGAs are particularly suitable for use in the neural network reasoning stage due to their low power consumption, low latency and reconfigurability. Compared with other hardware, FPGAs can flexibly change the hardware structure through configuration, customize for specific applications, and adapt to the needs of the rapid development of deep learning. However, existing FPGAs have the disadvantages of low development efficiency, insufficient parallelism, inefficient processor architecture, poor accuracy and versatility, etc., which makes it difficult for existing remote sensing image change detection methods to be directly deployed on FPGAs. Therefore, a remote sensing image change detection method based on FPGA is urgently needed. Summary of the invention

[0004] The present invention provides a remote sensing image change detection method based on FPGA to solve the technical problems mentioned in the background technology.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] The present invention provides a remote sensing image change detection method based on FPGA, comprising the following steps:

[0007] S1. Collect or select data sets and build a remote sensing change detection network;

[0008] S2, select a dual-phase image from the data set and input it into the remote sensing change detection network. The efficient early fusion module in the remote sensing change detection network extracts, enhances and superimposes the feature map of the dual-phase image, and inputs the output of the efficient early fusion module into the decoding module to obtain the change detection result, i.e., the predicted value;

[0009] S3, construct a total loss function, calculate the loss between the predicted value and the true value of the sample, loop S2 to S3 until the total loss function converges, and then use the back propagation algorithm to update the weights of the remote sensing change detection network to obtain the trained remote sensing change detection network;

[0010] S4. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, deploy the quantized remote sensing change detection network in FPGA and test it to obtain the result graph after change detection.

[0011] Furthermore, constructing a remote sensing change detection network in S1 includes the following steps:

[0012] S11. First, the common unit and the downsampling unit in the lightweight deep learning network are improved respectively to obtain an improved lightweight deep learning network;

[0013] S12, building an efficient early fusion module based on the improved lightweight deep learning network and the improved common unit in S11;

[0014] S13. A decoding module connected with an efficient early fusion module is constructed based on an improved lightweight deep learning network. The efficient early fusion module and the decoding module constitute a remote sensing change detection network.

[0015] Furthermore, the S11 specifically includes the following steps:

[0016] S111, respectively removing the channel shuffling operations in the common unit and the downsampling unit in the lightweight deep learning network; thus, an improved common unit is obtained;

[0017] S112. Replace the channel-by-channel convolution layer with a stride of 2 in the downsampling unit after removing the channel shuffling operation with a maximum pooling layer with a stride of 2×2, and then connect a channel-by-channel convolution layer with a stride of 1 to the maximum pooling layer. At this point, an improved downsampling unit and an improved lightweight deep learning network are obtained.

[0018] Further, the decoding module in S13 includes a lightweight backbone network, two lightweight feature pyramid structures and a classification layer;

[0019] The lightweight backbone network is connected to two lightweight feature pyramid structures respectively, and the two lightweight feature pyramid structures are connected to the classification layer respectively;

[0020] The lightweight backbone network mainly consists of a plurality of sequentially connected combined unit blocks, each of which contains one improved downsampling unit and three improved ordinary units;

[0021] Both lightweight feature pyramid structures are composed of upsampling operators, feature addition operators, point-by-point convolution layers, and channel-by-channel convolution layers, and are responsible for fusing and decoding features of multiple different scales.

[0022] Furthermore, the S2 specifically includes the following steps:

[0023] S21, selecting a dual-phase image from the data set, the dual-phase image including two remote sensing images in different time periods in the same area; and then inputting the dual-phase image into a remote sensing change detection network;

[0024] S22, the dual-phase image first undergoes a point-by-point convolution, a channel-by-channel convolution layer, and a 2×2 maximum pooling layer in the efficient early fusion module to extract initial features and obtain initial features;

[0025] S23, after the initial features are extended through a point-by-point convolution channel, the features are enhanced through n improved common units, and then two enhanced features are obtained through a point-by-point convolution compression channel;

[0026] S24, performing a channel superposition operation on the two enhanced features, i.e., a Concat operation, and then performing point-by-point convolution to obtain the output of the efficient early fusion module;

[0027] S25, inputting the output of the efficient early fusion module into the decoding module, and the output of the efficient early fusion module is passed through a lightweight backbone network to generate multiple feature maps of different scales;

[0028] S26. Input multiple feature maps of different scales into two lightweight feature pyramid structures for fusion decoding, and finally obtain the change detection result through the classification layer. The change detection result is the output of the decoding module.

[0029] Furthermore, the total loss function in S3 is specifically as follows:

[0030] Loss = αLoss1 + βLoss2

[0031] Among them, Loss represents the total loss function, Loss1 and Loss2 represent the supervised guided loss function and the binary cross entropy loss function respectively; a and β represent the proportional coefficients of the supervised guided loss function and the binary cross entropy loss function respectively;

[0032] The calculation formula of the supervised guided loss function Loss1 is as follows:

[0033] Loss1=-[1+GlogP-Glog(1-P)]

[0034] Among them, P and G represent the predicted value and sample true value output by the efficient early fusion module respectively;

[0035] The calculation formula of the binary cross entropy loss function Loss2 is as follows:

[0036]

[0037] Among them, H and W are the height and width of the output feature map respectively; h and w represent the rows and columns of the corresponding pixels respectively; g h,w represents the true value of the sample corresponding to the pixel in the hth row and the wth column; p h,w Represents the predicted value of the pixel corresponding to the hth row and the wth column.

[0038] Furthermore, the S4 specifically includes the following steps:

[0039] S41, performing BN layer fusion and network weight quantization on the trained remote sensing change detection network, and exporting the quantized remote sensing change detection network and storing it in the SD card in the FPGA development board;

[0040] S42, HLS technology is used to design and optimize operators in the quantized remote sensing change detection network in a highly parallel manner, and an FPGA accelerator system is built based on these operators;

[0041] S43, collecting the image to be detected, and inputting the image to be detected into the FPGA accelerator system of the remote sensing change detection network after quantization is deployed, and the FPGA accelerator system performs board-level reasoning verification to obtain a result map after change detection.

[0042] Furthermore, the S41 specifically includes the following steps:

[0043] S411. In the trained remote sensing change detection network, the convolution operation including the BN layer is subjected to BN layer fusion optimization. After the fusion optimization, a fused convolution module is obtained. The convolution operation of the fused convolution module is specifically as follows:

[0044] Y′ conv =w′x+b′

[0045] Among them, Y′ convrepresents the output of the fused convolution module; w′ represents the weight of the convolution kernel of the fused convolution module; x represents the input; b′ represents the bias term of the fused convolution module, that is, the output after the BN layer;

[0046] Wherein, the weight w′ and the bias term b′ are calculated by the following formula:

[0047]

[0048]

[0049] Among them, Y represents the scaling factor, w represents the weight of the convolution kernel of the convolution module before fusion optimization; σ represents the standard deviation of the input data, ε represents a fixed value that tends to zero to prevent the denominator from being 0; b represents the bias term of the convolution module before fusion optimization, that is, the output of the convolution module before fusion optimization; μ represents the average value of the input data; β represents another bias term;

[0050] S412, quantizing the parameters of the 32-bit floating-point arithmetic numbers in the fused and optimized remote sensing change detection network obtained in S411 into fixed-point integers to reduce the computing and storage space occupied by the remote sensing change detection network; the quantization calculation is specifically as follows:

[0051] Y fixed =round(Y float ×2 i )

[0052] Among them, Y fixed Represents a quantized fixed-point integer, round is a rounding function, Y float Represents the floating point number to be quantized, i represents the quantization factor;

[0053] At this point, the quantized remote sensing change detection network is obtained;

[0054] S413, exporting the weights in the quantized remote sensing change detection network obtained in S412 to the SD card on the FPGA development board for use in remote sensing change detection network reasoning.

[0055] Furthermore, the S42 specifically includes the following steps:

[0056] S421. Select an FPGA development board. The ZYNQ series development board includes an ARM end, an FPGA end, and related peripheral devices that are interconnected. The ARM end and the FPGA end communicate through the AXI bus. The ARM end is responsible for image input and decoding and task scheduling, and the FPGA end is responsible for computing and reasoning acceleration in the remote sensing change detection network.

[0057] S422. Use HLS to design operator IPs corresponding to the image preprocessing unit, the channel-by-channel convolution unit, the point-by-point convolution unit, the maximum pooling unit, the upsampling unit, the feature addition unit, and the post-processing unit on the FPGA side;

[0058] S423. Build a quantized remote sensing change detection network on the ARM side, and obtain the built FPGA accelerator system by calling the image preprocessing IP, channel-by-channel convolution IP, point-by-point convolution IP, maximum pooling IP, upsampling IP, feature addition IP and post-processing IP designed in S422.

[0059] Furthermore, the S43 specifically includes the following steps:

[0060] S431, collecting the dual-phase remote sensing image to be detected and storing it in the same SD card with the weight file of the quantized remote sensing change detection network;

[0061] S432, the FPGA accelerator system loads the dual-phase remote sensing image to be detected and the weight file of the quantized remote sensing change detection network into the DDR, starts the constructed FPGA accelerator system to obtain the final change detection result map, and stores it in the SD card.

[0062] Beneficial effects of the present invention:

[0063] Compared with the prior art, the present invention has the following significant advantages:

[0064] 1. The remote sensing change detection network proposed in the present invention includes a lightweight and efficient early fusion module and a lightweight decoding module, so the structure of the entire remote sensing change detection network is simple and lightweight, which is very suitable for FPGA deployment. Under the premise of ensuring accuracy, the number of parameters, calculation amount and memory usage are greatly reduced;

[0065] 2. While ensuring the accuracy of change detection, the present invention performs BN layer fusion and network quantization on the trained remote sensing change detection network, which greatly reduces the resource consumption of FPGA;

[0066] 3. The present invention also constructs an FPGA accelerator system to perform deployment reasoning verification on the proposed remote sensing change detection network, and uses HLS to perform highly parallel design and optimization of the operators in the remote sensing change detection network, verifying the feasibility of the proposed lightweight remote sensing change detection network to run efficiently on FPGA, solving the technical problems of the existing technology that the network model is too large and difficult to deploy on the device side. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of the present invention;

[0068] Figure 2Schematic diagrams of improved common units and improved downsampling units in the present invention; (a) is a schematic diagram of a common unit, and (b) is a schematic diagram of a downsampling unit;

[0069] Figure 3 This is a schematic diagram of an efficient early fusion module in the present invention;

[0070] Figure 4 It is a schematic diagram of a decoding module in the present invention;

[0071] Figure 5 This is a schematic diagram of the FPGA accelerator system in the present invention;

[0072] Figure 6 The result diagram obtained by the method of the present invention, wherein (a) is a real remote sensing image of a place obtained by Google Earth with seasonal changes, (b) is a real remote sensing image of the place at another time point obtained by Google Earth with seasonal changes, (c) is the true value of the data set change detection, and (d) is the board-level reasoning result diagram using the method of the present invention. DETAILED DESCRIPTION

[0073] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0074] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0076] It should also be noted that, in the embodiments of the present application, the same figure mark is used to represent the same component or the same part. For the same parts in the embodiments of the present application, the figure may only mark one of the parts or components as an example. It should be understood that the figure mark also applies to other identical parts or components.

[0077] Reference Figure 1 The present application embodiment provides a remote sensing image change detection method based on FPGA, comprising the following steps:

[0078] S1. Collect or select data sets and build a remote sensing change detection network;

[0079] S2, select a dual-phase image from the data set and input it into the remote sensing change detection network. The efficient early fusion module in the remote sensing change detection network extracts, enhances and superimposes the feature map of the dual-phase image, and inputs the output of the efficient early fusion module into the decoding module to obtain the change detection result, i.e., the predicted value;

[0080] S3, construct a total loss function, calculate the loss between the predicted value and the true value of the sample, loop S2 to S3 until the total loss function converges, and then use the back propagation algorithm to update the weights of the remote sensing change detection network to obtain the trained remote sensing change detection network;

[0081] S4. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, deploy the quantized remote sensing change detection network in FPGA and test it to obtain the result graph after change detection.

[0082] In some embodiments, constructing a remote sensing change detection network in S1 includes the following steps:

[0083] S11. First, the common unit and the downsampling unit in the lightweight deep learning network are improved respectively to obtain an improved lightweight deep learning network;

[0084] S12, building an efficient early fusion module based on the improved lightweight deep learning network and the improved common unit in S11;

[0085] S13. A decoding module connected with an efficient early fusion module is constructed based on an improved lightweight deep learning network. The efficient early fusion module and the decoding module constitute a remote sensing change detection network. The remote sensing change detection network does not contain any complex information interaction modules, has a small number of parameters, and is easy to deploy.

[0086] In some embodiments, the structure of the efficient early fusion module is as follows Figure 3 As shown in the figure, it mainly includes multiple point-by-point convolution layers, maximum pooling layers, multiple improved ordinary units and FCN heads. The weights of the point-by-point convolution layers at the same layer are shared; the FCN head is used to calculate the supervised guided loss function Loss1 that appears below to supervise the fusion of features by the efficient early fusion module.

[0087] In some embodiments, the S11 specifically includes the following steps:

[0088] S111, respectively remove the channel shuffle operations in the common unit and the downsampling unit in the lightweight deep learning network (ShuffleNetV2 Unit); thus, an improved common unit is obtained; the channel shuffle operation is removed mainly because it involves a large amount of data transfer, which is very inefficient to implement on FPGA;

[0089] S112, replace the channel-by-channel convolution layer with a step size of 2 in the downsampling unit after the channel shuffle operation is removed with a maximum pooling layer with a step size of 2×2, and then connect a channel-by-channel convolution layer with a step size of 1 to the maximum pooling layer, so that an improved downsampling unit and an improved lightweight deep learning network are obtained. The improved downsampling unit saves more FPGA resources in operator design, providing more space for subsequent optimization.

[0090] Specifically, refer to Figure 2 , where (a) is a structural block diagram of an improved common unit; (b) is a structural block diagram of an improved downsampling unit;

[0091] The improved common unit mainly includes channel rearrangement, multiple point-by-point convolutional layers, channel-by-channel convolutional layers, and Concat operations; the specific connection relationship of each layer can be found in Figure 2 As shown in (a);

[0092] The improved downsampling unit mainly includes multiple point-by-point convolution layers, multiple channel-by-channel convolution layers, multiple maximum pooling layers and Concat operations. The specific structure of the improved downsampling unit is shown in Figure 2 As shown in (b) in .

[0093] In some embodiments, the decoding module in S13 includes a lightweight backbone network, two lightweight feature pyramid structures and a classification layer;

[0094] The lightweight backbone network is connected to two lightweight feature pyramid structures respectively, and the two lightweight feature pyramid structures are connected to the classification layer respectively;

[0095] The lightweight backbone network mainly comprises a plurality of sequentially connected combination unit blocks (Block), each combination unit block comprises an improved downsampling unit and three improved ordinary units; preferably, the number of combination unit blocks in the lightweight backbone network is 3, respectively Figure 4 Combination unit block 1 to combination unit block 3 in the.

[0096] Both lightweight feature pyramid structures are composed of upsampling operators, feature addition operators, point-by-point convolution layers, and channel-by-channel convolution layers, and are responsible for fusing and decoding features of multiple different scales.

[0097] In some embodiments, S2 specifically includes the following steps:

[0098] S21, selecting a dual-phase image from the data set, the dual-phase image including two remote sensing images in different time periods in the same area; and then inputting the dual-phase image into a remote sensing change detection network;

[0099] S22, the dual-phase image first undergoes a point-by-point convolution, a channel-by-channel convolution layer, and a 2×2 maximum pooling layer in the efficient early fusion module to extract initial features and obtain initial features;

[0100] S23, after the initial features are expanded through a point-by-point convolution channel, the features are enhanced through four improved common units, and then two enhanced features are obtained through a point-by-point convolution compression channel;

[0101] S24, performing a channel superposition operation on the two enhanced features, i.e., a Concat operation, and then performing point-by-point convolution to obtain the output of the efficient early fusion module;

[0102] S25, inputting the output of the efficient early fusion module into the decoding module, and the output of the efficient early fusion module is passed through a lightweight backbone network to generate multiple feature maps of different scales;

[0103] Specifically, refer to Figure 4 The output of the efficient early fusion module is input into the lightweight backbone network. The three sequentially connected combination unit blocks in the lightweight backbone network extract features and obtain features of multiple different scales, which are Figure 4 Features 1 to 3 in .

[0104] S26. Input multiple feature maps of different scales into two lightweight feature pyramid structures for fusion decoding, and finally obtain the change detection result through the classification layer. The change detection result is the output of the decoding module.

[0105] In some embodiments, the total loss function in S3 is specifically as follows:

[0106] Loss = αLoss1 + βLoss2

[0107] Among them, Loss represents the total loss function, Loss1 and Loss2 represent the supervised guided loss function and the binary cross entropy loss function respectively; a and β represent the proportional coefficients of the supervised guided loss function and the binary cross entropy loss function respectively;

[0108] The calculation formula of the supervised guided loss function Loss1 is as follows:

[0109] Loss1=-[1+GlogP-Glog(1-P)]

[0110] Among them, P and G represent the predicted value and sample true value output by the efficient early fusion module respectively;

[0111] The calculation formula of the binary cross entropy loss function Loss2 is as follows:

[0112]

[0113] Among them, H and W are the height and width of the output feature map respectively; h and w represent the rows and columns of the corresponding pixels respectively; g h,w represents the true value of the sample corresponding to the pixel in the hth row and the wth column; p h,w Represents the predicted value of the pixel corresponding to the hth row and the wth column.

[0114] In some embodiments, the S4 specifically includes the following steps:

[0115] S41, performing BN layer fusion and network weight quantization on the trained remote sensing change detection network, and exporting the quantized remote sensing change detection network and storing it in the SD card in the FPGA development board;

[0116] S42, HLS technology is used to design and optimize the operators in the quantized remote sensing change detection network in high parallelism, and an FPGA accelerator system is built based on these operators; the specific structure of the FPGA accelerator system is shown in Figure 5 As shown; the FPGA accelerator system specifically includes an FPGA development board and a memory electrically connected to the FPGA development board, wherein the memory includes a DDR memory and a DDR controller electrically connected to the DDR memory; the FPGA development board mainly includes a Block RAM, a data input module, a weight cache module, a PS end and a data output module; wherein the Block RAM is electrically connected to the data input module; the data input module, the weight cache module, and the data output module are electrically connected to the PS end respectively.

[0117] S43, collecting the image to be detected, and inputting the image to be detected into the FPGA accelerator system of the remote sensing change detection network after quantization is deployed, and the FPGA accelerator system performs board-level reasoning verification to obtain a result map after change detection.

[0118] In some embodiments, the S41 specifically includes the following steps:

[0119] S411. In the trained remote sensing change detection network, the convolution operation including the BN layer is optimized by BN layer fusion. The fusion of the BN layer and the convolution operation is lossless fusion, which does not cause precision loss, reduces the number of parameters, and improves the computational efficiency of the remote sensing change detection network. After the fusion optimization, a fused convolution module is obtained. The convolution operation of the fused convolution module is as follows:

[0120] Y′ conv =w′x+b′

[0121] Among them, Y′ conv represents the output of the fused convolution module; w′ represents the weight of the convolution kernel of the fused convolution module; x represents the input; b′ represents the bias term of the fused convolution module, that is, the output after the BN layer;

[0122] Wherein, the weight w′ and the bias term b′ are calculated by the following formula:

[0123]

[0124]

[0125] Among them, Y represents the scaling factor, w represents the weight of the convolution kernel of the convolution module before fusion optimization; σ represents the standard deviation of the input data, ε represents a fixed value that tends to zero to prevent the denominator from being 0; b represents the bias term of the convolution module before fusion optimization, that is, the output of the convolution module before fusion optimization; μ represents the average value of the input data; β represents another bias term;

[0126] S412, quantizing the parameters of the 32-bit floating-point arithmetic numbers in the remote sensing change detection network after the fusion optimization obtained in S411 into fixed-point integers to reduce the computing and storage space occupied by the remote sensing change detection network; in this embodiment, the quantized fixed-point integer is 16 bits; the quantization calculation is specifically as follows:

[0127] Y fixed =round(Y float ×2 i )

[0128] Among them, Y fixed Represents a quantized fixed-point integer, round is a rounding function, Y float Represents the floating point number to be quantized, i represents the quantization factor;

[0129] At this point, the quantized remote sensing change detection network is obtained;

[0130] S413, exporting the weights in the quantized remote sensing change detection network obtained in S412 to the SD card on the FPGA development board for use in remote sensing change detection network reasoning.

[0131] In some embodiments, the S42 specifically includes the following steps:

[0132] S421. Select an FPGA development board. The ZYNQ series development board includes an ARM end, an FPGA end, and related peripheral devices that are interconnected. The ARM end and the FPGA end communicate through the AXI bus. The ARM end is responsible for image input and decoding and task scheduling, and the FPGA end is responsible for computing and reasoning acceleration in the remote sensing change detection network.

[0133] S422. Use HLS to design operator IPs corresponding to the image preprocessing unit, the channel-by-channel convolution unit, the point-by-point convolution unit, the maximum pooling unit, the upsampling unit, the feature addition unit, and the post-processing unit on the FPGA side;

[0134] S423. Build a quantized remote sensing change detection network on the ARM side, and obtain the built FPGA accelerator system by calling the image preprocessing IP, channel-by-channel convolution IP, point-by-point convolution IP, maximum pooling IP, upsampling IP, feature addition IP and post-processing IP designed in S422.

[0135] In some embodiments, the S43 specifically includes the following steps:

[0136] S431, collecting the dual-phase remote sensing image to be detected and storing it in the same SD card with the weight file of the quantized remote sensing change detection network;

[0137] S432, the FPGA accelerator system loads the dual-phase remote sensing image to be detected and the weight file of the quantized remote sensing change detection network into the DDR, starts the constructed FPGA accelerator system to obtain the final change detection result map, and stores it in the SD card.

[0138] In this paper, the LEVIR-CD dataset, which is widely used in the field of change detection, is used for experiments. LEVIR-CD is a new large-scale remote sensing building change detection dataset consisting of 637 very high-resolution (0.5 m / pixel) Google Earth image patch pairs with a size of 1024×1024 pixels.

[0139] In order to quantitatively evaluate the accuracy of the designed network, we used four common indicators in change detection research, including precision, recall, F1-Score and average intersection over Union (IoU) for comparative experiments, plus the number of model parameters and multiplication operations (MACs) as evaluation indicators of model size and efficiency.

[0140] At present, in the prior art, the lightest remote sensing image change detection network is TINYCD in "TINYCD: A (not so) deep learning model for change detection", and the model size is only 0.28M. The comparison of the change detection indicators of the method of the present invention and the existing methods is shown in the following table.

[0141] Table 1 Comparison of change detection indicators between the method of the present invention and the prior art method

[0142] Method Type Param(M) MACs(G) Precision Recall F1-Score IoU Prior art 0.28 1.54 92.68 89.47 91.05 83.57 The present invention 0.18 1.15 92.74 90.44 91.58 84.46

[0143] As can be seen from Table 1, compared with the prior art method "TINYCD: A (not so) deep learning model for change detection", the indicators of the method of the present invention are superior to the prior art in six aspects: model parameter quantity, product operation, accuracy, precision, recall rate, F1 value and average intersection ratio. The method of the present invention can effectively improve the detection efficiency of the remote sensing image change detection network on the basis of ensuring the detection accuracy.

[0144] Reference Figure 6 , Figure 6 1 is a board-level reasoning result diagram using the method of the present invention in one embodiment of the present invention. (a) is a real remote sensing image of a place acquired by Google Earth that changes with the seasons, (b) is a real remote sensing image of the place acquired by Google Earth at another time point that changes with the seasons, (c) is the true value of the dataset change detection, and (d) is a board-level reasoning result diagram using the method of the present invention.

[0145] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A remote sensing image change detection method based on FPGA, characterized in that: The steps include: S1. Collect or select data sets and build a remote sensing change detection network; S2, select a dual-phase image from the data set and input it into the remote sensing change detection network. The efficient early fusion module in the remote sensing change detection network extracts, enhances and superimposes the feature map of the dual-phase image, and inputs the output of the efficient early fusion module into the decoding module to obtain the change detection result, i.e., the predicted value; S3, construct a total loss function, calculate the loss between the predicted value and the true value of the sample, loop S2 to S3 until the total loss function converges, and then use the back propagation algorithm to update the weights of the remote sensing change detection network to obtain the trained remote sensing change detection network; S4. Perform BN layer fusion and network weight quantization on the trained remote sensing change detection network, deploy the quantized remote sensing change detection network in FPGA and test it to obtain the result graph after change detection.

2. The remote sensing image change detection method according to claim 1, characterized in that: The construction of the remote sensing change detection network in S1 includes the following steps: S11. First, the common unit and the downsampling unit in the lightweight deep learning network are improved respectively to obtain an improved lightweight deep learning network; S12, building an efficient early fusion module based on the improved lightweight deep learning network and the improved common unit in S11; S13. A decoding module connected with an efficient early fusion module is constructed based on an improved lightweight deep learning network. The efficient early fusion module and the decoding module constitute a remote sensing change detection network.

3. The remote sensing image change detection method according to claim 2, characterized in that: The S11 specifically includes the following steps: S111, respectively removing the channel shuffling operations in the common unit and the downsampling unit in the lightweight deep learning network; thus, an improved common unit is obtained; S112. Replace the channel-by-channel convolution layer with a stride of 2 in the downsampling unit after removing the channel shuffling operation with a maximum pooling layer with a stride of 2×2, and then connect a channel-by-channel convolution layer with a stride of 1 to the maximum pooling layer. At this point, an improved downsampling unit and an improved lightweight deep learning network are obtained.

4. The remote sensing image change detection method according to claim 3, characterized in that: The decoding module in S13 includes a lightweight backbone network, two lightweight feature pyramid structures and a classification layer: The lightweight backbone network is connected to two lightweight feature pyramid structures respectively, and the two lightweight feature pyramid structures are connected to the classification layer respectively; The lightweight backbone network mainly consists of a plurality of sequentially connected combined unit blocks, each of which contains one improved downsampling unit and three improved ordinary units; Both lightweight feature pyramid structures are composed of upsampling operators, feature addition operators, point-by-point convolution layers, and channel-by-channel convolution layers, and are responsible for fusing and decoding features of multiple different scales.

5. The remote sensing image change detection method according to claim 3, characterized in that: The S2 specifically includes the following steps: S21, selecting a dual-phase image from the data set, the dual-phase image including two remote sensing images in different time periods in the same area; and then inputting the dual-phase image into a remote sensing change detection network; S22, the dual-phase image first undergoes a point-by-point convolution, a channel-by-channel convolution layer, and a 2×2 maximum pooling layer in the efficient early fusion module to extract initial features and obtain initial features; S23, after the initial features are extended through a point-by-point convolution channel, the features are enhanced through n improved common units, and then two enhanced features are obtained through a point-by-point convolution compression channel; S24, performing a channel superposition operation on the two enhanced features, i.e., a Concat operation, and then performing point-by-point convolution to obtain the output of the efficient early fusion module; S25, inputting the output of the efficient early fusion module into the decoding module, and the output of the efficient early fusion module is passed through a lightweight backbone network to generate multiple feature maps of different scales; S26. Input multiple feature maps of different scales into two lightweight feature pyramid structures for fusion decoding, and finally obtain the change detection result through the classification layer. The change detection result is the output of the decoding module.

6. The remote sensing image change detection method according to claim 1, characterized in that: The total loss function in S3 is as follows: Loss = αLoss1 + βLoss2 Among them, Loss represents the total loss function, Loss1 and Loss2 represent the supervised guided loss function and the binary cross entropy loss function respectively; a and β represent the proportional coefficients of the supervised guided loss function and the binary cross entropy loss function respectively; The calculation formula of the supervised guided loss function Loss1 is as follows: Loss1=-[1+GlogP-Glog(1-P)] Among them, P and G represent the predicted value and sample true value output by the efficient early fusion module respectively; The calculation formula of the binary cross entropy loss function Loss2 is as follows: Among them, H and W are the height and width of the output feature map respectively; h and w represent the rows and columns of the corresponding pixels respectively; g h,w represents the true value of the sample corresponding to the pixel in the hth row and the wth column; p h,w Represents the predicted value of the pixel corresponding to the hth row and the wth column.

7. The remote sensing image change detection method according to claim 1, characterized in that: The S4 specifically includes the following steps: S41, performing BN layer fusion and network weight quantization on the trained remote sensing change detection network, and exporting the quantized remote sensing change detection network and storing it in the SD card in the FPGA development board; S42, HLS technology is used to design and optimize operators in the quantized remote sensing change detection network in a highly parallel manner, and an FPGA accelerator system is built based on these operators; S43, collecting the image to be detected, and inputting the image to be detected into the FPGA accelerator system of the remote sensing change detection network after quantization is deployed, and the FPGA accelerator system performs board-level reasoning verification to obtain a result map after change detection.

8. The remote sensing image change detection method according to claim 7, characterized in that: The S41 specifically includes the following steps: S411. In the trained remote sensing change detection network, the convolution operation including the BN layer is subjected to BN layer fusion optimization. After the fusion optimization, a fused convolution module is obtained. The convolution operation of the fused convolution module is specifically as follows: Y′ conv =w′x+b′ Among them, Y′ conv represents the output of the fused convolution module; w′ represents the weight of the convolution kernel of the fused convolution module; x represents the input; b′ represents the bias term of the fused convolution module, that is, the output after the BN layer; Wherein, the weight w′ and the bias term b′ are calculated by the following formula: Among them, Y represents the scaling factor, w represents the weight of the convolution kernel of the convolution module before fusion optimization; σ represents the standard deviation of the input data, ε represents a fixed value that tends to zero to prevent the denominator from being 0; b represents the bias term of the convolution module before fusion optimization, that is, the output of the convolution module before fusion optimization; μ represents the average value of the input data; β represents another bias term; S412, quantizing the parameters of the 32-bit floating-point arithmetic numbers in the fused and optimized remote sensing change detection network obtained in S411 into fixed-point integers to reduce the computing and storage space occupied by the remote sensing change detection network; the quantization calculation is specifically as follows: Y fixed =round(Y float ×2 i ) Among them, Y fixed Represents a quantized fixed-point integer, round is a rounding function, Y float Represents the floating point number to be quantized, i represents the quantization factor; At this point, the quantized remote sensing change detection network is obtained; S413, exporting the weights in the quantized remote sensing change detection network obtained in S412 to the SD card on the FPGA development board for use in remote sensing change detection network reasoning.

9. The remote sensing image change detection method according to claim 7, characterized in that: The S42 specifically includes the following steps: S421. Select an FPGA development board. The ZYNQ series development board includes an ARM end, an FPGA end, and related peripheral devices that are interconnected. The ARM end and the FPGA end communicate through the AXI bus. The ARM end is responsible for image input and decoding and task scheduling, and the FPGA end is responsible for computing and reasoning acceleration in the remote sensing change detection network. S422. Use HLS to design operator IPs corresponding to the image preprocessing unit, the channel-by-channel convolution unit, the point-by-point convolution unit, the maximum pooling unit, the upsampling unit, the feature addition unit, and the post-processing unit on the FPGA side; S423. Build a quantized remote sensing change detection network on the ARM side, and obtain the built FPGA accelerator system by calling the image preprocessing IP, channel-by-channel convolution IP, point-by-point convolution IP, maximum pooling IP, upsampling IP, feature addition IP and post-processing IP designed in S422.

10. The remote sensing image change detection method according to any one of claims 7 to 9, characterized in that: The S43 specifically includes the following steps: S431, collecting the dual-phase remote sensing image to be detected and storing it in the same SD card with the weight file of the quantized remote sensing change detection network; S432, the FPGA accelerator system loads the dual-phase remote sensing image to be detected and the weight file of the quantized remote sensing change detection network into the DDR, starts the constructed FPGA accelerator system to obtain the final change detection result map, and stores it in the SD card.

Images