Land cover classification method based on visible light and infrared satellite image fusion

Abstract

This invention discloses a ground object classification method based on the fusion of visible light and infrared satellite images, comprising: acquiring multiple sets of corresponding visible light images and infrared satellite images containing ground targets; labeling the ground targets in the visible light images and infrared satellite images to obtain labels, generating a ground object classification training dataset, wherein each training data in the ground object classification training dataset includes a corresponding visible light image and an infrared satellite image, and their corresponding ground target labels; constructing a ground target classification network; training the ground target classification network using the ground object classification training dataset; and performing ground target classification using the trained ground target classification network. This invention, by constructing a ground target classification network and using visible light images and infrared satellite images as inputs to the ground target classification network, can simultaneously extract the spatial and spectral features of ground targets, significantly improving the simultaneous detection and classification accuracy of various ground targets.

Description

Technical Field

[0001] This invention relates to the field of image information processing technology, and in particular to a method for classifying ground features based on the fusion of visible light and infrared satellite images. Background Technology

[0002] With the continuous development of remote sensing technology, the spatial resolution of satellite images has been constantly improving, providing strong support for researchers to acquire geographic information. It has significant application value in fields such as environmental monitoring, crop cover and type analysis, land observation, and urban planning and management. Satellite image semantic segmentation, as one of the core tasks of remote sensing image interpretation, is a key research topic in the field of computer vision. It assigns a label to each pixel for pixel-level classification. Traditional semantic segmentation methods include edge-based, cluster-based, and threshold-based methods. However, traditional semantic segmentation methods can only extract low-level features from satellite images, which cannot meet the requirements of practical land cover classification applications.

[0003] With the continuous development of deep learning technology, current research has applied convolutional neural networks to satellite image ground feature classification, proposing classic network architectures such as FCN, SegNet, U-Net, and DeepLab for satellite image ground feature classification.

[0004] Since satellite image land cover classification is a pixel-level classification task, each target in a satellite image carries semantic information and needs to be accurately separated from neighboring targets, which places pixel-level accuracy requirements on the network. Moreover, compared with the semantic segmentation task of everyday targets, satellite image land cover classification also pays attention to "background" targets such as buildings, water systems, and vegetation. Existing deep learning-based land cover classification methods all use a single satellite image for land cover classification, resulting in low classification accuracy and efficiency. Summary of the Invention

[0005] To address some or all of the technical problems existing in the prior art, this invention provides a method for land cover classification based on the fusion of visible light and infrared satellite images.

[0006] The technical solution of the present invention is as follows:

[0007] A method for land cover classification based on the fusion of visible light and infrared satellite imagery is provided, the method comprising:

[0008] Acquire multiple sets of corresponding visible light and infrared satellite images containing ground targets;

[0009] Ground targets in visible light images and infrared satellite images are labeled to obtain labels and generate a ground target classification training dataset. Each training data in the ground target classification training dataset includes a corresponding visible light image and an infrared satellite image, and their corresponding ground target labels.

[0010] Construct a ground target classification network;

[0011] A ground object classification network was trained using a ground object classification training dataset.

[0012] Ground target classification is performed using a trained ground target classification network.

[0013] In some possible implementations, multiple sets of corresponding visible light and infrared satellite images containing ground targets are acquired, including:

[0014] Select multiple satellite multispectral images containing ground targets;

[0015] The RGB three-channel data of the satellite multispectral image are fused to obtain the corresponding visible light image, and the infrared channel data of the satellite multispectral image is extracted to obtain the corresponding infrared satellite image.

[0016] In some possible implementations, ground targets in visible light images and infrared satellite images are labeled to obtain labels and generate a ground feature classification training dataset, including:

[0017] Rectangular bounding boxes are used to label different ground targets in visible light images and infrared satellite images respectively. The center point coordinates, width and height of the rectangles, and target categories within the rectangles are determined, and labels including the center point coordinates, width, height, and category information of the rectangles are obtained.

[0018] A visible light image and an infrared satellite image, along with the corresponding ground target labels, are used as training data to generate a ground feature classification training dataset that includes multiple training data.

[0019] In some possible implementations, a ground target classification network is built based on the YOLOv5 algorithm.

[0020] In some possible implementations, the ground target classification network includes: an input layer, a focus module, a backbone network, a fast spatial pyramid pooling module, a neck network, a decoding layer, and an output layer connected in sequence. The backbone network adopts a CSP structure, the neck network adopts a feature fusion network structure of feature pyramid network and pixel aggregation network, and the decoding layer adopts a multi-scale feature fusion network structure.

[0021] In some possible implementations, the fast spatial pyramid pooling structure includes:

[0022] Two CBS modules are provided. The input of the first CBS module is the input of the fast spatial pyramid pooling structure, and the output is connected to the input of the first max pooling module and the input of the Concat layer, respectively. The input of the second CBS module is connected to the output of the Concat layer, and the output is the output of the fast spatial pyramid pooling structure. The CBS module includes a convolutional layer, a data normalization layer and an activation function layer connected in sequence.

[0023] The three max pooling modules are configured such that the output of the first max pooling module is connected to the input of the second max pooling module and the input of the Concat layer, the output of the second max pooling module is connected to the input of the third max pooling module and the input of the Concat layer, and the output of the third max pooling module is connected to the input of the Concat layer.

[0024] The Concat layer is used to connect the input features.

[0025] Among some possible implementations, a ground object classification network is trained using a ground object classification training dataset, including:

[0026] The visible light images and infrared satellite images in the training data of the ground object classification training dataset are used as input, and the ground object labels in the training data are used as output to train the ground object classification network.

[0027] In some possible implementations, a ground object classification network is trained by taking visible light images and infrared satellite images from the training data in the ground object classification training dataset as input and ground object labels from the training data as output. Further implementations include:

[0028] Step S41: Input the visible light images and infrared satellite images from multiple training data into the ground target classification network in sequence to obtain the prediction results output by the ground target classification network;

[0029] Step S42: Compare the prediction results output by the ground target classification network with the labels in the training data, and calculate the prediction accuracy of the ground target classification network.

[0030] Step S43: Determine whether the prediction accuracy obtained in at least two consecutive steps is greater than the preset accuracy threshold. If yes, use the current ground target classification network as the trained ground target classification network. If no, calculate the preset classification loss function, update the parameters of the ground target classification network using the preset classification loss function, and return to step S41.

[0031] In some possible implementations, the default classification loss function is:

[0032] L = L loc +L cls +L obj +L seg

[0033] Among them, L loc L represents the localization loss. cls L represents the classification loss. obj L represents the confidence loss. seg Indicates the segmentation loss;

[0034] Positioning loss L loc Represented as:

[0035]

[0036] Classification loss L cls Represented as:

[0037]

[0038] Confidence loss L obj Represented as:

[0039]

[0040] Segmentation loss L seg Represented as:

[0041]

[0042] Where S×S represents the number of grids the object detection network divides the input image into, and M represents the number of candidate bounding boxes generated by each grid. This indicates whether the target is located in the j-th candidate target box of the i-th grid. If yes, If not, C represents the predicted confidence value of the target in the i-th grid. i This represents the true confidence score of the target in the i-th grid, where c represents the target class and class represents the total class. p represents the predicted classification probability of class c in the i-th grid. i (c) represents the true classification probability of class c in the i-th grid, IoU represents the intersection-over-union ratio between the predicted and ground truth bounding boxes, and Δ represents the distance loss. γ = 2 - Λ, e represents the natural constant. and b represents the x and y coordinates of the center point of the actual target bounding box, respectively. cx and b cyc represents the x-coordinate and y-coordinate of the predicted target box center point, respectively. w and c h Let represent the width and height of the bounding rectangles of the predicted and ground truth bounding boxes, respectively; Λ represent the angular loss; α represent the angle between the line connecting the centers of the predicted and ground truth bounding boxes and the x-axis; and Ω represent the shape loss. w gt and h gt θ represents the width and height of the actual target bounding box, respectively; w and h represent the width and height of the predicted target bounding box, respectively; and θ represents the preset shape cost.

[0043] In some possible implementations, the parameters of the ground target classification network are updated using the following formula:

[0044]

[0045] Where θ represents the parameter set of the ground target classification network, Δ[·] represents the optimizer, and η represents the learning rate.

[0046] The main advantages of the technical solution of this invention are as follows:

[0047] The ground object classification method based on the fusion of visible light and infrared satellite images of the present invention constructs a ground target classification network and uses visible light images and infrared satellite images as inputs to the ground target classification network. It can simultaneously extract the spatial features and spectral features of ground targets, realize the joint utilization of spatial information and spectral information, significantly improve the simultaneous detection and classification accuracy of various ground targets, and achieve pixel-level classification of various ground targets. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and constitute a part of this invention, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0049] Figure 1 This is a flowchart of a land cover classification method based on the fusion of visible light and infrared satellite images according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of a ground target classification network according to an embodiment of the present invention;

[0051] Figure 3 for Figure 2 The diagram shows the structure of the fast spatial pyramid pooling module in the ground target classification network.

[0052] Figure 4 for Figure 3The diagram shows the structure of the CBS in the fast spatial pyramid pooling module.

[0053] Figure 5 This is a schematic diagram of the vector angle between the predicted target box and the true target box provided in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram comparing classification results under different classification networks, as an example of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] See Figure 1 An embodiment of the present invention provides a method for land cover classification based on the fusion of visible light and infrared satellite images, the method comprising the following steps S1-S5:

[0058] Step S1: Acquire multiple sets of corresponding visible light images and infrared satellite images containing ground targets.

[0059] In one embodiment of the present invention, multiple sets of corresponding visible light images and infrared satellite images containing ground targets are obtained by utilizing satellite multispectral images.

[0060] Specifically, in one embodiment of the present invention, acquiring multiple sets of corresponding visible light images and infrared satellite images containing ground targets includes:

[0061] Select multiple satellite multispectral images containing ground targets;

[0062] The RGB three-channel data of the satellite multispectral image are fused to obtain the corresponding visible light image, and the infrared channel data of the satellite multispectral image is extracted to obtain the corresponding infrared satellite image.

[0063] In one embodiment of the present invention, the satellite multispectral images are selected from the BigEarthNet-S2 remote sensing image dataset. BigEarthNet-S2 is a large-scale remote sensing image dataset constructed by the Remote Sensing Image Analysis Group and Database Systems and Information Management Group of the Technical University of Berlin. The dataset consists of 125 Sentinel-2 satellite images, covering ten European countries from June 2017 to May 2018, and is divided into 590,326 non-overlapping image tiles. The dataset includes 19 label categories such as buildings, roads, water systems, and vegetation.

[0064] In one embodiment of the present invention, ENVI software is used to perform data fusion and data extraction processing on satellite multispectral images to obtain corresponding visible light images and infrared satellite images.

[0065] Step S2: Annotate the ground targets in the visible light image and the infrared satellite image, obtain the labels, and generate a ground target classification training dataset. Each training data in the ground target classification training dataset includes a corresponding visible light image and an infrared satellite image, and their corresponding ground target labels.

[0066] In one embodiment of the present invention, ground targets in visible light images and infrared satellite images are labeled to obtain labels and generate a ground feature classification training dataset, including:

[0067] Rectangular bounding boxes are used to label different ground targets in visible light images and infrared satellite images respectively. The center point coordinates, width and height of the rectangles, and target categories within the rectangles are determined, and labels including the center point coordinates, width, height, and category information of the rectangles are obtained.

[0068] A visible light image and an infrared satellite image, along with the corresponding ground target labels, are used as training data to generate a ground feature classification training dataset that includes multiple training data.

[0069] In one embodiment of the present invention, the selected ground targets include: buildings, roads, water systems, and vegetation.

[0070] Step S3: Construct a ground target classification network.

[0071] In one embodiment of the present invention, a ground target classification network is constructed based on the YOLOv5 algorithm.

[0072] Specifically, see Figure 2In one embodiment of the present invention, the ground target classification network includes: an input layer, a focus module, a backbone network, a fast spatial pyramid pooling module (SPPF), a neck network, a decoding layer, and an output layer connected in sequence. The backbone network adopts a CSP structure, the neck network adopts a feature fusion network structure of feature pyramid network (FPN) and pixel aggregation network (PAN), and the decoding layer adopts a multi-scale feature fusion network structure.

[0073] In one embodiment of the present invention, the Focus module is used to slice the image before it enters the backbone network of the ground target classification network. Specifically, the Focus module divides the original data input to the ground target classification network into four parts, which is equivalent to downsampling the original data by a factor of 2. Then, it concatenates the data along the channel dimension and finally performs a convolution operation to obtain a downsampled feature map without information loss, thereby reducing the number of floating-point operations per second (FLOPS) and improving the computation speed.

[0074] Furthermore, in one embodiment of the present invention, the backbone network adopts a CSPDarknet53 structure.

[0075] Further, see Figure 3 In one embodiment of the present invention, the fast spatial pyramid pooling module includes:

[0076] There are two CBS modules. The input of the first CBS module is the input of the fast spatial pyramid pooling module, and the output is connected to the input of the first max pooling module and the input of the Concat layer, respectively. The input of the second CBS module is connected to the output of the Concat layer, and the output is the output of the fast spatial pyramid pooling module.

[0077] There are three max pooling modules (MaxPool). The output of the first max pooling module is connected to the input of the second max pooling module and the input of the Concat layer. The output of the second max pooling module is connected to the input of the third max pooling module and the input of the Concat layer. The output of the third max pooling module is connected to the input of the Concat layer.

[0078] The Concat layer is used to connect the input features.

[0079] Further, see Figure 4 In one embodiment of the present invention, the CBS module includes a convolutional layer (Conv), a batch-normalization (BN) layer, and an activation function layer connected in sequence. The activation function may be a SiLu function.

[0080] Traditional Spatial Pyramid Pooling (SPP) works by performing max pooling on the received feature map at three different scales and then concatenating the feature vectors to obtain a fixed-size feature vector. This can transform feature maps of arbitrary sizes into fixed-size feature vectors, mitigating information loss caused by image distortion, reducing convolution operations, achieving the fusion of local and global features, and enriching the expressive power of feature maps. When the target size varies significantly, it can improve detection accuracy. An embodiment of this invention provides a fast Spatial Pyramid Pooling method that, compared to traditional Spatial Pyramid Pooling, changes the downsampling factor of the pooling layer and replaces the original parallel structure with a cascaded structure. Mathematically, both methods have the same effect, but fast Spatial Pyramid Pooling has fewer floating-point operations per second (FLOPS) and is faster, while retaining more information in multi-scale feature extraction.

[0081] Furthermore, since there are always categories with significantly different target sizes within an image, theoretically, after the same feature extraction, each pixel on the feature map can perceive the entire image. However, in practice, due to the limited neighborhood perception ability of convolutional kernels and the layering and sliding windowing methods, the perception is stronger closer to the target center, while the target edges receive very little feature extraction. In land cover classification, typical elements include buildings, roads, water bodies, and vegetation. Different categories of targets exhibit significant differences in size and geometric features. For example, the size of building targets differs greatly from that of water bodies and vegetation targets, and the elongated geometry of roads differs greatly from the geometry of water bodies and vegetation targets. This imbalance in sample distribution caused by these differences poses a challenge to land cover classification tasks. In one embodiment of the present invention, a multi-scale feature fusion network is set in the decoding layer. The multi-scale feature fusion network extracts features with different receptive fields on the feature map, and then fuses these features to obtain a feature that can perceive both large and small targets. The feature map with a large receptive field can connect with contextual information during training and adjust in time for some unreasonable predictions, which can further improve the network performance of the ground target classification network.

[0082] In one embodiment of this invention, a ground target classification network is constructed based on the YOLOv5 algorithm. The backbone network employs a CSP structure inspired by Cross-Stage Local Networks (CSPNet), dividing the feature maps of the base layers into two parts, which are then merged through a cross-stage hierarchical structure. This reduces computational cost while maintaining accuracy. A fast spatial pyramid pooling structure is introduced between the backbone and the neck network. The neck network utilizes a feature fusion network structure combining Feature Pyramid Network (FPN) and Pixel Aggregation Network (PAN), enhancing the network's feature fusion capabilities and improving the detection and classification performance of the ground target classification network. A decoding layer employing a multi-scale feature fusion network structure is placed after the neck network to further improve the detection and classification performance of the ground target classification network.

[0083] Step S4: Train the ground target classification network using the ground feature classification training dataset.

[0084] After constructing the ground target classification network, in order to improve the classification accuracy of the ground target classification network and reduce the classification error, it is necessary to train and update the ground target classification network using the ground feature classification training dataset.

[0085] In one embodiment of the present invention, training a ground object classification network using a ground object classification training dataset includes:

[0086] The visible light images and infrared satellite images in the training data of the ground object classification training dataset are used as input, and the ground object labels in the training data are used as output to train the ground object classification network.

[0087] Specifically, in one embodiment of the present invention, the visible light images and infrared satellite images in the training data of the ground object classification training dataset are used as input, and the ground object labels in the training data are used as output to train the ground object classification network, further including the following steps S41-S43:

[0088] Step S41: Input the visible light images and infrared satellite images from multiple training data into the ground target classification network in sequence to obtain the prediction results output by the ground target classification network.

[0089] In one embodiment of the present invention, visible light images and infrared satellite images from the training data are input to the input terminal of the ground target classification network. These images are processed sequentially through the parameters of each layer in the ground target classification network and output from the output terminal. The information output is the prediction result corresponding to the visible light image and the infrared satellite image. Specifically, the prediction result includes the center point coordinates, width, height, and category information of the predicted target bounding box of the ground target. The initial ground target classification network can be an untrained ground target classification network or a ground target classification network that has not been fully trained. Each layer of the network has initialized parameters, which can be continuously updated and adjusted during training.

[0090] Traditional YOLO series networks take RGB three-channel images as input, i.e., visible light images. As the number of network layers increases, deep networks will continuously discard the detailed features of the target, leading to a decline in classification performance.

[0091] In one embodiment of the present invention, visible light images and infrared satellite images are used as inputs to the constructed ground target classification network. That is, in addition to the visible light image input channel in the ground target classification network, an infrared image input channel is added, and the visible light image and infrared image are merged into a four-channel image input to the ground target classification network. This can compensate for the loss of ground target features during the detection and classification process and improve the classification performance of the ground target classification network.

[0092] Step S42: Compare the prediction results output by the ground target classification network with the labels in the training data, and calculate the prediction accuracy of the ground target classification network.

[0093] In one embodiment of the present invention, the prediction accuracy of the ground target classification network can be calculated by comparing the prediction results output by the ground target classification network with the labels in the training data. Specifically, the prediction results output by the ground target classification network, which include the center point coordinates, width, height, and category information of the predicted target box of the ground target, are compared with the labels, which include the center point coordinates, width, height, and category information of the real target box of the ground target.

[0094] Step S43: Determine whether the prediction accuracy obtained in at least two consecutive steps is greater than the preset accuracy threshold. If yes, use the current ground target classification network as the trained ground target classification network. If no, calculate the preset classification loss function, update the parameters of the ground target classification network using the preset classification loss function, and return to step S41.

[0095] In one embodiment of the present invention, the preset classification loss function includes a localization loss component, a classification loss component, a confidence loss component, and a segmentation loss component, specifically described as follows:

[0096] L = Lloc +L cls +L obj +L seg

[0097] Among them, L loc L represents the localization loss. cls L represents the classification loss. obj L represents the confidence loss. seg This represents the segmentation loss.

[0098] Further, see Figure 5 , attached Figure 5 In the diagram, b represents the predicted target box. gt In one embodiment of the present invention, when determining the localization loss, the vector angle between the predicted target box and the true target box is introduced to construct the SIoU loss function to describe the localization loss.

[0099] Specifically, the positioning loss L loc Represented as:

[0100]

[0101] Where IoU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, and Δ represents the distance loss. γ = 2 - Λ, e represents the natural constant. and b represents the x and y coordinates of the center point of the actual target bounding box, respectively. cx and b cy c represents the x-coordinate and y-coordinate of the predicted target box center point, respectively. w and c h Let represent the width and height of the bounding rectangles of the predicted and ground truth bounding boxes, respectively; Λ represent the angular loss; α represent the angle between the line connecting the centers of the predicted and ground truth bounding boxes and the x-axis; and Ω represent the shape loss. w gt and h gt θ represents the width and height of the actual target bounding box, respectively; w and h represent the width and height of the predicted target bounding box, respectively; and θ represents the preset shape cost, used to control attention to shape loss.

[0102] According to the positioning loss formula set above, when α→0, the distance loss tends to be constant, which can reduce the degree of freedom of network regression, accelerate network convergence, and improve regression accuracy.

[0103] Furthermore, in one embodiment of the present invention, the classification loss L cls And confidence loss L obj Both use the binary cross-entropy loss function, specifically expressed as:

[0104]

[0105]

[0106] Segmentation loss L seg Set to:

[0107]

[0108] Where S×S represents the number of grids that the ground object classification network divides the input image into, and M represents the number of candidate object boxes generated by each grid. This indicates whether the target is located in the j-th candidate target box of the i-th grid. If yes, If not, C represents the predicted confidence value of the target in the i-th grid. i This represents the true confidence score of the target in the i-th grid, where c represents the target class and class represents the total class. p represents the predicted classification probability of class c in the i-th grid. i (c) represents the true classification probability of class c in the i-th grid.

[0109] Furthermore, in one embodiment of the present invention, the parameters of the ground target classification network are optimized and updated using gradient descent. Specifically, the parameters are first differentiated using the loss function using the chain rule, and then the parameters are updated using the differentiation result and the preset learning rate.

[0110] Specifically, the parameters of the ground target classification network can be updated using the following formula:

[0111]

[0112] Where θ represents the parameter set of the ground target classification network, Δ[·] represents the optimizer, and η represents the learning rate. The optimizer and learning rate can be selected according to the actual situation. The optimizer can be Adam, SGD, etc., and the learning rate is used to control the speed of parameter updates.

[0113] Step S5: Use the trained ground target classification network to classify ground targets.

[0114] Specifically, after training the ground target classification network, the corresponding visible light and infrared images to be detected and classified are input into the ground target classification network to obtain the corresponding information of the ground targets in the images.

[0115] An embodiment of the present invention provides a ground object classification method based on the fusion of visible light and infrared satellite images. By constructing a ground target classification network and using visible light images and infrared satellite images as inputs to the ground target classification network, it can simultaneously extract the spatial features and spectral features of ground targets, realize the joint utilization of spatial information and spectral information, significantly improve the simultaneous detection and classification accuracy of various ground targets, and achieve pixel-level classification of various ground targets.

[0116] The following examples illustrate the beneficial effects of a land cover classification method based on the fusion of visible light and infrared satellite images provided by an embodiment of the present invention:

[0117] In this example, the BigEarthNet-S2 remote sensing image dataset is used to create the ground cover classification training dataset. The obtained ground cover classification training dataset is divided into training, validation, and test sets in a ratio of 7:2:1 for subsequent training, validation, and testing of the target classification network. When training the target classification network, the SGD optimizer is selected, with an initial learning rate of 0.01, momentum of 0.9, and weight decay of 0.00001. The performance evaluation metrics for the target classification network are mIoU, false alarm rate, and false negative rate.

[0118] Based on the above-mentioned ground object classification training dataset, parameters, and performance evaluation indicators, the target classification network was trained, validated, and tested. The performance comparison results of the ground target classification network obtained based on the method provided in an embodiment of the present invention, the single-modal target classification network based on visible light images, and the single-modal target classification network based on infrared images are shown in Table 1 below.

[0119] Table 1 Performance Comparison Results of Target Classification Networks

[0120]

[0121] As shown in the table above, the overall performance index of the ground target classification network obtained by the method provided in an embodiment of the present invention is better than that of the single-modal target classification network, and its corresponding mIoU can reach 61.24%.

[0122] Further, see Figure 6 , attached Figure 6 The diagram shows the original image, the classification results of the original image using a ground target classification network according to an embodiment of the present invention, the classification results of the original image using a visible light-based single-modal target classification network, and the annotation of the real targets. (According to the appendix...) Figure 6 The results show that, compared with a single-modal target classification network, the ground target classification network obtained based on the method provided in an embodiment of the present invention can detect various ground targets more accurately.

[0123] Furthermore, based on the above-mentioned ground object classification training dataset, parameters, and performance evaluation indicators, the ground object classification network before and after the introduction of the multi-scale feature fusion network was trained, validated, and tested, and the performance comparison results of the ground object classification network before and after the introduction of the multi-scale feature fusion network were obtained as shown in Table 2 below.

[0124] Table 2 shows the performance comparison of ground target classification networks before and after introducing a multi-scale feature fusion network.

[0125]

[0126] As shown in the table above, after introducing a multi-scale feature fusion network into the ground target classification network, the mIoU increased from 43.15% to 61.24%, which means that the introduction of a multi-scale feature fusion network retains more contextual semantic information of the image and can significantly improve the network performance.

[0127] Furthermore, based on the above-mentioned ground object classification training dataset, parameters, and performance evaluation indicators, different target classification networks were trained, validated, and tested, resulting in the performance comparison results of the ground target classification network obtained based on the method provided in an embodiment of the present invention and various traditional target classification networks, as shown in Table 3 below.

[0128] Table 3 Performance Comparison Results of Different Target Classification Networks

[0129]

[0130] As can be seen from the table above, the performance indicators of the ground target classification network obtained based on the method provided in one embodiment of the present invention are superior to those of other types of target classification networks.

[0131] It should be noted that, in this document, relational terms such as “first” and “second” are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for land cover classification based on the fusion of visible light and infrared satellite images, characterized in that, include: Acquire multiple sets of corresponding visible light and infrared satellite images containing ground targets; Ground targets in visible light images and infrared satellite images are labeled to obtain labels and generate a ground target classification training dataset. Each training data in the ground target classification training dataset includes a corresponding visible light image and an infrared satellite image, and their corresponding ground target labels. Construct a ground target classification network; A ground object classification network was trained using a ground object classification training dataset. Ground target classification is performed using a trained ground target classification network. The ground target classification network is constructed based on the YOLOv5 algorithm. The ground target classification network includes: an input layer, a focus module, a backbone network, a fast spatial pyramid pooling module, a neck network, a decoding layer, and an output layer connected in sequence. The backbone network adopts a CSP structure, the neck network adopts a feature fusion network structure of feature pyramid network and pixel aggregation network, and the decoding layer adopts a multi-scale feature fusion network structure. The focus module divides the original data input to the ground target classification network into 4 parts, then concatenates them in the channel dimension, and finally performs a convolution operation to obtain a double-downsampled feature map without information loss. The fast spatial pyramid pooling structure includes: two CBS modules, the first CBS module's input being the input of the fast spatial pyramid pooling structure, and its output being connected to the input of the first max pooling module and the input of the Concat layer, respectively; the second CBS module's input being connected to the output of the Concat layer, and its output being the output of the fast spatial pyramid pooling structure; each CBS module includes a convolutional layer, a data normalization layer, and an activation function layer connected in sequence; three max pooling modules, the first max pooling module's output being connected to the input of the second max pooling module and the input of the Concat layer, respectively; the second max pooling module's output being connected to the input of the third max pooling module and the input of the Concat layer, respectively; and the third max pooling module's output being connected to the input of the Concat layer; the Concat layer is used to concatenate the input features. In this method, visible light images and infrared images are combined into a four-channel image input to the ground target classification network.

2. The method for land cover classification based on visible light and infrared satellite image fusion according to claim 1, characterized in that, Acquire multiple sets of corresponding visible light and infrared satellite images containing ground targets, including: Select multiple satellite multispectral images containing ground targets; The RGB three-channel data of the satellite multispectral image are fused to obtain the corresponding visible light image, and the infrared channel data of the satellite multispectral image is extracted to obtain the corresponding infrared satellite image.

3. The method for land cover classification based on visible light and infrared satellite image fusion according to claim 1, characterized in that, Ground targets in visible light images and infrared satellite images are labeled to obtain labels and generate a ground feature classification training dataset, including: Rectangular bounding boxes are used to label different ground targets in visible light images and infrared satellite images respectively. The center point coordinates, width and height of the rectangles, and target categories within the rectangles are determined, and labels including the center point coordinates, width, height, and category information of the rectangles are obtained. A visible light image and an infrared satellite image, along with the corresponding ground target labels, are used as training data to generate a ground feature classification training dataset that includes multiple training data.

4. The method for land cover classification based on visible light and infrared satellite image fusion according to any one of claims 1-3, characterized in that, Training a ground object classification network using a ground object classification training dataset includes: The visible light images and infrared satellite images in the training data of the ground object classification training dataset are used as input, and the ground object labels in the training data are used as output to train the ground object classification network.

5. The method for land cover classification based on visible light and infrared satellite image fusion according to claim 4, characterized in that, Using visible light images and infrared satellite images from the training data in the ground object classification training dataset as input and ground object labels from the training data as output, a ground object classification network is trained, further including: Step S41: Input the visible light images and infrared satellite images from multiple training data into the ground target classification network in sequence to obtain the prediction results output by the ground target classification network; Step S42: Compare the prediction results output by the ground target classification network with the labels in the training data, and calculate the prediction accuracy of the ground target classification network. Step S43: Determine whether the prediction accuracy obtained in at least two consecutive steps is greater than the preset accuracy threshold. If yes, use the current ground target classification network as the trained ground target classification network. If no, calculate the preset classification loss function, update the parameters of the ground target classification network using the preset classification loss function, and return to step S41.

6. The method for land cover classification based on visible light and infrared satellite image fusion according to claim 5, characterized in that, The preset classification loss function is: ； in, This represents the preset classification loss function. Indicates location loss. Represents classification loss. Indicates confidence loss. Indicates the segmentation loss; Location loss Represented as: ； Classification loss Represented as: ； Confidence loss Represented as: ； Segmentation loss Represented as: ； in, This represents the number of grid cells the object detection network divides the input image into. This indicates the number of candidate bounding boxes generated by each grid cell. This indicates whether the target is located in the j-th candidate target box of the i-th grid. If yes, , if not, , This represents the predicted confidence value of the target in the i-th grid. This represents the true confidence value of the target in the i-th grid. Indicates the target category, Indicates the general category. This represents the predicted classification probability of class c in the i-th grid. This represents the true classification probability of class c in the i-th grid. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Indicates distance loss. , , , , , Represents the natural constant. and These represent the x and y coordinates of the center point of the actual target bounding box, respectively. and These represent the x and y coordinates of the predicted target box center point, respectively. and Let represent the width and height of the bounding rectangles of the predicted bounding box and the ground truth bounding box, respectively. Indicates angle loss, This represents the angle between the line connecting the centers of the predicted bounding box and the ground truth bounding box and the x-axis. Indicates shape loss, , , , and These represent the width and height of the actual target bounding box, respectively. and These represent the width and height of the predicted bounding box, respectively. This indicates the cost of the preset shape.

7. The method for land cover classification based on visible light and infrared satellite image fusion according to claim 6, characterized in that, The parameters of the ground target classification network are updated using the following formula: ； in, This represents the set of parameters for a ground target classification network. Indicates the optimizer, This represents the learning rate.

Images