Model training method and device, storage medium and electronic equipment

Abstract

The invention discloses a model training method and device, a storage medium and electronic equipment. The method comprises the following steps: firstly, inputting an unlabeled remote sensing image into an initial classification model to obtain a pseudo-label remote sensing image, and then training the initial classification model based on the unlabeled remote sensing image, the pseudo-label remote sensing image and an initial training data set to obtain a target classification model. A round of training of the initial classification model comprises a first training stage and a second training stage. In the first training stage, the initial classification model is trained by using a pseudo-label remote sensing image and an initial training data set so as to enhance the adaptability of the model to unseen data. And in the second training stage, focusing on images with relatively high model prediction uncertainty, and cooperatively training the initial classification model with the initial training data set. The technical problem of low classification accuracy of the remote sensing image caused by insufficient robustness and generalization ability of the target classification model trained by the existing training method is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular, to a model training method and device, a storage medium, and an electronic device. BACKGROUND

[0002] Existing training methods are often limited by the scarcity and unevenness of labeled data when constructing classification models, resulting in insufficient robustness and generalization ability of the model. Taking target classification of remote sensing images as an example, when processing multi-source heterogeneous data, the model is difficult to effectively cope with the internal complexity and variability of various ground objects in spatial distribution, geometric shape and spectral characteristics, limiting the accuracy of remote sensing image classification. That is, the robustness and generalization ability of the target classification model trained by the existing training method are insufficient, resulting in the technical problem of low classification accuracy of remote sensing images.

[0003] At present, no effective solution has been proposed to solve the above problems. SUMMARY

[0004] The embodiments of the present application provide a model training method and device, a storage medium, and an electronic device to at least solve the technical problem of insufficient robustness and generalization ability of the target classification model trained by the existing training method, resulting in low classification accuracy of remote sensing images.

[0005] According to an aspect of an embodiment of the present application, a model training method is provided, comprising: inputting an unlabeled remote sensing image into an initial classification model to obtain a pseudo-labeled remote sensing image; training the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image, and an initial training data set to obtain a target classification model, wherein one round of training of the initial classification model includes a first training phase and a second training phase, in the first training phase, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set; in the second training phase, the initial classification model is trained using the unlabeled remote sensing image and the initial training data set that meet the information entropy condition.

[0006] According to another aspect of the embodiments of the present application, a model training apparatus is also provided, comprising: a generation module configured to input an unlabeled remote sensing image into an initial classification model to obtain a pseudo-labeled remote sensing image; and a training module configured to train the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image and an initial training data set to obtain a target classification model, wherein one round of training of the initial classification model comprises a first training stage and a second training stage, in the first training stage, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set, and in the second training stage, the initial classification model is trained using the unlabeled remote sensing image satisfying an information entropy condition and the initial training data set.

[0007] According to still another aspect of the embodiments of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute the model training method when running.

[0008] According to still another aspect of the embodiments of the present application, a computer program product or computer program is provided, and the computer program product or computer program comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the model training method.

[0009] According to still another aspect of the embodiments of the present application, an electronic device is also provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the model training method by using the computer program.

[0010] In the embodiment of the present application, the unlabeled remote sensing image is first input into the initial classification model to obtain a pseudo-labeled remote sensing image. Then, the initial classification model is trained based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image and the initial training data set to obtain a target classification model. Specifically, one round of training of the initial classification model includes a first training phase and a second training phase: in the first training phase, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set to enhance the adaptability of the model to unseen data; in the second training phase, the initial classification model is trained cooperatively with the initial training data set focusing on images with high prediction uncertainty, and the robustness and generalization ability of the target classification model are effectively improved through dynamic optimization of the training data set and fine-tuning of the model parameters. The technical effect of fully mining and utilizing the value of a large number of unlabeled remote sensing images while optimizing the initial training data set in a targeted manner is achieved, ensuring that the initial classification model can continuously learn samples with high information content and representativeness during the training process. Not only does this reduce the dependence on labeled data, but it also improves the classification accuracy, thereby solving the technical problem of insufficient robustness and generalization ability of the target classification model trained by the existing training method, resulting in low classification accuracy of remote sensing images. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0012] Figure 1 is a schematic diagram of an application environment of an optional model training method according to an embodiment of the present application;

[0013] Figure 2 is a flowchart of an optional model training method according to an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of an optional model training according to an embodiment of the present application;

[0015] Figure 4 is a flowchart of an optional model training according to an embodiment of the present application;

[0016] Figure 5 is a flowchart of an optional model training method according to an embodiment of the present application;

[0017] Figure 6 is a schematic diagram of an optional deformable cross-source relationship encoding architecture according to an embodiment of the present application;

[0018] Figure 7 is a schematic diagram of an optional feature skip connection according to an embodiment of the present application;

[0019] Figure 8 is a structural schematic diagram of an optional model training device according to an embodiment of the application;

[0020] Figure 9 is a structural schematic diagram of an optional electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0021] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] The present application will be described below in conjunction with the embodiments:

[0024] According to an aspect of an embodiment of the present application, a model training method is provided.

[0025] Optionally, in the present embodiment, the above-mentioned model training method can be applied to a wide range of remote sensing data analysis fields, applied to natural resource management, environmental protection, urban planning, agricultural monitoring, disaster assessment and the like. Specifically, the above-mentioned model training method can be applied in a hardware environment composed of a server 101 and a terminal device 103 as shown in the following. Figure 1

[0026] Specifically, the training method can be applied in a hardware environment composed of a server 101 and a terminal device 103 as shown in the following. Figure 1 ​The illustrated distributed hardware environment consisting of server 101 and terminal device 103 is effectively applied. Among them, the server 101 as the core computing unit is responsible for storing a large amount of remote sensing image data set and training and optimizing the initial classification model to obtain the target classification model, and its high-performance computing capability and large-capacity storage space provide strong support for the complex operation of the initial classification model. At the same time, the terminal device 103 (such as a drone, a satellite receiving station or a mobile device) can perform data collection and preliminary processing, including but not limited to collecting remote sensing images under different geographical conditions, and preliminarily screening and uploading to the server 101 for further analysis and model training.

[0027] Exemplarily, taking the training method of the above-mentioned model executed by the server 101 and the terminal device 103 as an example, the flow of the method can include the following steps:

[0028] S1, the terminal device 103 collects the original remote sensing image, performs preprocessing operation on the original remote sensing image, including but not limited to radiation correction, geometric correction, image registration, etc., to obtain the unlabeled remote sensing image, and transmits the unlabeled remote sensing image to the server 101, or directly transmits the collected original remote sensing image to the server 101 as the unlabeled remote sensing image;

[0029] S2, the server 101 receives the unlabeled remote sensing image, inputs the unlabeled remote sensing image into the initial classification model stored in itself to obtain the pseudo-labeled remote sensing image;

[0030] S3, the server 101 trains the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image and the initial training data set pre-labeled with object classification labels stored in itself to obtain the target classification model, wherein one round of training of the initial classification model includes a first training phase and a second training phase. In the first training phase, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set; in the second training phase, the initial classification model is trained using the unlabeled remote sensing image satisfying the information entropy condition and the initial training data set.

[0031] It should be noted that the above-mentioned server 101 and terminal device 103 can be any node in a distributed system, wherein the distributed system can be a blockchain system, etc. The blockchain system can be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes can form any form of network, and any form of computing device, such as any electronic device, can become a node in the distributed system by joining the network formed by the nodes.

[0032] The above-mentioned server 101 can be a single server, a server cluster composed of multiple servers, or a cloud server.

[0033] The network can include, but is not limited to, a wired network including a local area network, a metropolitan area network, and a wide area network, and a wireless network including Bluetooth, WIFI, and other networks that enable wireless communication.

[0034] The terminal device 103 can include, but is not limited to, at least one of a satellite, a mobile phone (such as an Android phone, an iOS phone, etc.), a notebook computer, a tablet computer, a palm computer, a MID (Mobile Internet Device), a PAD, a desktop computer, a smart television, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a mixed reality (MR) terminal, and other computer devices.

[0035] Optionally, as an optional implementation, as shown in Figure 2 The training method of the model includes:

[0036] S202, inputting the unlabeled remote sensing image into the initial classification model to obtain a pseudo-labeled remote sensing image;

[0037] Optionally, in the embodiments of the present application, the unlabeled remote sensing image refers to a remote sensing data set that has not been manually labeled, including but not limited to high-resolution optical images, synthetic aperture radar images, laser radar images, hyperspectral images, etc.

[0038] Specifically, the number of unlabeled remote sensing images is not limited in the embodiments of the present application, and the unlabeled remote sensing images are pre-registered in geographical position to ensure that different types of images can be accurately aligned to achieve effective information fusion. Specifically, the unlabeled remote sensing images include at least two types, one of which is a hyperspectral image, and the other is a radar image. For example, the hyperspectral image provides spectral information, and the radar image supplements accurate three-dimensional terrain and structural information.

[0039] Optionally, in the embodiments of the present application, the initial classification model refers to a pre-trained deep learning model, including but not limited to Swin-Transformer, ResNet, U-Net, DeepLab, etc. For example, the initial classification model takes Swin-Transformer pre-trained based on the ImageNet-21K data set as the backbone network, and the model architecture can be as shown in Figure 3 It has strong feature extraction capability and adaptability to high-resolution remote sensing images. By fine-tuning the model parameters, it can be effectively applied to the classification task of multi-source remote sensing images.

[0040] Optionally, in the embodiments of the present application, the pseudo-labeled remote sensing image refers to an image with predicted labels generated after predicting the unlabeled remote sensing image.

[0041] S204, training the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image and the initial training data set to obtain a target classification model, wherein one round of training of the initial classification model includes a first training phase and a second training phase, in the first training phase, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set; in the second training phase, the initial classification model is trained using the unlabeled remote sensing image satisfying the information entropy condition and the initial training data set.

[0042] Optionally, in the embodiments of the present application, the initial training data set contains a set of labeled remote sensing images, including but not limited to hyperspectral images, synthetic aperture radar images and lidar images, etc.

[0043] Specifically, the embodiments of the present application do not limit the number of remote sensing images in the initial training data set, and the remote sensing images in the initial training data set are all pre-processed by the registration operation of geographical position, to ensure that different types of images can be accurately aligned to realize effective information fusion. Specifically, the remote sensing images in the initial training data set include at least two types, one of which is a hyperspectral image, and the other is a radar image, which is consistent with the above-mentioned unlabeled remote sensing image.

[0044] Optionally, in the embodiments of the present application, the first training phase refers to a sample expansion training phase, in which the initial classification model not only uses the existing initial training data set with real labels, but also uses the pseudo-labeled remote sensing image obtained by predicting the unlabeled remote sensing image in advance as a training sample for model training.

[0045] Optionally, in the embodiments of the present application, the second training phase refers to a sample updating training phase, in which the initial classification model actively labels the pixels in the unlabeled remote sensing image that meet a specific condition (such as the information entropy condition), further updates the training data set using the labeled remote sensing image, and improves the model performance. The labeling method here includes but is not limited to manual labeling, automatic labeling, etc., and the present application does not limit the specific labeling method.

[0046] It can be understood that the initial training data set includes a plurality of image pairs, each image pair including a hyperspectral image and a radar image at a corresponding position, wherein the hyperspectral image provides spectral information, and the radar image supplements spatial structure and terrain information, which together constitute the basis for model training.

[0047] In an exemplary embodiment, the training framework of the initial segmentation model is as followsFigure 4 The overall training process is shown in FIG. 1. Figure 5 As shown in FIG. 1, the model training is divided into four stages: model initialization, sample expansion training stage, sample update training stage, and full parameter fine-tuning stage. In the model initialization stage, a small amount of manually labeled remote sensing images are used for pre-training, and only the deep model parameters are updated at this time; in the sample expansion training stage, the training set is expanded using pseudo labels, the deep parameters are continuously updated, and the shallow parameters are kept unchanged; in the sample update training stage, high-confidence pseudo label images are obtained as new labeled remote sensing images, and finally, after accumulating a large number of new labeled remote sensing images, the full parameter fine-tuning stage is entered, the shallow parameters are unfrozen, and all model parameters are comprehensively updated to obtain the target classification model. Taking agricultural monitoring as an example, including but not limited to:

[0048] S1, collecting unlabeled remote sensing images, which include but are not limited to remote sensing images of farmland collected from different data sources such as hyperspectral satellites and laser radar unmanned aerial vehicles. The unlabeled remote sensing images cover different stages of the crop growth cycle, as well as different weather and lighting conditions;

[0049] S2, inputting the unlabeled remote sensing images into an initial classification model to obtain pseudo label remote sensing images, the initial classification model can be a Swin-Transformer pre-trained using an ImageNet-21K dataset;

[0050] S3, training the initial classification model based on the unlabeled remote sensing images, the pseudo label remote sensing images, and the initial training dataset pre-labeled with object classification labels to obtain the target classification model. Specifically, the initial classification model can be determined as the target classification model after N rounds of training, and each round of training at least includes the above first training stage and the second training stage, including but not limited to:

[0051] S3-1, in the first training stage, updating the initial training dataset according to the pseudo label remote sensing images to obtain an intermediate training dataset, and training the initial classification model using the intermediate training dataset;

[0052] After training the initial classification model using the intermediate training dataset for a preset number of training sub-rounds, proceed to S2 step;

[0053] S3-2, in the second training stage, performing object labeling operation on the unlabeled remote sensing images that meet the information entropy condition to obtain new labeled remote sensing images, and training the initial classification model based on the new labeled remote sensing images.

[0054] Optionally, in the embodiments of the present application, the object labeling operation includes but is not limited to being performed by the following way: artificial inspection and correction of the pseudo labels generated by the model to ensure the accuracy and integrity of the labeled data, or direct artificial labeling.

[0055] It can be understood that the overall training process of the initial classification model is as shown in Figure 1 The training process of the initial classification model is as shown in

[0056] Specifically, the next round of training of the initial classification model based on the unlabeled remote sensing images without image annotation operation, the pseudo-label remote sensing images and the initial training data set can include but is not limited to the following methods:

[0057] S1, inputting the unlabeled remote sensing images without image annotation operation in the current training round into the initial classification model to obtain first pseudo-label remote sensing images;

[0058] After the initial classification model is trained with the intermediate training data set for a preset number of training sub-rounds, step S2 is entered;

[0059] S2, training the initial classification model based on the unlabeled remote sensing images without image annotation operation in the current training round, the first pseudo-label remote sensing images and the initial training data set updated in the current training round to obtain a target classification model.

[0060] Specifically, the initial training data set is updated using the first pseudo-label remote sensing image in the first training stage, so as to obtain a first intermediate training data set, and the first intermediate training data set is further divided into a plurality of training batches, and training is performed in batches until the initial classification model is trained for a preset number of training sub-rounds using the first intermediate training data set, then the training data is further updated, the object labeling operation is performed on the unlabeled remote sensing image meeting the information entropy condition to obtain a first newly labeled remote sensing image, the initial training data set is updated according to the first newly labeled remote sensing image to obtain a first target training data set, and the initial classification model is further trained using the first target training data set. When the initial classification model meets the preset model convergence condition, the initial classification model is determined as the target classification model. Conversely, the above update of the training data is continued to enter the next round of training.

[0061] It should be noted that the pseudo-label remote sensing image can be added to the initial training data set to obtain an intermediate training data set, wherein the number of pseudo-label remote sensing images in the intermediate training data set is greater than the number of images in the initial training data set, and the pseudo-label remote sensing image can be added to the initial training data set, which can include but is not limited to the following operations:

[0062] S1, inputting the unlabeled remote sensing image into the initial classification model to determine the probability value of each pixel in the unlabeled remote sensing image belonging to each object category;

[0063] S2, obtaining a preset adjustment coefficient and determining the mean parameter and the standard deviation parameter corresponding to each object category, wherein the adjustment coefficient decreases with the increase of the number of training rounds;

[0064] S3, setting a pseudo-label for each pixel in the unlabeled remote sensing image based on the adjustment coefficient, the probability value mean parameter and the probability value standard deviation parameter corresponding to each object category to obtain a pseudo-label remote sensing image.

[0065] Specifically, S3 can include but is not limited to the following steps:

[0066] S3-1, determining the product of the adjustment coefficient and the probability value standard deviation parameter corresponding to the target object category, determining the difference between the product and the probability value mean parameter corresponding to the target object category as an adaptive threshold, wherein the target object category is any object category;

[0067] S3-2, setting the pseudo-label of the target pixel in the case that the target pixel meets the target pixel condition, wherein the pseudo-label of the target pixel is used to indicate that the target pixel belongs to the target object category, and the target pixel is any pixel in the unlabeled remote sensing image;

[0068] S3-3, not setting the pseudo-label of the target pixel in the case that the target pixel does not meet the target pixel condition.

[0069] Further, the determination of whether the target pixel condition is met or not can include but is not limited to being determined by the following ways:

[0070] S3-3-a, in a case where the probability value of the target pixel belonging to the target object category is greater than or equal to the adaptive threshold, it is determined that the target pixel meets the target pixel condition;

[0071] S3-3-b, in a case where the probability value of the target pixel belonging to the target object category is less than the adaptive threshold, it is determined that the target pixel does not meet the target pixel condition;

[0072] S3-3-c, in a case where the probability value of any pixel in the unlabeled remote sensing image belonging to the target object category is less than the adaptive threshold, it is determined that the target pixel meets the target pixel condition, wherein the probability value of the target pixel belonging to the target object category is in a preset interval, the preset interval is related to the probability value of the maximum pixel, and the probability value of the maximum pixel belonging to the target object category is greater than the probability value of any pixel other than the maximum pixel belonging to the target object category.

[0073] It can be understood that the determination of whether the target pixel condition is met or not in S3-3-a to S3-3-c above is not related, and can be any combination or a single condition, which is not limited in the present application.

[0074] It should be further pointed out that the newly labeled remote sensing image can be added to the initial training data set to obtain the target training data set.

[0075] It should be further pointed out that the unlabeled remote sensing image can be determined to meet the information entropy condition in a case where the number of the unlabeled remote sensing image is greater than the preset target image number, and the sampling parameter of the unlabeled remote sensing image is greater than or equal to the preset sampling parameter threshold, and then the object labeling operation can be performed on the unlabeled remote sensing image; the unlabeled remote sensing image can also be determined to meet the information entropy condition in a case where the number of the unlabeled remote sensing image is less than or equal to the target image number, and then the object labeling operation can be performed on the unlabeled remote sensing image.

[0076] Optionally, in the embodiments of the present application, the target image number can be a threshold number of each class of sample, and is related to the performance evaluation result of the current model, so as to dynamically adjust the model training strategy.

[0077] Optionally, in the embodiments of the present application, the sampling parameter threshold can be a parameter for deciding whether to manually label the unlabeled image, and is related to the number of labeled samples in the initial training data set and the current learning state of the model, and by adjusting the threshold, the efficiency and accuracy of model iteration can be controlled.

[0078] It should be further noted that whether the model convergence condition is met or not can include but is not limited to being determined by the following manner:

[0079] S1, verifying the initial classification model using the remote sensing image test set to obtain an average intersection over union parameter, and in a case where a variation range of the average intersection over union parameter in continuous N rounds of training is less than a preset variation parameter, determining that the initial classification model meets the model convergence condition, where N is a positive integer;

[0080] S2, in a case where a number of times of generating the newly-labeled remote sensing image is equal to a preset generation number, determining that the initial classification model meets the model convergence condition;

[0081] S3, in a case where the object labeling operation is performed on each unlabeled remote sensing image, determining that the initial classification model meets the model convergence condition.

[0082] It can be understood that the steps of whether the model convergence conditions of S1 to S3 are met are not associated with each other, and can be any combination item or a single condition, and the present application does not make any limitation on this.

[0083] Optionally, in the embodiment of the present application, the sampling parameter of the unlabeled remote sensing image is used to indicate index information quantifying the potential contribution of the image to the model training, which can include but is not limited to being determined by the following steps:

[0084] S1, determining target probability values of each pixel in the unlabeled remote sensing image belonging to each object category;

[0085] S2, determining an uncertainty parameter and a clustering distance parameter according to the target probability values, where the uncertainty parameter is used to indicate a confidence degree of object classification of the unlabeled remote sensing image, and the clustering distance parameter is used to indicate a feature coverage degree of the unlabeled remote sensing image;

[0086] S3, performing a weighted summation operation on the uncertainty parameter and the distance parameter to obtain the sampling parameter of the unlabeled remote sensing image.

[0087] Optionally, in the embodiment of the present application, the uncertainty parameter refers to an index measuring the uncertainty of the model prediction result, which includes but is not limited to being determined by calculating an entropy of a classification probability distribution of each pixel of the model, and the greater the entropy value, the higher the classification uncertainty of the model, and the more suitable the image is as a new sample for labeling to provide an opportunity for the model to learn new features.

[0088] Optionally, in the embodiments of the present application, the clustering distance parameter is used to quantify the diversity and coverage of the features in the unlabeled image, including but not limited to extracting the last layer features of the decoder, performing clustering analysis on the global features, and calculating the distance from the sample to the nearest cluster center to determine. The greater the distance, the more representative the sample, the wider the feature coverage, and the more it can contribute to the generalization ability of the model.

[0089] Through the embodiments of the present application, the unlabeled remote sensing image is first input into the initial classification model to obtain the pseudo-labeled remote sensing image; then, the initial classification model is trained based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image and the initial training data set to obtain the target classification model. Specifically, one round of training of the initial classification model includes a first training phase and a second training phase: in the first training phase, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set to enhance the adaptability of the model to unseen data; in the second training phase, the initial classification model is trained cooperatively with the initial training data set focusing on images with high prediction uncertainty, and the robustness and generalization ability of the target classification model are effectively improved through dynamic optimization of the training data set and fine-tuning of the model parameters. The technical effect of fully mining and utilizing the value of a large number of unlabeled remote sensing images while optimizing the initial training data set in a targeted manner is achieved, ensuring that the initial classification model can continuously learn samples with high information content and representativeness during the training process. Not only does it reduce the dependence on labeled data, but it also improves the classification accuracy, thereby solving the technical problem of insufficient robustness and generalization ability of the target classification model trained by the existing training method, resulting in low classification accuracy of remote sensing images.

[0090] As an optional solution, before the unlabeled remote sensing image is input into the initial classification model to obtain the pseudo-labeled remote sensing image, the method further includes: obtaining the initial training data set, wherein the initial training data set includes at least one image pair, the image pair includes a hyperspectral image and a lidar image representing the same area, and the images in the initial training data set all carry object classification labels; and training the initial classification model using the initial training data set.

[0091] It should be noted that the hyperspectral image and the lidar image in the present embodiment both need to be geographically registered to ensure the spatial correspondence between the images, so as to facilitate the model to learn the combined features of the two images. At the same time, the number and type of image pairs in the present embodiment are not limited to specific values, and can be flexibly adjusted according to actual task requirements and data availability.

[0092] For example, the image pair can also include a synthetic aperture radar (SAR) image to adapt to more complex remote sensing data fusion tasks.

[0093] Exemplarily, the embodiment of the application first uses an initial training data set containing artificial annotation to pre-train an initial classification model, so that the model can initially master the feature correlation of different modal remote sensing images. In the pre-training stage, the model learns how to extract and fuse features from hyperspectral images and laser radar images through a supervised learning mechanism such as cross-entropy loss, to achieve accurate classification of ground objects. After pre-training, the model already has basic classification capabilities and can begin to try to generate pseudo-labels for unlabeled remote sensing images.

[0094] In an exemplary embodiment, the land use classification application scenario in urban planning is taken as an example:

[0095] S1, obtain large-area remote sensing images of a specific area, including but not limited to hyperspectral images (HSI), laser radar images (LiDAR), and synthetic aperture radar images (SAR), for subsequent pseudo-label generation and model iterative training.

[0096] S2, annotate the remote sensing images, including but not limited to artificial annotation, use model prediction to annotate and perform artificial correction, to obtain an initial training data set;

[0097] S2, use the initial training data set to supervise the training of the initial classification model.

[0098] Through the embodiment of the application, the joint feature extraction and fusion of multi-source remote sensing image pairs is adopted, achieving the technical effect of fully pre-training the classification model in the initial stage, and laying a foundation for the sample expansion in the first training stage and the sample update optimization process in the second training stage, to improve the initial performance of the model in the fusion classification of hyperspectral and laser radar images.

[0099] As an optional solution, the above training of the above initial classification model using the above initial training data set includes: performing a spectral feature extraction operation on a target hyperspectral image to obtain shallow spectral features and deep spectral features, performing a radar feature extraction operation on a target radar image to obtain shallow radar features and deep radar features, wherein the target hyperspectral image and the target radar image belong to the same image pair; performing a first feature fusion operation on the deep spectral features and the deep radar features to obtain initial fusion features; performing a second feature fusion operation on the shallow spectral features, the shallow radar features, and the initial fusion features to obtain target fusion features; and updating model parameters of the initial classification model according to the target fusion features.

[0100] Optionally, in the embodiment of the application, the target hyperspectral image refers to a high-resolution image representing any specific geographic area, which can be used to identify different types of ground objects.

[0101] Optionally, in this embodiment, the spectral feature extraction operation refers to performing multi-level feature extraction on the hyperspectral image through a specific network module (such as Swin-Transformer combined with the SE module), including but not limited to shallow spectral features and deep spectral features. Shallow spectral features typically reflect the basic spectral characteristics of the image, while deep spectral features contain more complex and abstract spectral details.

[0102] Similarly, target radar imagery refers to radar images paired with hyperspectral images that cover the same geographical area and can provide spatial structure information of ground objects.

[0103] Optionally, in this embodiment, radar feature extraction refers to using a specific neural network architecture (such as Swin-Transformer combined with SAM module) to extract shallow and deep radar features from radar images, providing support for subsequent feature fusion and classification.

[0104] It should be noted that the extraction of spectral and radar features can employ different neural network architectures and techniques, and is not limited to Swin-Transformer, SE, and SAM modules. Furthermore, the first and second feature fusion operations are not fixed in form; they can be any form of feature integration strategy, such as weighted summation, concatenation, attention mechanisms, or more advanced fusion modules.

[0105] For example, a Swin-Transformer is used as the backbone network for feature extraction, combined with SE and SAM modules to enhance the representation capabilities of spectral and spatial features, respectively. In the feature fusion stage, deformable cross-source relation coding (CRE) is used for dynamic feature shifting and alignment. The architecture of this deformable cross-source relation coding is illustrated below. Figure 6 As shown, where, Indicates deep spectral characteristics, To represent deep radar features, a weighted bilinear operation is used to interact deep spectral features with deep radar features, resulting in an initial fused feature. Subsequently, a skip connection mechanism is used to integrate shallow spectral features, shallow radar features, and the initial fused feature to generate a target fused feature. This feature integrates multi-scale and multi-modal information, providing a more comprehensive description of the spectral and spatial attributes of ground features.

[0106] In one exemplary embodiment, taking crop disease monitoring in precision agriculture as an example:

[0107] S1 acquires a hyperspectral image and a radar image of the target, both representing the same farmland area, and are an image pair belonging to the initial training dataset.

[0108] S2 performs spectral feature extraction on the target hyperspectral image, using the Swin-Transformer and SE modules to obtain shallow and deep spectral features. Similarly, it performs radar feature extraction on the target radar image, using the Swin-Transformer and SAM modules to obtain shallow and deep radar features.

[0109] S3 feeds deep spectral features and deep radar features into the improved CRE module. By dynamically learning the local offset of the features, feature alignment is achieved. Then, feature interaction and fusion are performed through outer product operation to obtain the initial fused features.

[0110] S4, shallow spectral features, shallow radar features, and initial fusion features are further fused in the decoder through a skip connection mechanism to generate target fusion features. This feature retains shallow detail information while integrating deep semantic information, thus enhancing the model's classification performance.

[0111] S5 uses target fusion features as input, trains the model through a decoder and a classification head, updates the model parameters based on the comparison between the model's prediction results and the real land cover labels, and optimizes the model's classification performance.

[0112] Through the embodiments of this application, multi-level feature extraction and dynamic fusion are employed to achieve efficient training of the initial classification model. Specifically, by extracting and fusing shallow and deep spectral and radar features from the initial classification model, the model's ability to understand and classify multi-source remote sensing data is effectively improved, reducing the reliance on extensive manual annotation and achieving the goal of training a high-precision remote sensing image classification model on a limited labeled dataset.

[0113] As an optional approach, the first feature fusion operation performed on the deep spectral features and the deep radar features to obtain initial fused features includes: performing an offset prediction operation on the deep spectral features and the deep radar features to obtain spatial offset parameters; performing a feature offset operation on the deep radar features based on the spatial offset parameters to adjust the deep radar features to the feature space corresponding to the deep spectral features, thereby obtaining radar alignment features; performing a weighted outer product operation on the radar alignment features and the deep spectral features to obtain the first fused features; and adjusting the feature dimensions of the first fused features using a convolutional compression interaction matrix to obtain the initial fused features.

[0114] Optionally, in the embodiments of this application, deep spectral features refer to high-level feature representations rich in spectral information of ground features extracted from the target hyperspectral image by the initial classification model, while deep radar features refer to depth features reflecting the shape and structure of ground features extracted from the target radar image by the initial classification model.

[0115] Optionally, in the embodiments of this application, the weighted outer product operation is used to fuse two feature matrices, which can capture the high-order interdependencies between different features and generate a more comprehensive feature representation.

[0116] Optionally, in this embodiment, the convolution compression interaction matrix adjustment is used to adjust the feature dimension generated by the outer product operation to ensure the consistency and applicability of the feature fusion.

[0117] It should be noted that the module performing the offset prediction operation can be based on, for example... Figure 6 The learnable offset prediction network shown, as part of a deformable convolutional network, is used to dynamically estimate the geometric deviation of radar features relative to spectral features. Feature offset operations can be implemented through interpolation, deformable convolution, or other image transformation techniques to ensure precise spatial alignment between radar and spectral features.

[0118] Furthermore, the weighted outer product operation can be implemented using a learned dynamic weight matrix to adjust the importance of different features during the outer product process, thereby more effectively preserving key information during fusion. The operation of adjusting the convolutional compression interaction matrix can be achieved through the cascading of multiple convolutional layers, reducing feature dimensionality while maintaining the integrity of key feature information, thus optimizing the model's computational efficiency and storage requirements.

[0119] For example, in the feature offset operation, the coordinates of the radar features are adjusted according to these offset parameters to precisely match the spectral features in space. Next, a weighted outer product operation is used to combine the radar alignment features and the spectral features to generate a first fused feature reflecting the cross-modal feature interaction. Finally, by adjusting the interaction matrix through convolution compression, the dimension of the first fused feature is adjusted to match the single-modal features, forming an initial fused feature that includes both subtle differences in spectral information and structural details of the radar data.

[0120] In one exemplary embodiment, taking the application scenario of urban building recognition as an example:

[0121] S1. Select hyperspectral images and radar images containing urban topography from unlabeled remote sensing images, ensuring that both cover the same geographical area, and extract deep spectral features and deep radar features.

[0122] S2 performs offset prediction operations on deep spectral features and deep radar features to obtain spatial offset parameters that reflect the relative spatial position deviation between the two features.

[0123] S3. Based on the predicted spatial offset parameters, a feature offset operation is performed on the deep radar features to adjust the position of the radar features to achieve precise alignment with the spectral features, thus obtaining the radar aligned features.

[0124] S4 feeds radar alignment features and deep spectral features into the weighted outer product operation module to generate a first fused feature that can reflect cross-modal feature interaction.

[0125] S5, through the convolution compression interaction matrix adjustment step, compresses the dimension of the first fused feature to ensure that the output initial fused feature has the same size as the single-modal feature, which facilitates the subsequent fusion with other layer features.

[0126] This application's embodiments employ a deformable cross-source relation encoding mechanism to achieve precise alignment and high-order fusion of deep spectral features and deep radar features. Specifically, by dynamically learning the local spatial offset of features, the deep radar features are adjusted to ensure accurate spatial matching. Then, through a weighted outer product operation, high-order interaction information between different modal features is captured, generating a comprehensive initial fused feature. Finally, by adjusting the interaction matrix through convolution compression, the dimensionality of the fused feature is optimized, ensuring the model's computational efficiency and generalization ability when processing multi-source features.

[0127] As an optional approach, the second feature fusion operation performed on the shallow spectral features, shallow radar features, and initial fusion features to obtain target fusion features includes: determining a first shallow spectral feature among the shallow spectral features and determining a first shallow radar feature among the shallow radar features, wherein the first shallow spectral feature is the first spectral feature generated among the shallow spectral features, and the first shallow radar feature is the first spectral feature generated among the shallow radar features; performing an accumulation operation on the first shallow spectral feature, the first shallow radar feature, and the initial fusion features to obtain a bottom-level fusion feature; performing an upsampling operation on the bottom-level fusion features to obtain an upsampled fusion feature; and determining the target fusion feature based on the upsampled fusion feature, the shallow spectral features, and the shallow radar features.

[0128] As an optional approach, determining the target fusion feature based on the upsampled fusion feature, the shallow spectral feature, and the shallow radar feature includes: determining a second shallow spectral feature among the shallow spectral features and determining a second shallow radar feature among the shallow radar features, wherein the second shallow spectral feature is generated later than the first shallow spectral feature, and the second shallow radar feature is generated later than the first shallow radar feature; splicing the upsampled fusion feature, the second shallow spectral feature, and the second shallow radar feature to obtain a spliced ​​fusion feature; performing a feature refinement operation on the spliced ​​fusion feature to obtain a mid-level fusion feature; and performing the upsampled operation on the mid-level fusion feature to obtain the target fusion feature.

[0129] Optionally, in the embodiments of this application, the first shallow spectral features and the first shallow radar features refer to the most basic features extracted from the hyperspectral image and the radar image, respectively, reflecting the intuitive information and preliminary feature representation of the original image, and are generated in the front-end layer of the initial classification model (such as Stage 1 or Stage 2 of Swin-Transformer).

[0130] Optionally, in the embodiments of this application, the accumulation operation is a way to merge multiple feature representations into a comprehensive feature. It can be a simple element-level addition, a weighted summation, or feature integration through other network modules (such as gating mechanisms). The purpose is to fuse features from different modalities and levels to obtain a more comprehensive representation of ground features.

[0131] Optionally, in this embodiment, the upsampling operation is used to restore the resolution of the features to be close to or equal to the size of the original unlabeled remote sensing image, so as to retain sufficient detail information in the final classification stage.

[0132] Optionally, in the embodiments of this application, the splicing and fusion feature is to combine the upsampling fusion feature, the second shallow spectral feature, and the second shallow radar feature according to the channel dimension to form a composite feature representation. This step is beneficial for integrating multimodal and multi-level information.

[0133] Optionally, in the embodiments of this application, the feature refinement operation may include, but is not limited to, further processing of spliced ​​and fused features through residual modules, attention mechanisms or other neural network components to remove redundant information and enhance the recognizability of features.

[0134] In one exemplary embodiment, taking the application scenario of urban building detection as an example:

[0135] S1, determine the first shallow spectral features (HSI shallow features) and the first shallow radar features (SAR / LiDAR shallow features), which reflect basic information of the original image, such as color, texture and elevation profile.

[0136] S2, the first shallow spectral features, the first shallow radar features, and the initial fusion features obtained by deformable cross-source relation encoding are accumulated to generate the bottom-level fusion features, which initially combine multimodal information.

[0137] S3 performs an upsampling operation on the underlying fused features to restore their resolution and obtain the upsampled fused features.

[0138] S4 determines the second shallow-layer spectral features and the second shallow-layer radar features, which contain detailed information after further processing.

[0139] S5 combines the upsampling fusion features, the second shallow spectral features, and the second shallow radar features to form a spliced ​​fusion feature, thus realizing the hierarchical fusion of shallow information.

[0140] S6 performs feature refinement operations on the splicing and fusion features, such as removing redundant information and enhancing key features through residual blocks or attention gating units, to obtain mid-level fusion features.

[0141] S7, perform upsampling operation again on the mid-layer fusion features to restore the resolution that matches the original image, and finally obtain the target fusion features.

[0142] Through the embodiments of this application, a multi-level feature fusion and optimization strategy is adopted to achieve efficient integration of shallow spectral features, shallow radar features, and initial fused features. This significantly improves the model's feature representation capability and classification accuracy, achieving the goal of training a high-accuracy classification model on limited labeled datasets and multi-source remote sensing images.

[0143] As an optional approach, the above-mentioned input of unlabeled remote sensing images into an initial classification model to obtain pseudo-labeled remote sensing images includes: inputting the unlabeled remote sensing images into the initial classification model to determine the probability value of each pixel in the unlabeled remote sensing images belonging to each object category; obtaining a preset adjustment coefficient and determining the mean parameter and standard deviation parameter corresponding to each of the object categories, wherein the adjustment coefficient decreases as the number of training rounds increases; and setting pseudo-labels for each pixel in the unlabeled remote sensing images based on the adjustment coefficient, the mean parameter and standard deviation parameter of the probability value corresponding to each of the object categories, to obtain the pseudo-labeled remote sensing images.

[0144] Optionally, in the embodiments of this application, the adjustment coefficient is a parameter used to control the strictness of pseudo-label generation. It is usually set to a larger value in the early stage of training to ensure the high quality of pseudo-labels. As training progresses, the adjustment coefficient is gradually reduced to relax the standard for pseudo-label generation and allow more unlabeled data to participate in training.

[0145] Optionally, in this embodiment, the probability mean parameter and standard deviation parameter are statistical measures obtained by statistically analyzing each object category based on the model prediction results. These are used to dynamically adjust the generation threshold of pseudo-labels to ensure the reliability and diversity of pseudo-labels.

[0146] It should be noted that the dynamic adjustment strategy for the adjustment coefficient can be customized according to actual needs and data characteristics, including but not limited to linear decay, exponential decay, or adaptive adjustment algorithms, to achieve the best pseudo-label generation effect. The mean and standard deviation parameters of the probability values ​​corresponding to the object categories can be calculated in real time using sliding window techniques or batch statistical methods to ensure dynamic monitoring and feedback of the model prediction results.

[0147] For example, for each category Calculate the probability of all pixels predicted to be of this class. And calculate the probability value mean parameter. With standard deviation parameter , ,in For prediction as a category The total number of pixels, i and j are used to indicate pixel positions. The adaptive threshold is then calculated. , To adjust the coefficients, the stringency of the threshold is reduced in each iteration: ,initial , This is an adaptive threshold decay coefficient. The threshold is iteratively changed to ensure accuracy and utilize more data.

[0148] Furthermore, retain the satisfaction The pixels are used as high-confidence pseudo-labels, and pseudo-label masks are generated to obtain the pseudo-label dataset. (That is, the above-mentioned images were labeled with pseudo-labels). In addition, if no pixels of a certain category meet the threshold, the top 5% of pixels with the highest probability of that category are retained as high-confidence pseudo-labels to avoid underutilization of data in small categories.

[0149] In one exemplary embodiment, taking the application scenario of crop disease detection as an example:

[0150] S1. Input a batch of unlabeled hyperspectral remote sensing images into the initial classification model. The model predicts the probability that each pixel belongs to a different crop health or disease state.

[0151] S2 represents the mean and standard deviation parameters of the probability values ​​for each crop disease state in the statistical model prediction results.

[0152] S3, calculate the dynamic confidence threshold based on the preset adjustment coefficient (e.g., starting from 1.0 and decreasing by 0.1 in each iteration).

[0153] S4. For each pixel, if the model predicts that the probability of the pixel belonging to a certain disease state is higher than the calculated dynamic confidence threshold, then the pixel is assigned a corresponding pseudo-label; otherwise, the pixel remains unlabeled.

[0154] S5 uses remote sensing images with pseudo-labels as additional training data, along with manually labeled images, for further training of the model, gradually expanding the labeled dataset and optimizing the model parameters.

[0155] This application's embodiments employ dynamic probability adjustment and pseudo-label generation strategies to achieve automatic labeling of high-confidence pixels in unlabeled remote sensing images. This ensures that the model's predictive capabilities are fully utilized with limited manual labeling costs, transforming large amounts of unlabeled data into usable training data.

[0156] As an optional approach, the above-mentioned setting of pseudo-labels for each pixel in the unlabeled remote sensing image based on the adjustment coefficient, the mean probability value parameter and the standard deviation probability value parameter corresponding to each of the above-mentioned object categories, to obtain the pseudo-labeled remote sensing image, includes: determining the product of the adjustment coefficient and the standard deviation probability value parameter corresponding to the target object category; determining the difference between the product and the mean probability value parameter corresponding to the target object category as an adaptive threshold, wherein the target object category is any of the above-mentioned object categories; setting a pseudo-label for the target pixel when the target pixel meets the target pixel condition, wherein the pseudo-label for the target pixel is used to indicate that the target pixel belongs to the target object category, and the target pixel is any pixel in the unlabeled remote sensing image; and not setting a pseudo-label for the target pixel when the target pixel does not meet the target pixel condition.

[0157] As an optional approach, the above method further includes at least one of the following: determining that the target pixel satisfies the target pixel condition when the probability value of the target pixel belonging to the target object category is greater than or equal to an adaptive threshold; determining that the target pixel does not satisfy the target pixel condition when the probability value of the target pixel belonging to the target object category is less than the adaptive threshold; determining that the target pixel satisfies the target pixel condition when the probability value of any pixel in the unlabeled remote sensing image belonging to the target object category is less than the adaptive threshold, wherein the probability value of the target pixel belonging to the target object category is within a preset range, the preset range is related to the probability value of the largest pixel, and the probability value of the largest pixel belonging to the target object category is greater than the probability value of any pixel other than the largest pixel belonging to the target object category.

[0158] Optionally, in the embodiments of this application, the target object category refers to any type of land cover to be classified in the remote sensing image, including but not limited to buildings, vegetation, roads, or water bodies.

[0159] Optionally, in this embodiment, the adaptive threshold is a threshold calculated based on the aforementioned statistical parameters and adjustment coefficients, used to determine whether to assign a pixel to a specific target object category as a high-confidence pseudo-label. The target pixel condition is the criterion for determining whether a pixel meets the criteria to become a pseudo-label candidate, specifically including whether the predicted probability of the pixel belonging to the target object category reaches or exceeds the adaptive threshold.

[0160] It should be noted that the conditions and processes for generating pseudo-labels can be adjusted according to the actual application scenario and data characteristics. For example, the decay strategy of the adjustment coefficient can be linear, exponential, or other non-linear, with the goal of gradually relaxing the pseudo-label generation criteria over time. The calculation method for the adaptive threshold may also vary depending on the category; some rare or difficult-to-distinguish categories may require a higher threshold to ensure the quality of the pseudo-labels.

[0161] In addition, the setting of target pixel conditions can also take into account the spatial connectivity between pixels or global context information to improve the consistency and rationality of pseudo-labels, which is not limited in this application.

[0162] For example, first, the product of the adjustment coefficient and the standard deviation of the probability value of the target object category is determined. Then, the difference between this product and the mean probability value parameter is calculated to obtain the adaptive threshold. Next, each pixel is examined. If the probability value belonging to a certain target object category is greater than or equal to the adaptive threshold, the pixel is considered to meet the target pixel condition, and a pseudo-label is generated for it; otherwise, no pseudo-label is generated. In some cases, even if no pixel's probability value reaches the adaptive threshold, the pixel with the highest probability value within a preset range can be selected to generate a pseudo-label. This ensures that each category has a certain number of pseudo-labels to expand the training data of the initial classification model and avoid model skewness or performance degradation due to scarce samples in some categories.

[0163] In one exemplary embodiment, taking the application scenario of distinguishing different land cover types in environmental monitoring as an example:

[0164] S1, determine the adjustment coefficient for the current training round, assuming it is 0.7, and calculate the mean and standard deviation parameters of the predicted probability for each target object category (such as forest, grassland, water).

[0165] S2, calculate the adaptive threshold based on the adjustment coefficient and the standard deviation parameter of the target object category. For example, for the forest category, the adaptive threshold = mean - adjustment coefficient. Standard deviation.

[0166] S3. Input the unlabeled remote sensing image into the initial classification model to obtain the probability value of each pixel belonging to different target object categories.

[0167] S4. Examine each pixel. If its predicted probability of belonging to the forest category is greater than or equal to the adaptive threshold of 0.7, it is considered to meet the target pixel condition, and a pseudo-label of the forest category is generated for the pixel; if it is less than the adaptive threshold, no pseudo-label is generated.

[0168] S5. In some cases, if no pixel in the forest category meets the condition, select pixels whose predicted probability is within a preset interval (e.g., the 95% interval of the predicted probability of the category) and whose predicted probability is higher than all other pixels except the selected pixels in the interval, and generate pseudo-labels for the forest category for them.

[0169] S6. Repeat steps S4 and S5 until all eligible pixels are assigned pseudo-labels, generating a pseudo-labeled remote sensing image containing pseudo-label information.

[0170] This application's embodiments employ a dynamically adjusted adaptive threshold mechanism to generate pseudo-labels, enabling automatic labeling of high-confidence pixels in unlabeled remote sensing images. This ensures the quality of the pseudo-labels while maximizing the utilization of unlabeled data, promoting model self-optimization and performance improvement.

[0171] As an optional approach, the above-mentioned training of the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image, and the initial training dataset to obtain the target classification model includes: in the first training stage, updating the initial training dataset according to the pseudo-labeled remote sensing image to obtain an intermediate training dataset, and using the intermediate training dataset to train the initial classification model; in the second training stage, performing object labeling operations on the unlabeled remote sensing image that satisfies the information entropy condition to obtain a newly labeled remote sensing image, and training the initial classification model based on the newly labeled remote sensing image.

[0172] It should be noted that the information entropy condition can be set based on the uncertainty of the pixel prediction probability distribution. Generally, high information entropy means that the prediction probability distribution is relatively uniform, and the model has low confidence in classifying such pixels; conversely, low information entropy indicates that the prediction probability distribution is relatively concentrated, and the model has high confidence in classifying pixels. By setting an information entropy threshold, the most uncertain or most certain parts of the model can be filtered out to guide subsequent annotation and model optimization.

[0173] Furthermore, object annotation operations can be pixel-level, object-level, or image-level annotations, depending on application requirements and data characteristics, and this application does not limit them.

[0174] For example, in the first training phase, the model parameters are updated by combining the initial training dataset with pseudo-labeled remote sensing images, enabling the model to initially utilize information from unlabeled data. In the second training phase, the model focuses on unlabeled remote sensing images with low information entropy, performing object labeling operations on the pixels in these images to generate newly labeled remote sensing images, and then training the model to further improve its performance on these high-confidence samples.

[0175] In one exemplary embodiment, taking the application scenario of crop type identification in agricultural monitoring as an example:

[0176] S1, at the beginning of the first training phase, combines a small number of manually labeled crop type samples (such as wheat, corn, and soybean) with pseudo-labeled remote sensing images generated by the model to form an intermediate training dataset.

[0177] S2 uses an intermediate training dataset to train the initial classification model, and the model parameters are adjusted based on the supervision signals from the labeled and pseudo-labeled data.

[0178] S3, entering the second training phase, the model makes predictions on unlabeled remote sensing images and calculates the information entropy of the predicted probability distribution for each pixel.

[0179] S4 identifies pixels whose information entropy is below a preset threshold. These pixels have clear prediction results and high confidence, and are therefore high-quality candidates for manual annotation.

[0180] S5 performs object labeling on the selected pixels, supplements their true category labels, and generates a new labeled remote sensing image.

[0181] S6 uses newly labeled remote sensing images as additional supervisory information to further train and optimize the initial classification model until the predetermined performance metric or training stopping condition is met.

[0182] Through the embodiments of this application, different training datasets are used to train the model at different training stages, achieving efficient utilization of pixels in unlabeled remote sensing images. Throughout the training process, the model can not only perform initial self-boosting using pseudo-labels, but also update samples based on the most valuable samples, ensuring the quality and diversity of the training dataset.

[0183] As an optional approach, the above-mentioned updating of the initial training dataset based on the pseudo-labeled remote sensing images to obtain an intermediate training dataset, and training the initial classification model using the intermediate training dataset, includes: adding the pseudo-labeled remote sensing images to the initial training dataset to obtain the intermediate training dataset, wherein the number of pseudo-labeled remote sensing images in the intermediate training dataset is greater than the number of images in the initial training dataset; training the initial classification model using the intermediate training dataset, determining a first cross-entropy loss value and a second cross-entropy loss value, wherein the first cross-entropy loss value is related to the images in the intermediate training dataset that belong to the initial training dataset, and the second cross-entropy loss value is related to the pseudo-labeled remote sensing images; merging the first cross-entropy loss value and the second cross-entropy loss value to obtain a target loss value, and updating the model parameters of the initial classification model based on the target loss value.

[0184] Optionally, in this embodiment, the first cross-entropy loss value is related to the difference between the true label and the model's predicted label of the manually labeled image, and is used to measure the model's performance on known classification results. The second cross-entropy loss value is related to the difference between the predicted label and the pseudo-label of the pseudo-labeled image, and is used to supervise the model's prediction performance on unlabeled images. The target loss value is the overall loss obtained by comprehensively considering the first and second cross-entropy loss values, and is used to guide the updating of model parameters.

[0185] Optionally, in the embodiments of this application, the above-mentioned merging includes, but is not limited to, weighted summation, gating mechanism fusion, attention-weighted fusion, etc., and this application does not limit it.

[0186] For example, in each training batch, the labeled data from the initial training dataset is... and pseudo-labeled remote sensing images by Hybrid training, calculating the first cross-entropy loss on the labeled data. Calculate the second cross-entropy loss for the high-confidence pseudo-label data (only for high-confidence pseudo-label pixels). The target loss value is obtained by weighted summation. The weighting coefficient of pseudo-labels Set it to 0.5.

[0187] It should be noted that factors such as the frequency of model parameter updates, the weight allocation of the first and second cross-entropy loss values, and the selection criteria and number of pseudo-label images can all be adjusted according to the actual situation to optimize the model training process, and this application does not impose any restrictions on them.

[0188] For example, pseudo-labeled remote sensing images are added to the initial training dataset to form an intermediate training dataset, where the number of pseudo-labeled images exceeds the number of images in the initial training set, to leverage information from unlabeled data to enhance model training. Subsequently, during model training, a first cross-entropy loss value between the model prediction and the true labels of the manually labeled images, and a second cross-entropy loss value between the model prediction and the pseudo labels of the pseudo-labeled images are calculated simultaneously. These two loss values ​​are then combined with certain weights to obtain a target loss value used to guide the update of model parameters.

[0189] In one exemplary embodiment, taking the application scenario of urban land classification as an example:

[0190] S1 collected a small number of manually labeled remote sensing images containing different types of land (such as residential areas, parks, industrial areas, etc.) to form the initial training dataset.

[0191] S2 uses the current version of the initial classification model to predict a large number of unlabeled remote sensing images and generates pseudo-labels for pixels with high confidence in the prediction results.

[0192] S3. These remote sensing images with high-confidence pseudo-labels are added to the initial training dataset to form an intermediate training dataset, in which the number of pseudo-labeled images exceeds the number of images in the initial training set.

[0193] S4. Start a new round of model training, using the intermediate training dataset as input, calculate the first cross-entropy loss value between the model prediction and the true label of the initial training set, and the second cross-entropy loss value between the model prediction and the pseudo-label of the pseudo-label image.

[0194] S5, set the weights of the first and second cross-entropy loss values, for example, the former is weighted at 0.7 and the latter at 0.3, and combine the two according to this ratio to obtain the target loss value.

[0195] S6 updates the model parameters based on the target loss value. By minimizing the target loss value, the model can exhibit good classification performance on both manually labeled data and pseudo-labeled data.

[0196] This application's embodiments employ a pseudo-label augmentation and hybrid loss function strategy to train the model, achieving full utilization of unlabeled data information with limited manually labeled data. Specifically, by adding images with high-confidence pseudo-labels to the initial training set to form an intermediate training set, and by comprehensively considering the loss contributions of manually labeled data and pseudo-labeled data during training, the model's classification ability is effectively improved, while reducing dependence on expensive manually labeled data, thus achieving the goal of improving model generalization ability while reducing costs.

[0197] As an optional approach, the above-mentioned object labeling operation performed on the unlabeled remote sensing image that satisfies the information entropy condition to obtain a new labeled remote sensing image includes: determining the target probability value of each pixel in the target unlabeled remote sensing image belonging to each object category, wherein the target unlabeled remote sensing image is any of the above-mentioned unlabeled remote sensing images; determining an uncertainty parameter and a clustering distance parameter based on the target probability value, wherein the uncertainty parameter is used to indicate the confidence level of the object classification of the target unlabeled remote sensing image, and the clustering distance parameter is used to indicate the feature coverage of the target unlabeled remote sensing image; performing a weighted summation operation on the uncertainty parameter and the distance parameter to obtain a target sampling parameter; and performing the above-mentioned object labeling operation on the target unlabeled remote sensing image when the target unlabeled remote sensing image satisfies the above-mentioned information entropy condition to obtain a new labeled remote sensing image.

[0198] As an optional approach, the above method further includes at least one of the following: when the number of unlabeled remote sensing images is greater than the number of target images, and the target sampling parameter is greater than or equal to a preset sampling parameter threshold, the unlabeled remote sensing image of the target image satisfies the information entropy condition; when the number of unlabeled remote sensing images is less than or equal to the number of target images, the unlabeled remote sensing image of the target image satisfies the information entropy condition.

[0199] Optionally, in the embodiments of this application, the target probability value refers to the probability distribution of each pixel in the image belonging to each object category as predicted by the model.

[0200] Optionally, in the embodiments of this application, the uncertainty parameter quantifies the confidence of the model in the image classification decision, and is usually calculated based on the entropy of the predicted probability. The larger the entropy value, the higher the uncertainty of the model in the classification result.

[0201] Optionally, in the embodiments of this application, the clustering distance parameter describes the distance between the target image features and the set of labeled image features, reflecting the feature coverage and uniqueness of the image.

[0202] Optionally, in this embodiment, the target sampling parameter is a comprehensive evaluation index obtained by weighted summation of the uncertainty parameter and the clustering distance parameter, used to determine which images are most valuable for manual annotation. The information entropy condition is a screening criterion used to determine which unannotated images should be prioritized for manual annotation to improve model performance.

[0203] It should be noted that the weight allocation in the weighted summation, the setting of the information entropy condition, and the threshold of the target sampling parameters can all be adjusted according to the specific application scenario and data characteristics to achieve the best annotation and model optimization results. The selection of unlabeled remote sensing images of the target can be based on a comprehensive consideration of uncertainty parameters and clustering distance parameters, or on one of them alone, depending on the stage and requirements of model training. This application does not impose any restrictions on this.

[0204] For example, the target probability value of each pixel belonging to each object category is first calculated. Then, based on these probability values, the image's uncertainty parameter and clustering distance parameter are determined. These two parameters are then weighted and summed according to preset weights to obtain the target sampling parameters. If the target sampling parameters satisfy the information entropy condition (e.g., high uncertainty or high clustering distance), the unlabeled remote sensing image is considered valuable and should be manually labeled to generate a new labeled remote sensing image. This process enables the model to selectively label images with high information content, high classification difficulty, or significant uniqueness, thereby improving the quality of the training dataset and the model's generalization ability.

[0205] For example, the prediction entropy of each pixel is calculated. Where x indicates the input target unlabeled image, Indicates position The predicted probability that a pixel belongs to class n, where, For row index, For column indexes, This is used as a category index. This leads to the uncertainty parameters. .

[0206] For example, extracting the features of the last layer of the decoder from the initial classification model. Image-level features are obtained through global average pooling. And perform K-means clustering ( ), calculate the distance from the sample to the nearest cluster center Finally, the cluster distance parameters are obtained by weighting. ,set up Choose the one with the higher score Manual annotation is performed, among which, Indicates the minimum guaranteed quantity. This indicates the percentage of the sampled data.

[0207] It should be noted that cluster centers refer to the representative points of each cluster obtained through iterative optimization in the K-means clustering algorithm. In the feature space of remote sensing images, cluster centers reflect the central location of the distribution of image features of different types, and are obtained by calculating the average value of the sample features within each cluster.

[0208] In one exemplary embodiment, taking the application scenario of remote sensing image land cover classification as an example:

[0209] S1, identify the target unlabeled remote sensing image, i.e., an unlabeled land cover image.

[0210] S2, using the current version of the classification model, predict the probability that each pixel in the image belongs to a different land cover category, and obtain the target probability value distribution.

[0211] S3, based on the target probability value, calculate the uncertainty parameters (such as prediction entropy or maximum class probability value) of the unlabeled remote sensing image of the target, as well as the clustering distance parameter between its feature vector and the features of the labeled image.

[0212] S4. Set the weight of the uncertainty parameter to 0.6 and the weight of the clustering distance parameter to 0.4, and perform a weighted summation operation to obtain the target sampling parameters.

[0213] S5. Check whether the target sampling parameter is greater than or equal to the preset sampling parameter threshold (e.g., 0.8) and whether the number of unlabeled remote sensing images is greater than the number of target images. If the conditions are met, the unlabeled remote sensing images of the target are considered to meet the information entropy condition, and object labeling operation should be performed.

[0214] S6. Manually inspect and label the target unlabeled images that meet the conditions, supplement their real ground feature category labels, and generate new labeled remote sensing images.

[0215] This application's embodiments employ a weighted summation strategy of uncertainty parameters and clustering distance parameters to determine target sampling parameters, achieving accurate screening of unlabeled remote sensing images that meet the information entropy condition. This optimizes the model training dataset, improving classification accuracy and model robustness.

[0216] As an optional approach, training the initial classification model based on the newly labeled remote sensing images includes: adding the newly labeled remote sensing images to the initial training dataset to obtain a target training dataset; training the initial classification model using the target training dataset; determining the initial classification model as the target classification model if the initial classification model meets a preset model convergence condition; and training the initial classification model again based on the unlabeled remote sensing images (without the image labeling operation), the pseudo-labeled remote sensing images, and the initial training dataset if the initial classification model does not meet the model convergence condition.

[0217] As an optional approach, the above method further includes at least one of the following: verifying the initial classification model using a remote sensing image test set to obtain an average intersection-over-union (IoU) parameter; determining that the initial classification model satisfies the model convergence condition if the variation of the average IoU parameter in N consecutive training rounds is less than a preset variation parameter, where N is a positive integer; determining that the initial classification model satisfies the model convergence condition if the number of times the newly labeled remote sensing images are generated is equal to a preset number of generation times; and determining that the initial classification model satisfies the model convergence condition if the object labeling operation is performed on each of the unlabeled remote sensing images.

[0218] Optionally, in the embodiments of this application, the newly labeled remote sensing image refers to an image obtained through manual labeling during the sample update training phase, which carries a precise identification of the land cover category and is used to further train and optimize the initial classification model.

[0219] Optionally, in this embodiment, the target training dataset is an expanded dataset formed by incorporating newly labeled remote sensing images into the initial training dataset, which is used for subsequent training of the model.

[0220] Optionally, in the embodiments of this application, the model convergence condition refers to the indicator that the model training has reached a stable state. It is usually defined based on standards such as training loss, validation performance, or number of iterations to ensure that the model performance no longer significantly improves or fluctuates.

[0221] Optionally, in the embodiments of this application, the mean intersection-union ratio (mIoU) is a commonly used metric for evaluating the classification performance of semantic segmentation models. It measures the ratio of the intersection and union of the model's prediction results with the true labels, reflecting the accuracy and completeness of the model's classification.

[0222] Optionally, in this embodiment, the variable parameter is a threshold set to evaluate the magnitude of mIoU change, used to determine whether the model has reached convergence.

[0223] It should be noted that the model convergence conditions may include, but are not limited to, conditions such as the mIoU change being less than the preset variation parameter, the number of times new labeled remote sensing images are generated reaching a preset upper limit, and all unlabeled remote sensing images having undergone one round of manual labeling. The specific conditions should be selected based on a comprehensive consideration of factors such as the model training status, dataset size, and labeling cost, in order to achieve a balance between efficient training and ideal classification performance. This application does not impose any restrictions on these conditions.

[0224] For example, when training an initial classification model based on newly labeled remote sensing images, the newly labeled images are first added to the initial training dataset to form the target training dataset, and then the model is trained using the target training dataset. During training, the model performance is continuously monitored. If, after N consecutive training rounds, the change in mIoU is less than the set variation parameter, the number of times the newly labeled remote sensing images are generated is equal to the preset number of generation times, and any of the above conditions for object labeling operations are performed on each of the above unlabeled remote sensing images, it indicates that the model performance has stabilized and the model has reached the convergence condition. If the model reaches the convergence condition, the initial classification model in the current state is determined as the target classification model, and the model training process ends. Conversely, if the model has not yet met the convergence condition, the model is trained again using unlabeled remote sensing images that have not undergone image labeling operations, the generated pseudo-labeled remote sensing images, and the initial training dataset, until the model converges or reaches other termination conditions.

[0225] In one exemplary embodiment, crop classification in precision agriculture applications is taken as an example:

[0226] S1 collected a small number of labeled crop images as the initial training dataset, as well as a large number of unlabeled crop remote sensing images.

[0227] S2 performs object labeling operations on some unlabeled images to generate newly labeled remote sensing images.

[0228] S3 adds the newly labeled remote sensing images to the initial training dataset to form the target training dataset, which is used for the next round of model training.

[0229] S4, train the initial classification model using the target training dataset, and monitor the changes in mIoU during the process.

[0230] S5. If the change in mIoU in 5 consecutive training rounds is less than 0.01, the number of times the above-mentioned newly labeled remote sensing images are generated is equal to the preset number of generation times, and any of the above-mentioned object labeling operations are performed on each of the above-mentioned unlabeled remote sensing images, then the model is considered to have reached the convergence condition and the training process ends.

[0231] S6. After the model reaches the convergence condition, the initial classification model in the current state is determined as the final target classification model for crop classification tasks.

[0232] Through the embodiments of this application, a strategy of dynamically incorporating newly labeled remote sensing images and multi-round training is adopted to achieve continuous optimization and performance improvement of the initial classification model. Specifically, by continuously adding manually labeled images to the training dataset and repeating the training process before the model converges, classification accuracy is improved while training efficiency and labeling costs are considered, thereby maximizing model performance with limited resources.

[0233] As an optional approach, the above method further includes at least one of the following: during the training of the initial classification model using the initial training dataset, freezing the shallow model parameters of the initial classification model and updating the deep model parameters of the initial classification model; during the training of the initial classification model using the intermediate training dataset, freezing the shallow model parameters of the initial classification model and updating the deep model parameters of the initial classification model; during the training of the initial classification model using the target training dataset, updating the shallow model parameters and deep model parameters of the initial classification model to obtain the target classification model.

[0234] Optionally, in the embodiments of this application, shallow model parameters refer to the part of the classification model responsible for basic feature extraction, including the front-end convolutional layer or other preprocessing modules of the model. These parameters have learned relatively general feature representations during the model pre-training stage. Therefore, freezing them during model initialization or early training stages can prevent premature modifications from damaging their original feature extraction capabilities.

[0235] Optionally, in the embodiments of this application, deep model parameters refer to the parts of the model used for high-level feature combination and classification decision-making, such as deep convolutional layers, fully connected layers, classification heads, etc. These parameters are open for updating in the early stage of training so as to learn more discriminative and adaptive features from specific task data, laying the foundation for improving the performance of the model.

[0236] Optionally, in the embodiments of this application, the target classification model refers to the final classification model that has achieved the expected performance or convergence condition after a series of training, fine-tuning and parameter updates, and can effectively cope with the classification task of multi-source remote sensing images.

[0237] It should be noted that the model parameter update strategy can vary depending on the stage of the training process. In the early stages of model training, to preserve the basic feature extraction capabilities acquired during pre-training, shallow model parameters can be frozen, with only deep model parameters updated. However, in the later stages of training, especially after the introduction of a large amount of manually labeled data, shallow model parameters can be unfrozen, allowing the entire model to learn more detailed task features from new, larger datasets, further improving model performance. Parameter update methods are not limited to gradient descent; they can also include advanced optimization algorithms such as momentum optimization, Adam, and RMSprop. This application does not impose any limitations on these methods.

[0238] In one exemplary embodiment, taking the application scenario of urban land use classification as an example:

[0239] S1, Model Initialization: Use a small number of urban remote sensing images as the initial training dataset to perform preliminary training on the model. At this time, keep the shallow model parameters frozen and only train the deep model parameters to adapt to the land cover classification task.

[0240] S2, Sample Expansion Training Phase: After the initial model training, pseudo-labels are generated for the unlabeled remote sensing images, forming an intermediate training dataset. When training the model using the intermediate training dataset, the shallow model parameters are kept frozen, and only the deep model parameters are updated to further improve the model's classification ability on new data.

[0241] S3, Full Parameter Fine-Tuning: After accumulating a sufficient number of newly labeled remote sensing images and high-confidence pseudo-labeled remote sensing images, the training phase begins with the target training dataset. At this point, both shallow and deep parameters of the model are open for updating. Through training on a large-scale dataset, fine-tuning of the overall model performance is achieved, ultimately resulting in the target classification model.

[0242] Through the embodiments of this application, a phased parameter update strategy is adopted to achieve gradual optimization and performance improvement of the initial classification model. Specifically, by freezing shallow parameters during model initialization and sample expansion training, the basic feature extraction capability of the pre-trained model is prevented from being interfered with. However, all parameters are updated during the full parameter fine-tuning stage, allowing the model to learn more deeply on more complex task features. This achieves the dual goals of high accuracy and low annotation cost in multi-source remote sensing image classification.

[0243] In an exemplary embodiment, the above model training scheme can be applied to the field of remote sensing image processing technology, realizing intelligent annotation of multi-source remote sensing images based on dual-layer optimization. Through iterative training, the model parameters are finely adjusted in stages to ensure that the model can continuously learn and evolve from existing data. A dual-layer collaborative mechanism of dynamic dataset optimization and model parameter fine-tuning is constructed to achieve efficient collaborative annotation of multi-source heterogeneous remote sensing data (SAR / LiDAR, HSI), reducing annotation costs and improving the classification accuracy of cross-modal remote sensing data. Its training process includes, but is not limited to:

[0244] (1) Outer layer optimization: Dynamic optimization of the dataset may include, but is not limited to, selecting pseudo-labeled samples with high confidence for annotation to expand the training dataset. The annotation here includes, but is not limited to, manual annotation, automatic annotation, etc. This application does not make specific restrictions on the annotation method;

[0245] (2) Inner layer optimization: model parameter fine-tuning. First, the model parameters are initialized, and then the model is trained in stages using continuously optimized training datasets to iteratively optimize the model parameters.

[0246] Through a two-layer collaborative mechanism of dynamic dataset optimization and model parameter fine-tuning, efficient collaborative annotation of multi-source heterogeneous remote sensing data (SAR / LiDAR and HSI) is achieved, thereby reducing annotation costs and improving the classification accuracy of cross-modal remote sensing data.

[0247] Specifically, the remote sensing image segmentation model (the initial segmentation model mentioned above) achieves efficient feature extraction and fusion of multi-source remote sensing data by combining spectral and spatial attention mechanisms with a cross-source relation encoding module. The Swin-Transformer, pre-trained on the ImageNet-21K dataset, is used as the backbone network, which combines the window self-attention module W-MSA and the shifted window self-attention module SW-MSA to balance local details and global structure. Then, the HSI and SAR / LiDAR dual-branch models are initialized separately. For the HSI branch, a spectral attention module (Squeeze-and-Excitation (SE)) is embedded, which dynamically weights the channel dimensions of the feature map to enhance the response of important spectral channels and suppress redundant information. For the SAR / LiDAR branch, a spatial attention module (SAM) works in conjunction with the shifted window mechanism to calculate spatial weights within the window of the feature map, highlighting the edges and structural features of ground objects.

[0248] Furthermore, the features extracted from the two branches are then fed into an improved deformable cross-source relation coding (CRE) module. A deformable convolution is added before the CRE module to automatically align the features by learning the offsets of spectral features (HSI) and spatial features (SAR / LiDAR), thus adapting to the geometric differences of multi-source data.

[0249] An outer product operation is performed on the aligned features to capture high-order feature interactions, ultimately outputting the fused cross-source features (the initial fused features mentioned above). The fused cross-source features are then upsampled and refined by the decoder. The decoder employs a progressive upsampling architecture, using a skip connection structure to achieve multi-scale feature fusion while preserving high-level semantic information as well as shallow detail information, resulting in the target fused features. Finally, a 1×1 convolutional classifier maps the features output by the decoder to pixel-level classification probabilities and calculates the probability distribution of land cover categories using a Softmax activation function, thereby determining the final land cover classification result and completing the classification task. The parallel design of spectral and spatial branches allows the model to process data characteristics of different modalities separately, while the improved deformable CRE module and decoder achieve deep fusion and efficient utilization of multi-source features.

[0250] Specifically, in the embodiments of this application, semantic segmentation of multi-source remote sensing images can be achieved through a multi-stage hybrid learning strategy:

[0251] First, the ImageNet-21K pre-trained Swin-Transformer is used as the backbone network, and its shallow layers (Stage 1-2) are frozen to retain the general feature extraction capability. At the same time, the dual-branch encoder structure is initialized to process HSI and SAR / LiDAR data respectively.

[0252] Subsequently, the model performance was optimized through an iterative training process:

[0253] (1) During the sample expansion training phase, pseudo-labels are generated for unlabeled data (the above unlabeled remote sensing images) through model self-training, and high-confidence samples are selected by adaptive thresholding and added to the training set (the above initial training dataset).

[0254] (2) During the sample update training phase, samples with high confidence are sampled based on uncertainty and representativeness criteria for manual annotation, and the annotation dataset is dynamically updated.

[0255] (3) In each training cycle, the model fine-tuning stage is based on the enhanced dataset, unfreezing all parameters and optimizing the dual-branch network and classification head parameters.

[0256] In summary, the model undergoes end-to-end fine-tuning using mixed supervision signals, focusing on parameter optimization of deep networks and the classification head. This process leverages transfer learning to transfer knowledge and determine the initial model parameters. It maximizes annotation efficiency during the sample expansion training phase and fully exploits the value of unlabeled data during the sample update training phase, forming a closed-loop optimization system. This system continuously improves model performance with limited annotation costs until the model converges, ultimately outputting an optimized target classification model.

[0257] First, the processing flow of the initial segmentation model includes, but is not limited to:

[0258] 1. Pre-training loading:

[0259] The Swin-Transformer, pre-trained on ImageNet-21K, is used as the encoder basis for the dual-branch model. The weights of the shallow layers (Stage 1-2) are frozen to retain their general feature extraction capabilities and avoid overfitting in the initial training stage. Only the deep layers (Stage 3-4) and the classification head are fine-tuned to adapt to the remote sensing task.

[0260] 2. Dual-branch primary feature extraction:

[0261] (1) HSI branch: For an HSI image ,in Indicates spatial resolution. Represents the channel dimension, input to Extracting primary features from convolutional layers (stride 1, padding 1) C represents the number of channels.

[0262] (2) SAR / LiDAR branch: The structure is symmetrical to the HSI branch, and the input is a SAR / LiDAR image. Primary features are extracted through convolution. .

[0263] 3. Multi-scale deep feature extraction:

[0264] For the two branches, cascaded Swin-Transformer encoders (window size) , 4 SwinTransformer Block (STB)).

[0265] (1) For the HSI branch, each STB is followed by a spectral attention module, Squeeze-and-Excitation (SE). The SE module generates channel descriptors through global average pooling. Two fully connected layers (dimensionality reduction ratio) Generate channel weights and output features. ,in , The output feature after each STB+SE block is: .

[0266] (2) For the SAR / LiDAR branch, a Spatial Attention Module (SAM) is inserted between STB blocks to generate a spatial attention weight matrix. Then each output feature is .

[0267] 4. Deformable cross-source feature fusion:

[0268] In the CRE module, feature interaction is achieved through weighted bilinear operations, but the geometric deformation caused by differences in imaging mechanisms between multi-source data (HSI and SAR / LiDAR) is not explicitly considered. To address this, this application introduces deformable convolution as a preprocessing step to dynamically learn the local offsets of spectral and spatial features, achieving feature alignment and thus improving the accuracy of higher-order interactions.

[0269] Specifically, before the weighted bilinear interaction in the CRE module, a deformable convolutional layer is applied to the input single-source features. This layer first generates the spatial offset of each sampling point through an additional offset prediction network (implemented by a regular convolutional layer). :

[0270]

[0271] in Two layers Convolutions are used to generate a 2D offset field. Then, deformable convolutions are applied to the SAR / LiDAR features to geometrically align them to the HSI feature space, resulting in aligned features.

[0272]

[0273] Then, the input is processed by the CRE module, where a weighted outer product operation is used to perform feature interaction and fusion, resulting in the following fused features:

[0274]

[0275] Dynamic weights Activated by Sigmoid Convolution generation, i.e. The obtained weights Used to measure the significance of HSI features, This emphasizes SAR / LiDAR characteristics. For the outer product operation, the output dimension is... Therefore, it is still necessary to go through Convolution compresses the interaction matrix, ultimately yielding the fused cross-source features. .

[0276] 5. Feature downsampling and classification:

[0277] After feature extraction, the decoder uses a skip connection mechanism to achieve hierarchical fusion of multi-scale features, taking into account both local details and global semantic information, such as... Figure 7 As shown. First, the shallow features (Stage-1 high-resolution feature map) are... )pass After channel alignment by convolution, the resulting feature is added to the fused feature output from the improved CRE module to achieve low-level feature fusion. Then through transposed convolution ( Upsampled kernel (step size 2), concatenated with the Stage-2 features of the two branches, and processed through two residual blocks (containing Convolution + BN + ReLU) refines features, achieving mid-level feature refinement. Finally, after secondary upsampling and alignment with the original resolution, the high-level fusion features are output. .

[0278] It should be noted that, Indicates the first shallow spectral characteristics, Indicates the spectral characteristics of the second shallow layer. Indicates deep spectral characteristics, This indicates the characteristics of the first shallow radar layer. This indicates the characteristics of the second shallow radar layer. This indicates the characteristics of deep radar.

[0279] Finally, output the decoder. conduct Convolution dimensionality reduction to number of categories Preserve full spatial resolution ( The probability distribution of land cover categories is calculated using the Softmax activation function to determine the final land cover classification result.

[0280]

[0281] Optionally, the specific training process and parameter configuration are as follows:

[0282] S1, Initial annotation data loading:

[0283] Using a small number of manually labeled datasets To train the initial model, using the Houston 2018 dataset as an example, randomly select 10 samples from each class, resulting in 20 classes and an initial set of 200 samples. Freeze the shallow layer parameters (Stages 1-2) and fine-tune only the deeper layers (Stages 3-4 and the classifier head), setting the learning rate to [value missing]. Iterate for 5 epochs to stabilize feature extraction capability, and set the following during training: .

[0284] S2, Sample Expansion Training Phase:

[0285] In the sample expansion training phase, a self-training strategy is employed, combined with adaptive threshold pseudo-label generation, to improve model performance using a small amount of labeled data and a large amount of unlabeled data. The training process includes:

[0286] 1. Pseudo-tag generation:

[0287] The quality of pseudo-labels directly affects the training effect during the sample expansion training phase. A fixed threshold may cause the model to fail to fully utilize unlabeled data or introduce noise. This application's embodiments employ an adaptive threshold strategy to dynamically filter high-confidence pseudo-labels, addressing the issues of class imbalance and differences in prediction confidence distribution in remote sensing images.

[0288] (1) Model prediction: using manually labeled datasets The initial model trained on the unlabeled dataset To make predictions, extract the classification head output, which is the probability distribution after the last Softmax layer. , This represents the number of categories.

[0289] (2) Dynamic threshold calculation: For each category Calculate the probability of all pixels predicted to be of this class. And calculate the mean. with standard deviation , ,in For prediction as a category The total number of pixels. Then, the adaptive threshold is calculated. , To adjust the coefficients, the stringency of the threshold is reduced in each iteration: ,initial , This is an adaptive threshold decay coefficient. The threshold is iteratively changed to ensure accuracy and utilize more data.

[0290] (3) Pseudo-label filtering: retain those that meet the criteria The pixels are used as high-confidence pseudo-labels, and pseudo-label masks are generated to obtain the pseudo-label dataset. Furthermore, if no pixels in a certain category meet the threshold, the top 5% of pixels with the highest probability in that category are retained as high-confidence pseudo-labels to avoid underutilization of data in smaller categories.

[0291] 2. Mixed training:

[0292] During the sample expansion training phase, a pixel-level hybrid training strategy is employed. Labeled data is included in each batch. and pseudo-label data by Hybrid training, calculating cross-entropy loss on labeled data Calculate the cross-entropy loss for the high-confidence pseudo-label data (only for the high-confidence pseudo-label pixels in the image). The total loss is obtained by weighted summation. The weighting coefficient of pseudo-labels Set it to 0.5.

[0293] 3. Model fine-tuning:

[0294] Freeze the shallow layers (Stages 1-2), fine-tune only the deeper layers (Stages 3-4 and the classification head), and regenerate pseudo-labels every 3 epochs. Use the AdamW optimizer (learning rate...). ).

[0295] S3, Sample Update Training Phase:

[0296] After training for 3 epochs in the sample expansion training phase, the sample update training phase begins. By combining uncertainty and representativeness metrics, the most valuable samples are selected for manual annotation to optimize annotation efficiency and model performance.

[0297] For the uncertainty measure, for the input image x, the size of x is represented as (H W C), where H W represents the width and height, C represents the number of image channels, and the prediction entropy of each pixel in the unlabeled data is calculated. ,in Indicates position The predicted probability that a pixel belongs to class n, where For row index, For column indexes, This is used as a category index. This leads to the image-level uncertainty score. For representativeness metrics, extract the features from the last layer of the decoder. Image-level features are obtained through global average pooling. And perform K-means clustering ( ), calculate the distance from the sample to the nearest cluster center Finally, the joint sampling score is obtained by weighting. ,set up Choose the one with the highest score. Manual annotation was performed, among which Indicates the minimum guaranteed quantity. This indicates the percentage of the sampled data.

[0298] S4, Model Iterative Optimization:

[0299] Add the new labeled samples to the labeled dataset Simultaneously, the unlabeled dataset is updated, and a new pseudo-labeled dataset is generated. Then, all model parameters are unfrozen, and end-to-end fine-tuning is performed using labeled data, with the learning rate set to [value missing]. Repeat steps S1 to S3. If the mIoU on the test set does not improve for three consecutive rounds, or the number of training and labeling rounds for updating samples reaches the preset limit (e.g., 10 rounds), or the unlabeled set... If the value is empty, the iteration stops, and the final trained multi-source remote sensing image semantic segmentation model (the target classification model mentioned above) is obtained.

[0300] It should also be noted that the two-layer optimization training process in this application embodiment is an iterative process in which an outer loop (sample update training loop) drives an inner loop (sample expansion training loop). An outer loop consists of "sample expansion training - sample sampling - manual annotation - full parameter fine-tuning," while semi-supervised training is an enhancement step at the beginning of each sample update training round. As an inner loop, it aims to improve performance using the current model and unlabeled data, providing a stronger benchmark model for subsequent active sampling. The specific process and hyperparameter settings are as follows:

[0301] 1. Initialization phase: Input the initial manually labeled set Unlabeled dataset And the pre-trained model. Perform the initial labeled data loading as described above. In this stage, the AdamW optimizer is used to freeze the shallow layer parameters (Stage 1-2) and only fine-tune the deep layers (Stage 3-4 and the classification head), setting the learning rate to [value missing]. Number of training rounds Output the initial model .

[0302] 2. Outer loop (sample update training loop):

[0303] Set a circular index: ,in Update the training rounds to the preset maximum number of samples. In each round, perform the following steps:

[0304] (1) Inner loop (semi-supervised training): The inner loop is triggered at the beginning of each outer loop iteration, or after the label set is updated. Input the current model. The current annotation set (i.e., the initial training dataset mentioned above), unlabeled set (i.e., the unlabeled remote sensing image mentioned above). Then, proceed to "Step 3: Semi-supervised Co-training". The AdamW optimizer is used, and the learning rate is set... Number of cycles Pseudo-label weight Adaptive threshold decay coefficient (Every (Fallen once after each round). Output of the semi-supervised augmented model. .

[0305] (2) Sample Sampling: Sample sampling is performed after the inner loop (semi-supervised training) is completed. Input Model Unlabeled set Then perform sample correction operations. For example, uncertainty weights. Representative weight Number of samples ,in =50 (minimum sample size) =0.05 (sampling rate). Finally, the selected sample set to be labeled is output. .

[0306] (3) Manual annotation and dataset update: The selected training dataset to be annotated Manual annotation was performed to obtain the annotation set. (i.e., the newly labeled remote sensing images mentioned above), and then update the training dataset. Update the unlabeled set .

[0307] (4) Fine-tuning of all model parameters: After the annotation set is updated, the input model is Unfreeze all parameters and use the current annotation set. Supervised training was performed. The AdamW optimizer was used, and the learning rate was set accordingly. Training rounds Output the final model of the current sample to update the training cycle. .

[0308] 3. Loop termination condition:

[0309] The outer loop terminates and outputs the final optimized model when any of the following conditions are met. (1) On the retained validation set, the mIoU metric did not improve for three consecutive outer loops (< 0.5%). (2) The training rounds were updated after reaching the preset maximum number of samples. (3) Unlabeled set Empty.

[0310] This application implements a two-layer optimization collaborative mechanism: First, it selects the most valuable samples for manual annotation by combining uncertainty and representativeness in the training update strategy, and expands the labeled dataset using pseudo-label data. Second, it initializes model parameters through transfer learning, continuously expanding the training data based on pseudo-label data in each training round, thereby updating the training dataset and achieving iterative optimization of model parameters. Simultaneously, at the model architecture level, a pre-processing deformable convolutional layer dynamically learns the local offsets of spectral and spatial features, automatically aligning the geometric offsets of multi-source features, thus improving the accuracy of higher-order interactions. The introduction of deformable convolution enhances the geometric robustness of cross-source feature interactions, especially when processing multi-source data with significant differences in viewpoint or resolution, more effectively preserving discriminative structural information.

[0311] Furthermore, using an improved dual-branch multi-source remote sensing image segmentation model (i.e., the initial classification model), the decoder module achieves hierarchical fusion of multi-scale features through a multi-level skip connection mechanism. First, shallow high-resolution features are fused with cross-modal fusion features for low-level detail fusion. Then, upsampling and concatenation with mid-level features are performed, followed by residual refinement. Finally, resolution is restored to output fused features containing rich spatial-semantic information. This hierarchical feature fusion approach effectively solves the problems of detail loss and semantic ambiguity in traditional methods, achieving coordinated optimization of local details and global semantics.

[0312] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0313] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0314] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0315] Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0316] According to another aspect of the embodiments of this application, a model training apparatus for implementing a model training method is also provided. This model training apparatus can be used to implement the model training method provided in the embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0317] Figure 8 This is a structural block diagram of an optional model training device according to an embodiment of this application, such as... Figure 8 As shown, the training apparatus for this model includes:

[0318] The generation module 802 is used to input unlabeled remote sensing images into the initial classification model to obtain pseudo-labeled remote sensing images;

[0319] Training module 804 is used to train the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image, and the initial training dataset to obtain a target classification model. The initial classification model's training includes a first training phase and a second training phase. In the first training phase, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training dataset. In the second training phase, the initial classification model is trained using the unlabeled remote sensing image that satisfies the information entropy condition and the initial training dataset.

[0320] As an optional approach, the device is also used to: before inputting unlabeled remote sensing images into the initial classification model to obtain pseudo-labeled remote sensing images, obtain an initial training dataset, wherein the initial training dataset includes at least one image pair, the image pair including a hyperspectral image and a lidar image representing the same region, and the images in the initial training dataset all carry object classification labels; and train the initial classification model using the initial training dataset.

[0321] As an optional approach, the device is used to train an initial classification model using an initial training dataset in the following manner: performing spectral feature extraction on the target hyperspectral image to obtain shallow spectral features and deep spectral features; performing radar feature extraction on the target radar image to obtain shallow radar features and deep radar features, wherein the target hyperspectral image and the target radar image belong to the same image pair; performing a first feature fusion operation on the deep spectral features and deep radar features to obtain initial fused features; performing a second feature fusion operation on the shallow spectral features, shallow radar features, and initial fused features to obtain target fused features; and updating the model parameters of the initial classification model based on the target fused features.

[0322] As an optional approach, the device performs a first feature fusion operation on deep spectral features and deep radar features in the following manner to obtain initial fused features: performing an offset prediction operation on the deep spectral features and deep radar features to obtain spatial offset parameters; performing a feature offset operation on the deep radar features based on the spatial offset parameters to adjust the deep radar features to the feature space corresponding to the deep spectral features, thereby obtaining radar aligned features; performing a weighted outer product operation on the radar aligned features and deep spectral features to obtain the first fused features; and adjusting the feature dimensions of the first fused features using a convolutional compression interaction matrix to obtain the initial fused features.

[0323] As an optional approach, the device is used to perform a second feature fusion operation on shallow spectral features, shallow radar features, and initial fusion features to obtain target fusion features by: determining a first shallow spectral feature among the shallow spectral features and a first shallow radar feature among the shallow radar features, wherein the first shallow spectral feature is the first spectral feature generated among the shallow spectral features, and the first shallow radar feature is the first spectral feature generated among the shallow radar features; performing an accumulation operation on the first shallow spectral feature, the first shallow radar feature, and the initial fusion features to obtain a bottom-level fusion feature; performing an upsampling operation on the bottom-level fusion features to obtain an upsampled fusion feature; and determining the target fusion feature based on the upsampled fusion feature, the shallow spectral feature, and the shallow radar feature.

[0324] As an optional approach, the device is used to determine target fusion features based on upsampled fusion features, shallow spectral features, and shallow radar features in the following manner: determining a second shallow spectral feature among the shallow spectral features, and determining a second shallow radar feature among the shallow radar features, wherein the second shallow spectral feature is generated later than the first shallow spectral feature, and the second shallow radar feature is generated later than the first shallow radar feature; splicing the upsampled fusion features, the second shallow spectral feature, and the second shallow radar feature to obtain spliced ​​fusion features; performing feature refinement operations on the spliced ​​fusion features to obtain intermediate fusion features; and performing upsampling operations on the intermediate fusion features to obtain target fusion features.

[0325] As an optional approach, the device is used to input unlabeled remote sensing images into an initial classification model to obtain pseudo-labeled remote sensing images in the following manner: The unlabeled remote sensing images are input into the initial classification model to determine the probability value of each pixel in the unlabeled remote sensing images belonging to each object category; a preset adjustment coefficient is obtained, and the mean parameter and standard deviation parameter corresponding to each object category are determined, wherein the adjustment coefficient decreases as the number of training rounds increases; based on the adjustment coefficient, the mean parameter and standard deviation parameter of the probability value corresponding to each object category, pseudo-labels are set for each pixel in the unlabeled remote sensing images to obtain pseudo-labeled remote sensing images.

[0326] As an optional approach, the device is used to set pseudo-labels for each pixel in an unlabeled remote sensing image based on an adjustment coefficient, the mean probability value parameter corresponding to each object category, and the standard deviation probability value parameter, to obtain a pseudo-labeled remote sensing image: determining the product of the adjustment coefficient and the standard deviation probability value parameter corresponding to the target object category; determining the difference between the product and the mean probability value parameter corresponding to the target object category as an adaptive threshold, wherein the target object category is any object category; setting a pseudo-label for the target pixel if the target pixel meets the target pixel condition, wherein the pseudo-label of the target pixel is used to indicate that the target pixel belongs to the target object category, and the target pixel is any pixel in the unlabeled remote sensing image; not setting a pseudo-label for the target pixel if the target pixel does not meet the target pixel condition.

[0327] As an optional solution, the device is also used to: determine that the target pixel satisfies the target pixel condition when the probability value of the target pixel belonging to the target object category is greater than or equal to an adaptive threshold; determine that the target pixel does not satisfy the target pixel condition when the probability value of the target pixel belonging to the target object category is less than the adaptive threshold; determine that the target pixel satisfies the target pixel condition when the probability value of any pixel in the unlabeled remote sensing image belonging to the target object category is less than the adaptive threshold, wherein the probability value of the target pixel belonging to the target object category is within a preset range, the preset range is related to the probability value of the maximum pixel, and the probability value of the maximum pixel belonging to the target object category is greater than the probability value of any pixel other than the maximum pixel belonging to the target object category.

[0328] As an optional approach, the device is used to train an initial classification model based on unlabeled remote sensing images, pseudo-labeled remote sensing images, and an initial training dataset to obtain a target classification model: In the first training phase, the initial training dataset is updated based on the pseudo-labeled remote sensing images to obtain an intermediate training dataset, and the initial classification model is trained using the intermediate training dataset; In the second training phase, object labeling operations are performed on unlabeled remote sensing images that meet the information entropy condition to obtain newly labeled remote sensing images, and the initial classification model is trained based on the newly labeled remote sensing images.

[0329] As an optional approach, the device is used to update the initial training dataset based on pseudo-labeled remote sensing images to obtain an intermediate training dataset, and then use the intermediate training dataset to train the initial classification model: Pseudo-labeled remote sensing images are added to the initial training dataset to obtain an intermediate training dataset, wherein the number of pseudo-labeled remote sensing images in the intermediate training dataset is greater than the number of images in the initial training dataset; the initial classification model is trained using the intermediate training dataset to determine a first cross-entropy loss value and a second cross-entropy loss value, wherein the first cross-entropy loss value is related to images in the intermediate training dataset that belong to the initial training dataset, and the second cross-entropy loss value is related to pseudo-labeled remote sensing images; the first cross-entropy loss value and the second cross-entropy loss value are merged to obtain a target loss value, and the model parameters of the initial classification model are updated based on the target loss value.

[0330] As an optional approach, the device is used to perform object labeling on an unlabeled remote sensing image that satisfies the information entropy condition to obtain a new labeled remote sensing image by: determining the target probability value of each pixel in the target unlabeled remote sensing image belonging to each object category, wherein the target unlabeled remote sensing image is any unlabeled remote sensing image; determining an uncertainty parameter and a clustering distance parameter based on the target probability value, wherein the uncertainty parameter is used to indicate the confidence level of the object classification of the target unlabeled remote sensing image, and the clustering distance parameter is used to indicate the feature coverage of the target unlabeled remote sensing image; performing a weighted summation operation on the uncertainty parameter and the distance parameter to obtain the target sampling parameter; and performing object labeling on the target unlabeled remote sensing image, provided that the target unlabeled remote sensing image satisfies the information entropy condition, to obtain a new labeled remote sensing image.

[0331] As an optional solution, the device is also used to determine, at least one of the following: when the number of unlabeled remote sensing images is greater than the number of target images and the target sampling parameters are greater than or equal to a preset sampling parameter threshold, that the target unlabeled remote sensing images satisfy the information entropy condition; when the number of unlabeled remote sensing images is less than or equal to the number of target images, that the target unlabeled remote sensing images satisfy the information entropy condition.

[0332] As an optional approach, the device is used to train an initial classification model based on newly labeled remote sensing images in the following manner: adding the newly labeled remote sensing images to the initial training dataset to obtain the target training dataset; training the initial classification model using the target training dataset; determining the initial classification model as the target classification model when the initial classification model meets the preset model convergence condition; and training the initial classification model again based on unlabeled remote sensing images, pseudo-labeled remote sensing images, and the initial training dataset when the initial classification model does not meet the model convergence condition.

[0333] As an optional approach, the device is also used for at least one of the following: verifying the initial classification model using a remote sensing image test set to obtain the average intersection-over-union (IoU) parameter; determining that the initial classification model satisfies the model convergence condition if the variation of the average IoU parameter in N consecutive training rounds is less than a preset variation parameter, where N is a positive integer; determining that the initial classification model satisfies the model convergence condition if the number of times new labeled remote sensing images are generated is equal to a preset number of generation times; and determining that the initial classification model satisfies the model convergence condition if object labeling operations are performed on each unlabeled remote sensing image.

[0334] As an optional approach, the device is also used for at least one of the following: during the training of an initial classification model using an initial training dataset, freezing the shallow model parameters of the initial classification model and updating the deep model parameters of the initial classification model; during the training of an initial classification model using an intermediate training dataset, freezing the shallow model parameters of the initial classification model and updating the deep model parameters of the initial classification model; during the training of an initial classification model using a target training dataset, updating the shallow and deep model parameters of the initial classification model to obtain the target classification model.

[0335] Regarding the apparatus in the above embodiments, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit. The specific manner in which each module performs its operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

[0336] According to another aspect of the embodiments of this application, an electronic device is provided.

[0337] The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps in any of the above method embodiments via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor. Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0338] According to one aspect of this application, a computer program product is also provided, which includes a computer program.

[0339] The computer program product includes a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit 901, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0340] Figure 9 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 9 As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which performs various appropriate actions and processes based on programs stored in ROM 902 or loaded into RAM 903 from storage section 908. Random Access Memory 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / Output (I / O) interface 905 is also connected to bus 904.

[0341] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card, such as a local area network card or modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.

[0342] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0343] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer programs / instructions. For example, embodiments of this application include a computer program / instruction comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication portion, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions defined in the system of this application. In such embodiments, the computer program / instruction can be downloaded and installed from a network via a communication portion, and / or installed from a removable medium. When the computer program / instruction is executed by a central processing unit, it performs the training method of the aforementioned model.

[0344] According to one aspect of this application, a computer-readable storage medium is also provided.

[0345] The processor of the aforementioned electronic device can read the computer instructions from a computer-readable storage medium, and execute the computer instructions to cause the electronic device to perform the training method of the model provided in various optional implementations of the training aspect of the aforementioned model.

[0346] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store methods for performing the embodiments of this application.

[0347] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0348] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0349] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more electronic devices to execute all or part of the steps of the methods described in the various embodiments of this application.

[0350] In the several embodiments provided in this application, it should be understood that the disclosed application can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0351] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0352] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0353] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for training a model, characterized in that, The method comprises the following steps: inputting an unlabeled remote sensing image into an initial classification model to obtain a pseudo-labeled remote sensing image; training the initial classification model based on the unlabeled remote sensing image, the pseudo-labeled remote sensing image and an initial training data set to obtain a target classification model, wherein one round of training of the initial classification model comprises a first training stage and a second training stage, in the first training stage, the initial classification model is trained using the pseudo-labeled remote sensing image and the initial training data set; in the second training stage, the initial classification model is trained using the unlabeled remote sensing image satisfying an information entropy condition and the initial training data set.

2. The method of claim 1, wherein, Before the step of inputting the unlabeled remote sensing image into the initial classification model to obtain the pseudo-labeled remote sensing image, the method further comprises: obtaining the initial training data set, wherein the initial training data set comprises at least one image pair, the image pair comprises a hyperspectral image and a radar image representing the same region, and the images in the initial training data set all carry object classification labels; training the initial classification model using the initial training data set.

3. The method of claim 2, wherein, The step of training the initial classification model using the initial training data set comprises: performing a spectral feature extraction operation on a target hyperspectral image to obtain shallow spectral features and deep spectral features, and performing a radar feature extraction operation on a target radar image to obtain shallow radar features and deep radar features, wherein the target hyperspectral image and the target radar image belong to the same image pair; performing a first feature fusion operation on the deep spectral features and the deep radar features to obtain initial fusion features; performing a second feature fusion operation on the shallow spectral features, the shallow radar features and the initial fusion features to obtain target fusion features; updating model parameters of the initial classification model according to the target fusion features.

4. The method of claim 3, wherein, The step of performing a first feature fusion operation on the deep spectral features and the deep radar features to obtain initial fusion features comprises: performing an offset prediction operation on the deep spectral features and the deep radar features to obtain a spatial offset parameter; performing a feature offset operation on the deep radar features based on the spatial offset parameter, so that the deep radar features are adjusted to a feature space corresponding to the deep spectral features to obtain radar alignment features; performing a weighted outer product operation on the radar alignment features and the deep spectral features to obtain first fusion features; adjusting the feature dimension of the first fusion features using a convolution compression interaction matrix to obtain the initial fusion features.

5. The method of claim 3, wherein, The step of performing a second feature fusion operation on the shallow spectral features, the shallow radar features and the initial fusion features to obtain target fusion features comprises: determining a first shallow spectral feature in the shallow spectral features and a first shallow radar feature in the shallow radar features, wherein the first shallow spectral feature is the first generated spectral feature in the shallow spectral features, and the first shallow radar feature is the first generated spectral feature in the shallow radar features; performing an accumulation operation on the first shallow spectrum feature, the first shallow radar feature and the initial fusion feature to obtain a bottom layer fusion feature; performing an up-sampling operation on the bottom layer fusion feature to obtain an up-sampled fusion feature; determining the target fusion feature based on the up-sampled fusion feature, the shallow spectrum feature and the shallow radar feature.

6. The method of claim 5, wherein, The determining the target fusion feature based on the up-sampled fusion feature, the shallow spectrum feature and the shallow radar feature comprises: determining a second shallow spectrum feature in the shallow spectrum feature and a second shallow radar feature in the shallow radar feature, wherein the second shallow spectrum feature is generated later than the first shallow spectrum feature, and the second shallow radar feature is generated later than the first shallow radar feature; splicing the up-sampled fusion feature, the second shallow spectrum feature and the second shallow radar feature to obtain a spliced fusion feature; performing a feature refinement operation on the spliced fusion feature to obtain a middle layer fusion feature; performing the up-sampling operation on the middle layer fusion feature to obtain the target fusion feature.

7. The method of claim 1, wherein, The inputting the unlabeled remote sensing image into the initial classification model to obtain a pseudo-label remote sensing image comprises: inputting the unlabeled remote sensing image into the initial classification model to determine a probability value of each pixel in the unlabeled remote sensing image belonging to each object category; obtaining a preset adjustment coefficient, and determining a mean parameter and a standard deviation parameter corresponding to each object category, wherein the adjustment coefficient decreases with an increase of a training round number; setting a pseudo-label for each pixel in the unlabeled remote sensing image based on the adjustment coefficient, the probability value mean parameter and the probability value standard deviation parameter corresponding to each object category to obtain the pseudo-label remote sensing image.

8. The method of claim 7, wherein, The setting a pseudo-label for each pixel in the unlabeled remote sensing image based on the adjustment coefficient, the probability value mean parameter and the probability value standard deviation parameter corresponding to each object category to obtain the pseudo-label remote sensing image comprises: determining a product of the adjustment coefficient and a probability value standard deviation parameter corresponding to a target object category, and determining a difference between the product and a probability value mean parameter corresponding to the target object category as an adaptive threshold value, wherein the target object category is any object category; in a case where a target pixel satisfies a target pixel condition, setting a pseudo-label of the target pixel, wherein the pseudo-label of the target pixel is used to indicate that the target pixel belongs to the target object category, and the target pixel is any pixel in the unlabeled remote sensing image; in a case where the target pixel does not satisfy the target pixel condition, not setting the pseudo-label of the target pixel.

9. The method of claim 8, wherein, The method further comprises at least one of the following: in a case where a probability value of the target pixel belonging to the target object category is greater than or equal to an adaptive threshold value, determining that the target pixel satisfies the target pixel condition; in a case where the probability value of the target pixel belonging to the target object category is less than the adaptive threshold value, determining that the target pixel does not satisfy the target pixel condition; In a case where a probability value of any pixel in the unlabeled remote sensing image belonging to the target object category is less than an adaptive threshold, it is determined that the target pixel satisfies the target pixel condition, wherein the probability value of the target pixel belonging to the target object category is in a preset interval, the preset interval is related to a probability value of a maximum pixel, and the probability value of the maximum pixel belonging to the target object category is greater than a probability value of any pixel other than the maximum pixel belonging to the target object category.

10. The method of claim 1, wherein, The training of the initial classification model based on the unlabeled remote sensing image, the pseudo-label remote sensing image and an initial training data set comprises: In the first training stage, an initial training data set is updated according to the pseudo-label remote sensing image to obtain an intermediate training data set, and the initial classification model is trained using the intermediate training data set; In the second training stage, an object labeling operation is performed on the unlabeled remote sensing image satisfying the information entropy condition to obtain a newly labeled remote sensing image, and the initial classification model is trained based on the newly labeled remote sensing image.

11. The method of claim 10, wherein, The updating of the initial training data set according to the pseudo-label remote sensing image to obtain an intermediate training data set and the training of the initial classification model using the intermediate training data set comprise: The pseudo-label remote sensing image is added to the initial training data set to obtain the intermediate training data set, wherein the number of the pseudo-label remote sensing images in the intermediate training data set is greater than the number of images in the initial training data set; The initial classification model is trained using the intermediate training data set.

12. The method of claim 10, wherein, The object labeling operation performed on the unlabeled remote sensing image satisfying the information entropy condition to obtain a newly labeled remote sensing image comprises: Target probability values of each pixel in a target unlabeled remote sensing image belonging to each object category are determined, wherein the target unlabeled remote sensing image is any of the unlabeled remote sensing images; Uncertainty parameters and clustering distance parameters are determined according to the target probability values, wherein the uncertainty parameters are used to indicate the confidence of object classification of the target unlabeled remote sensing image, and the clustering distance parameters are used to indicate the feature coverage degree of the target unlabeled remote sensing image; A weighted sum operation is performed on the uncertainty parameters and the distance parameters to obtain a target sampling parameter; In a case where the target unlabeled remote sensing image satisfies the information entropy condition, the object labeling operation is performed on the target unlabeled remote sensing image to obtain a newly labeled remote sensing image.

13. The method of claim 12, wherein, The method further comprises at least one of the following: In a case where the number of the unlabeled remote sensing images is greater than a target image number, and the target sampling parameter is greater than or equal to a preset sampling parameter threshold, it is determined that the target unlabeled remote sensing image satisfies the information entropy condition; In a case where the number of the unlabeled remote sensing images is less than or equal to a target image number, it is determined that the target unlabeled remote sensing image satisfies the information entropy condition.

14. The method of claim 10, wherein, The training of the initial classification model based on the newly labeled remote sensing image comprises: The newly labeled remote sensing image is added to the initial training data set to obtain a target training data set; training the initial classification model using the target training data set; determining the initial classification model as the target classification model when the initial classification model meets a preset model convergence condition; training the initial classification model based on the unlabeled remote sensing image on which the object labeling operation is not performed, the pseudo-label remote sensing image, and an initial training data set for a next round when the initial classification model does not meet the model convergence condition.

15. The method of claim 14, wherein, The method further includes at least one of the following: verifying the initial classification model using a remote sensing image test set to obtain an average intersection over union parameter, and determining that the initial classification model meets the model convergence condition when a variation range of the average intersection over union parameter in continuous N rounds of training is less than a preset variation parameter, where N is a positive integer; determining that the initial classification model meets the model convergence condition when a number of times of generating the newly labeled remote sensing image is equal to a preset generation number; determining that the initial classification model meets the model convergence condition when the object labeling operation is performed on each of the unlabeled remote sensing images.

16. The method of claim 14, wherein, The method further includes at least one of the following: freezing shallow model parameters of the initial classification model and updating deep model parameters of the initial classification model in a process of training the initial classification model using the initial training data set; freezing shallow model parameters of the initial classification model and updating deep model parameters of the initial classification model in a process of training the initial classification model using the intermediate training data set; updating the shallow model parameters and the deep model parameters of the initial classification model to obtain the target classification model in a process of training the initial classification model using the target training data set.

17. A device for training a model, the device comprising: includes: a generation module configured to input an unlabeled remote sensing image into an initial classification model to obtain a pseudo-label remote sensing image; a training module configured to train the initial classification model based on the unlabeled remote sensing image, the pseudo-label remote sensing image, and an initial training data set to obtain a target classification model, where one round of training of the initial classification model includes a first training phase and a second training phase, in the first training phase, the initial classification model is trained using the pseudo-label remote sensing image and the initial training data set, and in the second training phase, the initial classification model is trained using the unlabeled remote sensing image that meets an information entropy condition and the initial training data set.

18. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 16.

20. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 16.

Images