Asymmetric cross-modal large model knowledge migration method and device for remote sensing scene

Abstract

The invention discloses a remote sensing scene-oriented asymmetric cross-modal large model knowledge migration method and device, and the method comprises the steps: obtaining training sample pairs, enabling a sample RGB image and a sample MS image in one training sample pair to correspond to the same scene classification, inputting the sample MS image into a teacher model, determining a first image feature extracted from the sample MS image by the teacher model, and carrying out the recognition of the first image feature; determining a first scene classification obtained by the teacher model according to the first image feature as a pseudo label, inputting the sample RGB image into the student model, determining a second image feature extracted by the student model from the sample RGB image, determining a second scene classification obtained by the student model according to the second image feature, and inputting the second scene classification into the student model; and the student model is trained according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-annotation, so that the semantic consistency requirement of training samples can be reduced, more RGB samples are trained by using a smaller number of MS training samples, and the performance of the student model is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large-scale model knowledge transfer method and device for asymmetric cross-modal remote sensing scenes. Background Art

[0002] Remote sensing image scene classification aims to classify images in different scenes according to their semantic information to determine the scene classification corresponding to each image. It plays an important role in geological exploration, national defense and security and other fields. Common remote sensing image classification methods are often based on visible light images. The features of RGB images are extracted and classified by designing deep feature extraction networks. In recent years, with the development of large language models, some researchers have proposed using multimodal large language models to classify remote sensing images. However, due to the small number of spectral bands and low information density of RGB images, the accuracy of the classification results cannot be guaranteed.

[0003] Based on the above reasons, some researchers have proposed using multi-spectral (MS) images to increase the information density of input information and thus improve recognition performance. Although MS images have significantly improved overall recognition performance compared to RGB images, in actual applications, using MS images for classification has problems such as high acquisition cost of MS images, large computational memory overhead, and slow inference speed.

[0004] To solve the above problems, some researchers have proposed the use of cross-modal distillation technology. In the training phase, the teacher model with MS images as input will teach the student model with RGB images as input. In the inference phase, only the student model with RGB images as input needs to be used. However, the implementation of the above distillation technology is based on the premise that MS images and RGB images need to have strict semantic consistency, that is, a pair of MS images and RGB images need to be collected for the same target. Due to the shortage of MS data, it is difficult to obtain sufficient training samples for training using this distillation technology, which affects the training efficiency of the student model.

[0005] Therefore, the present invention provides a large model knowledge transfer method and device for asymmetric cross-modality remote sensing scenes. Summary of the invention

[0006] This specification provides a large model knowledge transfer method and device for asymmetric cross-modal remote sensing scenes to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions: This specification provides a large model knowledge transfer method for asymmetric cross-modal remote sensing scenes, including: Obtaining a training sample pair consisting of a sample RGB image and a sample MS image, wherein the sample RGB image and the sample MS image correspond to the same scene classification; Inputting the sample MS image into a pre-trained teacher model, determining a first image feature extracted by the teacher model from the sample MS image, and determining a first scene classification obtained by the teacher model according to the first image feature as a pseudo-label; Inputting the sample RGB image into the student model, determining a second image feature extracted by the student model from the sample RGB image, and determining a second scene classification obtained by the student model according to the second image feature; The student model is trained based on at least a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling.

[0008] Optionally, also include: Determine at least one positive sample pair and several negative sample pairs as a sample group, input the sample group into a matching model to be trained, and determine a matching judgment result output by the matching model to be trained for the sample group, wherein the positive sample pair includes an RGB image and an MS image for the same target, and the negative sample pair includes an RGB image and an MS image corresponding to different scene classifications; Training the matching model to be trained according to the matching judgment result and the actual matching situation of each sample pair in the sample group; Acquire an RGB image group to be matched and an MS image group to be matched, wherein for any RGB image in the RGB image group to be matched, there is an MS image in the MS image group to be matched that has the same scene classification as the RGB image; For any RGB image in the RGB image group to be matched, the trained matching model is used to determine an MS image matching the RGB image in the MS image group to be matched as a target image, and the target image and the RGB image are matched as a pair of training samples.

[0009] Optional, pre-trained teacher model, specifically including: Get pre-trained MS images; Inputting the pre-trained MS image into the teacher model to be trained, and determining a third scene classification output by the teacher model to be trained; The teacher model to be trained is trained according to the difference between the third scene classification and the scene annotation of the pre-trained MS image.

[0010] Optionally, the data structure of the first image feature is the same as that of the second image feature; Training the student model at least according to a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling, specifically includes: Determine, according to a cross-modal attention mechanism, a first feature map corresponding to the first image feature and a second feature map corresponding to the second image feature; determining a difference between the first image feature and the second image feature according to a domain shift loss between the first feature map and the second feature map; The student model is trained based on at least a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling.

[0011] Optionally, training the student model at least according to a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling specifically includes: The student model is trained according to a difference between the second image feature and the first image feature, a difference between the second scene classification and the pseudo-annotation, and a difference between the second scene classification and a real scene annotation corresponding to the sample RGB image.

[0012] Optionally, obtaining a training sample pair consisting of a sample RGB image and a sample MS image includes: Acquire a training sample pair consisting of a sample RGB image and a sample MS image from a training sample set, wherein the training sample set includes a plurality of training sample pairs; After training the student model at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling, the method further includes: Reacquire training sample pairs from the training sample set, and continue to train the student model based on the reacquired training samples until the number of training times reaches a training threshold, and use the student model to re-determine the second scene classification corresponding to each sample RGB image in the training sample set, and update the sample MS image matched by each sample RGB image based on each second scene classification and the first scene classification corresponding to each sample MS image, and continue to train the student model based on the updated training sample pairs.

[0013] Optionally, updating the teacher sample matched by each student sample according to each second scenario classification and the first scenario classification corresponding to each teacher sample specifically includes: For any sample RGB image in the training sample set, determine a sample MS image corresponding to the first scene classification having the smallest difference with the second scene classification of the sample RGB image as a teacher sample matched by the sample RGB image.

[0014] This specification provides a large model knowledge transfer device for asymmetric cross-modality remote sensing scenes, including: An acquisition module is used to acquire a training sample pair consisting of a sample RGB image and a sample MS image, wherein the sample RGB image and the sample MS image correspond to the same scene classification; A teacher module, inputting the sample MS image into a pre-trained teacher model, determining a first image feature extracted by the teacher model from the sample MS image, and determining a first scene classification obtained by the teacher model according to the first image feature as a pseudo-label; a student module, inputting the sample RGB image into a student model, determining a second image feature extracted by the student model from the sample RGB image, and determining a second scene classification obtained by the student model according to the second image feature; A training module is used to train the student model at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling.

[0015] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned asymmetric cross-modal large model knowledge transfer method for remote sensing scenes.

[0016] This specification provides a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned asymmetric cross-modal large model knowledge transfer method for remote sensing scenes is implemented.

[0017] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: It can be seen from the above method that this method can reduce the semantic consistency requirements for training samples while ensuring training accuracy, use fewer MS training samples to train more RGB samples, and thus improve the performance of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1This is a flowchart of an asymmetric cross-modal large model knowledge transfer method for remote sensing scenes in this specification; Figure 2 A schematic diagram of a process for determining a loss caused by a difference between a first image feature and a second image feature in this specification; Figure 3 A flowchart of an asymmetric cross-modal large model knowledge transfer method for remote sensing scenarios for updating training sample pairs in this specification; Figure 4 A schematic diagram of an asymmetric cross-modal large model knowledge transfer device for remote sensing scenes provided in this specification; Figure 5 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 The flowchart of a large model knowledge transfer method for asymmetric cross-modality of remote sensing scenes in this specification is as follows. The large model knowledge transfer method for asymmetric cross-modality of remote sensing scenes specifically includes the following steps: S100: Obtain a training sample pair consisting of a sample RGB image and a sample MS image, wherein the sample RGB image and the sample MS image correspond to the same scene classification.

[0022] The student model is trained using cross-modal distillation technology, and there are strict requirements on the semantic consistency of the data input to the teacher model and the data input to the student model. When the data input to the teacher model and the data input to the student model differ greatly, for MS images, the spectral bands other than red, green, and blue in the MS image are removed to obtain an RGB image with strict semantic consistency with the MS image (which can be understood as being collected from the same target). However, in actual applications, due to the cost of acquisition equipment, it is difficult to obtain a large number of MS images, and a small amount of MS training data is difficult to train a student model with a wider range of usage scenarios. Therefore, this specification provides an asymmetric cross-modal large model knowledge transfer method for remote sensing scenes. The execution subject of this specification can be a server for training students or other electronic devices with computing capabilities. This specification does not limit this. For the sake of convenience, the following only uses the server as the execution subject to illustrate an asymmetric cross-modal large model knowledge transfer method for remote sensing scenes provided in this specification.

[0023] First, a training sample pair consisting of a sample RGB image and a sample MS image is obtained. A training sample pair includes a sample RGB image and a sample MS image. The sample RGB image and the sample MS image in a training sample pair are collected from the same scene classification, but the sample RGB image and the sample MS image in a training sample pair may not have strict semantic consistency, that is, the sample RGB image and the sample MS image in a training sample pair may not be collected from the same target location.

[0024] The above scene classifications may be scene classifications in a common sense such as rivers, mountains, plains, cities, etc. From the above requirements for training sample pairs, it can be seen that the method provided in this specification has relatively loose requirements for the data required for training.

[0025] S102: Input the sample MS image into a pre-trained teacher model, determine a first image feature extracted by the teacher model from the sample MS image, and determine a first scene classification obtained by the teacher model according to the first image feature as a pseudo-label.

[0026] After obtaining the training sample pair, the sample MS image can be input into a pre-trained teacher model, and the teacher model at least includes a feature extraction layer, and the feature extraction layer is used to extract the first graphic feature from the MS image input into the teacher model.

[0027] The feature extraction layer may adopt various convolution methods, Scale Invariant Feature Transform (SIFT), ORB (Oriented FAST and Rotated BRIEF), and other common image feature extraction methods, which are not limited in this specification.

[0028] After extracting the first image feature, other models in the teacher model can continue to process the first image feature and obtain the first scene classification corresponding to the sample MS image, which can be represented as a probability vector composed of the probabilities that the location represented by the sample MS image belongs to each scene type. Thus, the student model can learn the processing strategy of the teacher model through the probability vector.

[0029] S104: Input the sample RGB image into the student model, determine a second image feature extracted by the student model from the sample RGB image, and determine a second scene classification obtained by the student model according to the second image feature.

[0030] On the other hand, after obtaining the training sample, the sample RGB image is input into the student model, and the student model also includes at least a feature extraction layer, and the feature extraction layer is used to extract the second image feature from the RGB image input into the student model.

[0031] It should be noted that the feature extraction layer of the student model adopts the same image feature extraction method as the feature extraction layer of the teacher model, and the data structure of the second image feature is the same as that of the first image feature, so that the difference between the first image feature and the second image feature can be determined in subsequent steps.

[0032] S106: Training the student model at least according to a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling.

[0033] Using a preset loss function, the loss corresponding to the training sample pair is determined according to the difference between the first image feature and the second image feature, and the difference between the second scene classification and the pseudo-labeling, and the student model is trained according to the gradient of the loss relative to the parameters of each model layer in the student model.

[0034] Among them, the above-mentioned loss function can adopt various commonly used loss functions, such as: cross entropy loss combined with KL divergence, etc., and this specification does not limit it here.

[0035] In one or more embodiments of the present specification, the following formula may be used to determine the loss caused by the difference between the second scene classification and the pseudo-labeling: :

[0036] Among them, KL is KL divergence, For pseudo-annotation, For the second scene classification, is the distillation temperature.

[0037] Therefore, by utilizing the difference between the second scene classification and the pseudo-annotation, the student model can learn the teacher model's specific scene classification strategy for each image. Furthermore, by utilizing the difference between the first image feature and the second image feature, the student model can learn the teacher model's strategy for extracting important features related to scene recognition from images, so that the student model in the method provided in this specification can more effectively extract image features related to scene classification in the image than the student model in the traditional knowledge distillation method, thereby reducing the semantic consistency requirements for training samples.

[0038] In subsequent practical applications, the target image is input into the trained student model to determine the scene classification corresponding to the target image.

[0039] like Figure 1 The shown method is an asymmetric cross-modal large model knowledge transfer method for remote sensing scenes. It can reduce the semantic consistency requirements for training samples while ensuring training accuracy, use fewer MS training samples to train more RGB samples, and thus improve the performance of the student model.

[0040] In addition, in Figure 1 Before step S100 shown, at least one positive sample pair and several negative sample pairs are determined as a sample group, and the sample group is input into the matching model to be trained, and the matching judgment result for the sample group output by the matching model to be trained is determined, wherein the positive sample pair includes an RGB image and an MS image for the same target, and the negative sample pair includes an RGB image and an MS image corresponding to different scene classifications. According to the matching judgment result and the actual matching situation of each sample pair in the sample group, the matching model to be trained is trained to obtain an RGB image group to be matched and an MS image group to be matched. For any RGB image in the RGB image group to be matched, there is an MS image of the same scene classification as the RGB image in the MS image group to be matched. For any RGB image in the RGB image group to be matched, the MS image matching the RGB image in the MS image group to be matched is determined as a target image using the trained matching model, and the target image and the RGB image are matched as a pair of training sample pairs.

[0041] Before training the student model, each training sample pair can be determined in advance. Specifically, an RGB image group to be matched and a MS image group to be matched can be prepared in advance. The RGB image group to be matched includes a number of RGB images to be matched, each of which corresponds to a scene classification, and the RGB image group to be matched includes RGB images to be matched of all scene classifications. Similarly, the MS image group to be matched includes a number of MS images to be matched, each of which corresponds to a scene classification, and the MS image group to be matched includes MS images to be matched of all scene classifications. Subsequently, the trained matching model is used to match the RGB image group to be matched with the MS image group to be matched. Specifically, for any RGB image to be matched, all MS images of the MS image group to be matched can be used as a pre-matching sample pair with the RGB image to be matched to determine whether each pre-matching sample pair matches. According to the matching result, each training sample pair is determined in each matching sample pair.

[0042] When training the matching model to be trained, the data of R, G, and B channels in the MS image can be extracted to obtain an RGB image for the same target with strict semantic consistency with an MS image. A pair of MS images and RGB images with strict semantic consistency are used as positive sample pairs, and a pair of RGB images with different corresponding scene classifications are used as negative sample pairs. After obtaining the matching judgment result of the output of the matching model to be trained for the sample group, the loss corresponding to the sample group can be determined according to a preset loss function, and the gradient of the loss relative to the parameters of each model layer in the matching model to be trained can be determined. The matching model to be trained is trained. After completing the preset training conditions, the trained matching model to be trained is used as the matching model used to match the training sample pairs.

[0043] In one or more embodiments of the present specification, the above-mentioned matching model may be a matcher based on CLIP (Contrastive Language–Image Pre-training). The matcher includes an MS encoder and an RGB encoder, which encode the MS image and the RGB image respectively to obtain the corresponding MS feature vector and RGB feature vector: and , and then use the InfoNCE loss function for training. For each feature vector in the positive sample pair , and its loss function is: (1) (2) in, is the number of sample pairs in each sample group, is the distillation temperature.

[0044] The total loss function of the matching model is:

[0045] In addition, in Figure 1 Before step S100 shown, a pre-trained MS image is obtained, the pre-trained MS image is input into the teacher model to be trained, the third scene classification output by the teacher model to be trained is determined, and the teacher model to be trained is trained based on the difference between the third scene classification and the scene annotation of the pre-trained MS image.

[0046] Therefore, the trained teacher model can accurately identify the scene classification of the scene captured by each MS image.

[0047] In addition, in Figure 1 In step S104 shown, according to the cross-modal attention mechanism, a first feature map corresponding to the first image feature and a second feature map corresponding to the second image feature are determined, and the difference between the first image feature and the second image feature is determined according to the domain shift loss between the first feature map and the second feature map, and the student model is trained at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling.

[0048] Specifically, according to Figure 2 The process shown in FIG. 1 determines the loss caused by the difference between the first image feature and the second image feature. .

[0049] First, determine the attention map of the first image feature Attention map with the second image feature :

[0050] Where d is the dimension of the feature vector of the image feature; and The query term and key respectively.

[0051] Then determine and :

[0052]

[0053] in, , . and are equal, indicating the number of blocks for dividing the MS image or RGB image input to the student model and the teacher model.

[0054] Furthermore, the feature map based on the cross-modal attention mechanism is calculated and :

[0055] Wherein, H is the number of heads of the attention mechanism, which is set to 8 in this embodiment.

[0056] Finally, the domain shift loss function is calculated, that is, the loss caused by the difference between the first image feature and the second image feature :

[0057] in, and It is obtained from the following formula:

[0058]

[0059] By using the difference between the first image feature and the second image feature to know the training of the student model, the student model can learn the strategy of extracting image features of the teacher model, so that the second image feature extracted by the student model contains more relevant information for scene classification. Furthermore, the second scene classification obtained by the student model using the second image feature is more accurate, reducing the requirement of the training process for semantic consistency of the training samples.

[0060] On the other hand, in Figure 1 In step S104 shown, the student model is trained according to the difference between the second image feature and the first image feature, according to the difference between the second scene classification and the pseudo-annotation, and according to the difference between the second scene classification and the real scene annotation corresponding to the sample RGB image.

[0061] The total loss of the student model is set to :

[0062] in, is the loss caused by the difference between the second scene classification and the real scene annotation corresponding to the sample RGB image, and This has been fully explained above and will not be repeated here.

[0063] In this way, the accuracy of the second scene classification obtained by the trained student model can be further increased, thereby reducing the requirement of the training process for the semantic consistency of the training samples.

[0064] In contrast, in Figure 1 In step S100 shown in FIG. 1 , a training sample pair consisting of a sample RGB image and a sample MS image is obtained from a training sample set, wherein the training sample set includes a plurality of training sample pairs. Figure 1 After step S106 shown, the training sample pairs are reacquired from the training sample set, and the student model is continuously trained based on the reacquired training samples until the number of training times reaches the training threshold, and the second scene classification corresponding to each sample RGB image in the training sample set is re-determined using the student model, and the sample MS image matched by each sample RGB image is updated based on each second scene classification and the first scene classification corresponding to each sample MS image, and the student model is continuously trained based on each updated training sample pair.

[0065] During the training process, Figure 3 As shown, firstly, step S300 is performed: a training sample pair consisting of a sample RGB image and a sample MS image is obtained from a training sample set, wherein the training sample set includes a plurality of training sample pairs.

[0066] Then, step S302 is executed: the sample MS image is input into a pre-trained teacher model, a first image feature extracted from the sample MS image by the teacher model is determined, and a first scene classification obtained by the teacher model according to the first image feature is determined as a pseudo-label.

[0067] Next, step S304 is performed to input the sample RGB image into the student model, determine the second image feature extracted by the student model from the sample RGB image, and determine the second scene classification obtained by the student model according to the second image feature.

[0068] Further executing step S306: training the student model at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling.

[0069] Then execute step S308: determine whether the number of training times reaches the training threshold; if not, re-execute steps S300-S308; otherwise, execute step S310: use the student model to re-determine the second scene classification corresponding to each sample RGB image in the training sample set, update the sample MS image matching each sample RGB image according to each second scene classification and the first scene classification corresponding to each sample MS image, then reset the statistics of the number of training times for the training threshold, and re-execute steps S300-S308 until the training is completed.

[0070] The training completion can be set to when the total number of training times reaches a preset number, or the loss value calculated by the student model is lower than a preset loss threshold.

[0071] Therefore, the student model can be used to re-determine the scene classification of each sample RGB image in the training sample set after a certain number of training times. As the scene classification accuracy of the student model improves during the training process, the scene classification of the sample RGB image becomes more accurate, increasing the amount of information that the student model can learn from the training sample pairs during each training process.

[0072] Specifically, in Figure 3 In step S308 shown, for any sample RGB image in the training sample set, a sample MS image corresponding to the first scene classification having the smallest difference with the second scene classification of the sample RGB image is determined as a teacher sample matched by the sample RGB image.

[0073] Specifically, the sample MS image corresponding to the first scene classification having the smallest difference with the second scene classification of the sample RGB image can be determined by calculating the KL divergence between the second scene classification and each first scene classification.

[0074] The above is a large model knowledge transfer method for asymmetric cross-modality of remote sensing scenes provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding large model knowledge transfer device for asymmetric cross-modality of remote sensing scenes, such as Figure 4 shown.

[0075] Figure 4 A schematic diagram of a large model knowledge transfer device for asymmetric cross-modal remote sensing scenes provided in this specification specifically includes: An acquisition module 400 is configured to acquire a training sample pair consisting of a sample RGB image and a sample MS image, wherein the sample RGB image and the sample MS image correspond to the same scene classification; The teacher module 402 inputs the sample MS image into a pre-trained teacher model, determines a first image feature extracted by the teacher model from the sample MS image, and determines a first scene classification obtained by the teacher model according to the first image feature as a pseudo-label; The student module 404 inputs the sample RGB image into the student model, determines a second image feature extracted by the student model from the sample RGB image, and determines a second scene classification obtained by the student model according to the second image feature; The training module 406 trains the student model according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling.

[0076] Optionally, the acquisition module 400 is further used to: determine at least one positive sample pair and several negative sample pairs as a sample group, input the sample group into the matching model to be trained, and determine the matching judgment result for the sample group output by the matching model to be trained, wherein the positive sample pair includes an RGB image and an MS image for the same target, and the negative sample pair includes an RGB image and an MS image corresponding to different scene classifications; according to the matching judgment result and the actual matching situation of each sample pair in the sample group, the matching model to be trained is trained to obtain the RGB image group to be matched and the MS image group to be matched; for any RGB image in the RGB image group to be matched, there is an MS image of the same scene classification as the RGB image in the MS image group to be matched; for any RGB image in the RGB image group to be matched, using the trained matching model, determine the MS image matching the RGB image in the MS image group to be matched as the target image, and match the target image and the RGB image as a pair of training sample pairs.

[0077] Optionally, the teacher module 402 is also used to: obtain a pre-trained MS image, input the pre-trained MS image into the teacher model to be trained, determine a third scene classification output by the teacher model to be trained, and train the teacher model to be trained based on the difference between the third scene classification and the scene annotation of the pre-trained MS image.

[0078] Optionally, the data structure of the first image feature is the same as that of the second image feature; The training module 406 is specifically used to: determine, according to the cross-modal attention mechanism, a first feature map corresponding to the first image feature and a second feature map corresponding to the second image feature, determine the difference between the first image feature and the second image feature according to the domain shift loss between the first feature map and the second feature map, and train the student model at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling.

[0079] Optionally, the training module 406 is specifically used to train the student model according to the difference between the second image feature and the first image feature, the difference between the second scene classification and the pseudo-annotation, and the difference between the second scene classification and the real scene annotation corresponding to the sample RGB image.

[0080] Optionally, the acquisition module 400 is specifically used to: acquire a training sample pair consisting of a sample RGB image and a sample MS image from a training sample set, wherein the training sample set includes a plurality of training sample pairs; The training module 406 is further used to: reacquire training sample pairs from the training sample set, and continue to train the student model based on the reacquired training samples until the number of training times reaches a training threshold, and use the student model to re-determine the second scene classification corresponding to each sample RGB image in the training sample set, and update the sample MS image matched by each sample RGB image according to each second scene classification and the first scene classification corresponding to each sample MS image, and continue to train the student model according to each updated training sample pair.

[0081] Optionally, the training module 406 is specifically used to: for any sample RGB image in the training sample set, determine the sample MS image corresponding to the first scene classification with the smallest difference from the second scene classification of the sample RGB image as the teacher sample matched by the sample RGB image.

[0082] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A large-model knowledge transfer method for asymmetric cross-modal remote sensing scenes is provided.

[0083] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5As mentioned above, at the hardware level, the asymmetric cross-modal large model knowledge migration device for remote sensing scenes includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The asymmetric cross-modal large model knowledge transfer method for remote sensing scenes. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0084] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0085] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0086] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0087] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0088] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0093] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0094] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0095] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0096] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data classifications. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0098] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0099] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A large model knowledge transfer method for asymmetric cross-modal remote sensing scenes, characterized in that: The method comprises: Obtaining a training sample pair consisting of a sample RGB image and a sample MS image, wherein the sample RGB image and the sample MS image correspond to the same scene classification; Inputting the sample MS image into a pre-trained teacher model, determining a first image feature extracted by the teacher model from the sample MS image, and determining a first scene classification obtained by the teacher model according to the first image feature as a pseudo-label; Inputting the sample RGB image into the student model, determining a second image feature extracted by the student model from the sample RGB image, and determining a second scene classification obtained by the student model according to the second image feature; The student model is trained based on at least a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling.

2. The method according to claim 1, characterized in that Also includes: Determine at least one positive sample pair and several negative sample pairs as a sample group, input the sample group into a matching model to be trained, and determine a matching judgment result output by the matching model to be trained for the sample group, wherein the positive sample pair includes an RGB image and an MS image for the same target, and the negative sample pair includes an RGB image and an MS image corresponding to different scene classifications; Training the matching model to be trained according to the matching judgment result and the actual matching situation of each sample pair in the sample group; Acquire an RGB image group to be matched and an MS image group to be matched, wherein for any RGB image in the RGB image group to be matched, there is an MS image in the MS image group to be matched that has the same scene classification as the RGB image; For any RGB image in the RGB image group to be matched, the trained matching model is used to determine an MS image matching the RGB image in the MS image group to be matched as a target image, and the target image and the RGB image are matched as a pair of training samples.

3. The method according to claim 1, characterized in that Pre-train the teacher model, including: Get pre-trained MS images; Inputting the pre-trained MS image into the teacher model to be trained, and determining a third scene classification output by the teacher model to be trained; The teacher model to be trained is trained according to the difference between the third scene classification and the scene annotation of the pre-trained MS image.

4. The method according to claim 1, characterized in that The data structure of the first image feature is the same as that of the second image feature; Training the student model at least according to a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling, specifically includes: Determine, according to a cross-modal attention mechanism, a first feature map corresponding to the first image feature and a second feature map corresponding to the second image feature; determining a difference between the first image feature and the second image feature according to a domain shift loss between the first feature map and the second feature map; The student model is trained based on at least a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling.

5. The method according to claim 1, characterized in that Training the student model at least according to a difference between the second image feature and the first image feature and a difference between the second scene classification and the pseudo-labeling, specifically includes: The student model is trained according to a difference between the second image feature and the first image feature, a difference between the second scene classification and the pseudo-annotation, and a difference between the second scene classification and a real scene annotation corresponding to the sample RGB image.

6. The method according to claim 1, characterized in that Obtain a training sample pair consisting of a sample RGB image and a sample MS image, specifically including: Acquire a training sample pair consisting of a sample RGB image and a sample MS image from a training sample set, wherein the training sample set includes a plurality of training sample pairs; After training the student model at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling, the method further includes: Reacquire training sample pairs from the training sample set, and continue to train the student model based on the reacquired training samples until the number of training times reaches a training threshold, and use the student model to re-determine the second scene classification corresponding to each sample RGB image in the training sample set, and update the sample MS image matched by each sample RGB image based on each second scene classification and the first scene classification corresponding to each sample MS image, and continue to train the student model based on the updated training sample pairs.

7. The method according to claim 6, characterized in that According to each second scenario classification and the first scenario classification corresponding to each teacher sample, the teacher sample matched by each student sample is updated, specifically including: For any sample RGB image in the training sample set, determine a sample MS image corresponding to a first scene classification having the smallest difference with a second scene classification of the sample RGB image as a teacher sample matched by the sample RGB image.

8. A large model knowledge transfer device for asymmetric cross-modal remote sensing scenes, characterized in that: include: An acquisition module is used to acquire a training sample pair consisting of a sample RGB image and a sample MS image, wherein the sample RGB image and the sample MS image correspond to the same scene classification; A teacher module, inputting the sample MS image into a pre-trained teacher model, determining a first image feature extracted by the teacher model from the sample MS image, and determining a first scene classification obtained by the teacher model according to the first image feature as a pseudo-label; a student module, inputting the sample RGB image into a student model, determining a second image feature extracted by the student model from the sample RGB image, and determining a second scene classification obtained by the student model according to the second image feature; A training module is used to train the student model at least according to the difference between the second image feature and the first image feature and the difference between the second scene classification and the pseudo-labeling.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. A device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images