Remote sensing image multi-mode incremental panoramic sensing method and system

Abstract

The invention discloses a remote sensing image multi-modal incremental panoramic perception method and system, and the method comprises the steps: constructing a remote sensing image multi-modal panoramic perception model based on incremental learning, and the model comprises a cross-modal general image encoder module, a cross-modal interaction knowledge distillation module, a cross-modal image encoder module, a cross-modal image encoder module and a cross-modal image encoder module; a task bias pseudo-mark generation module and a symmetric query interaction collaborative classification module; performing incremental training on the model; and performing multi-level and multi-mode continuous interpretation on the remote sensing image by using the trained model, and completing cross-mode multiple tasks of pixel-level classification, instance-level segmentation and image-level description of the remote sensing image. The method breaks through the limitation that an existing remote sensing image interpretation method only realizes predefined single-task interpretation, expands an interpretation product from a single form to a pixel-level, instance-level and image-level multi-mode increment interpretation mode, and can realize brain-like perception of the remote sensing image; the dependence of a deep learning model on data storage and labeling is reduced, the model training and maintenance cost is reduced, and the intelligent degree is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image interpretation, and in particular to a remote sensing image multi-modal incremental panoramic perception method and system. Background Art

[0002] Remote sensing images have rich semantics and complex backgrounds. Remote sensing image interpretation is of great significance to earth observation, resource exploration and other fields. With the technological progress in the fields of aerospace remote sensing and computer vision in recent years, remote sensing image interpretation technology has developed in the direction of automation and intelligence. Remote sensing image interpretation technology involves knowledge in multiple fields such as artificial intelligence, computer science, and statistics.

[0003] At present, remote sensing image interpretation systems usually cover multiple interpretation models such as pixel classification, target detection and image description. However, most of the existing remote sensing image interpretation methods only support single-task interpretation, and remote sensing image interpretation methods based on multi-task learning face bottlenecks such as difficulty in labeling, difficulty in multi-task joint optimization, and poor model generalization ability. It is difficult to meet the needs of incremental interpretation of continuously growing remote sensing image data.

[0004] Therefore, how to achieve continuous multi-task interpretation at the pixel level, instance level, and image level is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a multi-modal incremental panoramic perception method and system for remote sensing images that solve at least some of the above-mentioned technical problems, while covering joint interpretation at the pixel level, instance level and image level, and combining incremental learning methods with cross-modal multi-task learning mechanisms to achieve multi-task continuous interpretation of remote sensing images.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, an embodiment of the present invention provides a remote sensing image multi-modal incremental panoramic perception method, the method comprising the following steps:

[0008] S1. Constructing a multimodal panoramic perception model of remote sensing images based on incremental learning; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal general image encoder module, a cross-modal interactive knowledge distillation module, a task bias pseudo-annotation generation module and a symmetric query interactive collaborative classification module;

[0009] S2. performing incremental training on the constructed multimodal panoramic perception model of remote sensing images;

[0010] S3. Input the remote sensing image data into the trained multimodal panoramic perception model of remote sensing images, perform multi-level and multimodal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images.

[0011] Furthermore, in the cross-modal universal image encoder module, the image features are divided into mask feature embedding and text feature embedding by sharing an encoder, wherein: the mask feature embedding is used for image segmentation tasks, and the text feature embedding is used for image description tasks, specifically:

[0012] For the input image x at step t, the image feature f is defined as:

[0013] f=Θ(ReLU(Θ(E(x))))+E(x)

[0014] In the formula, Θ(·) represents linear mapping, ReLU represents activation function, and E(·) represents image feature encoder;

[0015] After obtaining the image feature f, a Transformer decoder is used to generate an image mask, and another Transformer decoder is used to generate a text description. Specifically, semantic segmentation and instance segmentation are unified into the image segmentation task, and the image feature f is used to generate N mask feature embeddings. For the image description task, the image feature f is used to generate text feature embeddings.

[0016] Furthermore, the cross-modal interaction knowledge distillation module includes two parts: intermediate layer contrast distillation and cross-guided instance distillation, wherein:

[0017] In the middle layer contrast distillation part, the features output by the cross-modal universal image encoder module are used as input to integrate the learning tasks of the two modalities of image segmentation and image description into the contrastive learning framework. The calculation method is:

[0018]

[0019] Where, L ICD represents the intermediate layer contrast distillation loss, d(·) represents the similarity metric function, D CL (·) represents contrastive learning loss, |·| represents absolute value operation, and I(·) is the indicator function; and Respectively represent the t-1 step model M t-1 and the t-th step model M t The output segmentation mask, and Respectively represent the t-1 step model M t-1 and the t-th step model M t Output text description result, Mt-1 and M t They represent the model of step t-1 and the model trained in step t respectively; and Respectively represent anchor feature, positive sample feature and negative sample feature; C 0:t-1 Indicates the category that has been learned at step t-1; C 0:t Indicates the category that has been learned at step t, i, j indicates that it belongs to C 0:t The category number, k means it belongs to C 0:t-1 The category number of

[0020] In the cross-guided instance distillation part, an adaptive weight mechanism is used to improve the discrimination ability of fine-grained target instances. Specifically, the quality score q is constructed. r , which is used to guide the output of the new model in the distillation process to be closer to the output of the old model, so as to reduce the catastrophic forgetting of old knowledge; the quality score q r The calculation method is:

[0021]

[0022] In the formula, c r Represents the t-step model M t The output is the category prediction result for instance r, γ represents the parameter used to control the weight of classification and segmentation tasks, Represents the t-1 step model M t-1 The segmentation result of the output instance r is The features of the model output at step t are expressed by M t-1 The decoder outputs the segmentation result of instance r, and IoU represents the intersection over union calculation.

[0023] Furthermore, the optimization goal of cross-guided instance distillation is:

[0024]

[0025] Where, L CID represents the cross-guided instance distillation loss, and They represent the prediction results of the model output at the t-1th step and the prediction results of the model output at the tth step, respectively, and d(·) represents the similarity measurement function.

[0026] Furthermore, the optimization goal of the cross-modal interactive knowledge distillation module is to comprehensively calculate the intermediate layer contrast distillation loss and the cross-guided instance distillation loss:

[0027] L TKD =L ICD +L CID

[0028] Where, L TKD represents the knowledge distillation loss function.

[0029] Furthermore, in the task bias pseudo-annotation generation module, for the image segmentation task, for any pixel i in the image x, its annotation calculation method in the tth step is:

[0030]

[0031] In the formula, represents the generated annotation for pixel i during the t-th step of training, Represents the t-1 step model M t-1 The output prediction result for pixel i, x i represents pixel i, c in image x b represents the background category, C 0:t-1 represents the category that has been learned at step t-1, C t represents the incremental category of the t-th step training, Γ is the threshold used to control the confidence of the generated pseudo-annotation, and ζ is a positive constant used to measure the stability of the model prediction results; Represents the output text description result of image x;

[0032] For the image description task, at step t, the text description is annotated The calculation method is:

[0033]

[0034] in Represents the output text description of image x, represents the manually annotated text description that can be obtained in step t, Represents a concatenation operation.

[0035] Furthermore, in the symmetric query interactive collaborative classification module, the mask feature Q output by the cross-modal universal image encoder is m and text features Q t , two symmetrical interactive attention models are used to perform feature fusion interaction.

[0036] Q′ m =ψ(Q m ,Q t )

[0037] Q t ′=ψ(Q t ,Q m )

[0038] In the formula, ψ represents the interactive attention calculation process, Q′ m represents the fused mask embedding, Qt ′ represents the fused text embedding.

[0039] Furthermore, the constructed multimodal incremental panoramic perception model of remote sensing images supports end-to-end training, and the optimization goal is to comprehensively minimize the pixel classification loss L cls , image segmentation loss L seg , image description loss L cap and knowledge distillation loss L CIKD , calculated as:

[0040] L total =L cls +λL seg +L cap +L CIKD

[0041] Where, L total represents the joint loss value, λ represents the weight coefficient of image segmentation loss;

[0042] Where: Image classification loss L cls Defined using cross entropy loss:

[0043]

[0044] In the formula, p i Represents the model's prediction result for pixel i, y i Indicates the marked value;

[0045] Image segmentation loss L seg The calculation method is:

[0046]

[0047] Where I(·) is the indicator function, represents the label corresponding to category c, represents the empty set, L focal represents focal loss, L dice represents dice loss, η1 and η2 represent weight coefficients, represents the model prediction value, represents the annotation value of step t;

[0048] Image description loss L cap The calculation method is:

[0049]

[0050] In the formula, p cap Represents the image description prediction value output by the model, represents the text description annotation of the tth step, and L represents the length of the text description sentence.

[0051] Furthermore, the constructed multi-modal incremental panoramic perception model of remote sensing images is trained, and the training process includes:

[0052] ①、The t-1 step model M t-1 The parameters of the training dataset D are frozen using the t-th step training dataset D t , train and get the t-step model M t Parameters;

[0053] ② For the image x∈D used in the t-th step training t , first feed it into the cross-modal universal image encoder module to generate image features f, and then feed f into the Transformer decoder to generate mask feature embedding and text feature embedding;

[0054] ③、The t-1 step model M t-1 Generated mask feature embedding and text feature embedding, model M at step t t The generated mask feature embedding and text feature embedding are simultaneously fed into the cross-modal interactive knowledge distillation module to calculate the knowledge distillation loss function L TKD ;

[0055] ④. Using the t-1 step model M t-1 The prediction results generate pseudo annotations, which are combined with the real annotations obtained from the training dataset in step t to generate the model M in step t t All annotations required for training;

[0056] ⑤. Based on the fusion annotation obtained in step ④, calculate the t-th step model M t The pixel classification loss L of the output cls , image segmentation loss L seg and image description loss L cap , and the knowledge distillation loss function L obtained in step ③ CIKD Conduct joint optimization;

[0057] ⑥ Repeat steps ② to ⑤, iteratively train the model until convergence, and obtain the model M of the tth step t , used to interpret the newly added semantics in step t.

[0058] In a second aspect, an embodiment of the present invention further provides a remote sensing image multimodal incremental panoramic perception system, which applies the above-mentioned remote sensing image multimodal incremental panoramic perception method to perform multi-level and multimodal continuous interpretation of remote sensing images, and completes cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images. The system includes:

[0059] A model building module, used to build a multimodal panoramic perception model of remote sensing images based on incremental learning; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal general image encoder module, a cross-modal interactive knowledge distillation module, a task bias pseudo-annotation generation module and a symmetric query interactive collaborative classification module;

[0060] A model training module, used for performing incremental training on the constructed multimodal panoramic perception model of remote sensing images;

[0061] The model application module is used to input remote sensing image data into the trained multimodal panoramic perception model of remote sensing images, perform multi-level and multimodal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images.

[0062] Compared with the prior art, the present invention has at least the following beneficial effects:

[0063] The present invention aims at the multi-task collaborative interpretation requirements of panoramic perception interpretation of remote sensing images, and the continuous interpretation requirements of a steady stream of newly added data, and provides a remote sensing image multi-modal incremental panoramic perception method, which can realize pixel-level, instance-level and image-level multi-task interpretation, and can realize the generalization of newly added semantics for the continuously added interpretation requirements, and realize the learning of new knowledge without losing old knowledge. The present invention can use incremental training to realize the interpretation of newly added semantics of remote sensing images, including the cross-modal multi-task interpretation effects of pixel-level classification, instance-level segmentation and image-level description of images.

[0064] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0065] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0068] Figure 1 A schematic diagram of the flow of a multi-modal incremental panoramic perception method for remote sensing images provided in an embodiment of the present invention.

[0069] Figure 2 A schematic diagram of the working principle of the multimodal incremental panoramic perception method for remote sensing images provided in an embodiment of the present invention.

[0070] Figure 3 A schematic diagram of the effect of the multi-modal incremental panoramic perception method for remote sensing images provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0072] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0073] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Panoramic perception of remote sensing images is a new form of remote sensing image interpretation, covering multiple aspects and multiple levels of interpretation. It aims to address the deficiency of existing remote sensing image interpretation models that can only support a single task, unify multi-task interpretation forms and cross-modal interpretation results into the interpretation model, and achieve comprehensive perception and intelligent interpretation of remote sensing images. Incremental learning can drive deep learning models to continuously adapt to new tasks. Incremental learning aims to break through the limitation that deep learning models can only be trained and reasoned on fixed semantic categories on closed data sets, and achieve incremental interpretation of a steady stream of newly added semantics.

[0075] See also Figure 1 and Figure 2As shown, the present invention provides a multi-modal incremental panoramic perception method for remote sensing images, which realizes brain-like multi-level collaborative perception and interpretation from the joint incremental learning of semantic segmentation, instance segmentation and sentence description, and is a new technical approach to remote sensing image interpretation using deep learning models; the method mainly includes the following steps:

[0076] S1. Constructing a multimodal panoramic perception model of remote sensing images based on incremental learning; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal general image encoder module, a cross-modal interactive knowledge distillation module, a task bias pseudo-annotation generation module and a symmetric query interactive collaborative classification module;

[0077] S2. performing incremental training on the constructed multimodal panoramic perception model of remote sensing images;

[0078] S3. Input the remote sensing image data into the trained multimodal panoramic perception model of remote sensing images, perform multi-level and multimodal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images.

[0079] The working principle and specific implementation methods of the method of the present invention are described in detail below:

[0080] 1. Task definition:

[0081] In the embodiment of the present invention, the incremental semantic segmentation task for panoramic perception is described as follows: Given a training set D = {x i ,y i ,r i},in represents the input image, Represents x i The corresponding pixel-level annotation, r i represents the sentence description annotation. C represents the number of image channels, H and W represent the height and width of the image in pixels respectively. t represents the training set of the tth incremental learning step, C 0:t-1 represents the category that has been learned at step t-1, C t represents the category learned in the tth step. For the panoramic perception incremental learning task, when D t When training a model on 0 ,D 1 ,···,D t-1} Labeled data of old categories, and only the current category D t The corresponding annotation category. The training steps of category incremental semantic segmentation include {Step 0 ,Step 1 ,···,Step t-1 ,Stept ,···,Step T}, in the tth incremental learning step, M t-1 and M t They represent the model of the t-1th step and the model trained in the tth step respectively.

[0082] 2. Model structure:

[0083] In an embodiment of the present invention, a multimodal incremental panoramic perception model of remote sensing images is constructed; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal universal image encoder module (Cross-modal Universal Image Encoder, CUIE), a cross-modal interactive knowledge distillation module (Cross-modal Interactive Knowledge Distillation, CIKD), a task bias pseudo-labeling generation module (Task-bias Pseudo-labeling, TBPL) and a symmetric query interactive collaborative classification module (Symmetric Query Interactive Classification, SQIC); the functions of each module are:

[0084] (1) Cross-modal universal image encoder

[0085] In the panoramic perception incremental learning task, the model needs to be able to encode image features into mask feature embedding and text feature embedding at the same time. This paper proposes a cross-modal universal image encoder, which constructs a shared encoder to divide image features into mask feature embedding and text feature embedding, where mask feature embedding is used for image segmentation tasks and text feature embedding is used for image description tasks. For the input image of step t The image feature f is defined as:

[0086] f=Θ(ReLU(Θ(E(x))))+E(x)

[0087] Where Θ(·) represents the linear mapping, ReLU represents the activation function, and E(·) represents the image feature encoder.

[0088] After obtaining the image feature f, a standard Transformer decoder is used to generate an image mask and another standard Transformer decoder is used to generate a text description. The standard Transformer decoder consists of a self-attention layer, a cross-attention layer, and a forward network. Specifically, the present invention unifies semantic segmentation and instance segmentation into the image segmentation task, and uses the image feature f to generate N mask embeddings. Among them C qrepresents the mask dimension. For the image description task, image features are used to generate text embeddings Where L represents the length of the text.

[0089] (2) Cross-modal interactive knowledge distillation

[0090] In the embodiment of the present invention, in order to improve the knowledge transfer ability between different modalities during the learning process, a cross-modal interactive knowledge distillation module is constructed. and Respectively represent the segmentation masks output by the t-1th step model and the tth step model; for the image description task, and They represent the text description results output by the t-1th step model and the tth step model respectively.

[0091] The cross-modal interactive knowledge distillation module constructed by the present invention includes two parts: intermediate layer contrast distillation and cross-guided example distillation. Among them, the intermediate layer contrast distillation uses the features output by the cross-modal collaborative encoder as input, and integrates the learning tasks of the two modalities of image segmentation and image description into the contrastive learning framework, where:

[0092]

[0093] Where, L ICD represents the intermediate layer contrast distillation loss, and d(·) represents the similarity metric function used to constrain the t-1 step model M t-1 and the t-th step model M t The characterization consistency. CL (·) represents contrastive learning loss, |·| represents absolute value operation, and I(·) is the indicator function; C 0:t-1 Indicates the category that has been learned at step t-1; C 0:t Indicates the category that has been learned at step t, i, j indicates that it belongs to C 0:t The category number, k means it belongs to C 0:t-1 The category number of and Represent the anchor feature, positive sample feature and negative sample feature respectively, where the anchor feature is represented by M t-1 Output, positive sample features and negative sample features are represented by M t Output.

[0094] Cross-guided instance distillation is aimed at the characteristics of panoramic perception of remote sensing images, such as a large number of instances, fine category distinction, and complex background semantics. The distillation method in the existing incremental learning method does not distinguish between foreground instances and background semantics, and directly distills the feature map, which introduces more background interference. In response to the need to distinguish target instances in panoramic perception tasks, the present invention constructs an adaptive weight mechanism to improve the ability to distinguish fine-grained target instances in the incremental learning step. Specifically, the quality score q is constructed. r , which is used to guide the output of the new model in the distillation process to be closer to the output of the old model, so as to reduce the catastrophic forgetting of old knowledge. r The calculation method is:

[0095]

[0096] In the formula, c r Represents the t-step model M t The output is the category prediction result for instance r, γ is used to control the weights of classification and segmentation tasks, Represents the t-1 step model M t-1 The segmentation result of the output instance r is The features of the model output at step t are expressed by M t-1 The decoder outputs the segmentation result of instance r. IoU represents the intersection over union calculation process. The optimization target L of cross-guided instance distillation is CID for:

[0097]

[0098] In the formula, and They represent the prediction results of the model output at the t-1th step and the prediction results of the model output at the tth step, respectively, and d(·) represents the similarity measurement function.

[0099] In summary, the optimization goal of cross-modal interactive knowledge distillation is to comprehensively calculate the intermediate layer contrast distillation loss and the cross-guided instance distillation loss:

[0100] L TKD =L ICD +L CID

[0101] Where, L TKD Loss function representing task-interactive knowledge distillation.

[0102] (3) Task bias pseudo-annotation generation

[0103] In an embodiment of the present invention, the panoramic perception incremental learning method proposed in the present invention does not need to store old data, but adopts a pseudo-annotation generation method to obtain supervisory information of known categories. Since the model can only obtain data and annotations of newly added categories in the training of each incremental step, the background category contains both known categories and unknown categories. This semantic confusion phenomenon causes the degradation of the model classifier. The present invention proposes task biased pseudo-annotation generation that can adaptively obtain the results of subtasks with higher confidence as the basis for pseudo-annotation generation, thereby improving the confidence of pseudo-annotations.

[0104] Specifically, for the image segmentation task, for any pixel i in the image x, the calculation method of its annotation at the tth step is:

[0105]

[0106] in, represents the generated annotation for pixel i during the t-th step of training, Represents the t-1 step model M t-1 The output prediction result for pixel i, x i represents pixel i, c in image x b represents the background category, C 0:t-1 represents all categories known in the previous t-1 step training, C t represents the incremental category of the t-th step training, Γ is the threshold used to control the confidence of the generated pseudo-annotation, and ζ is a positive constant used to measure the stability of the model prediction results.

[0107] For the image description task, at step t, the text description is annotated The calculation method is:

[0108]

[0109] in Represents the output text description of image x, indicating The manually annotated text description that can be obtained in step t, Represents a concatenation operation.

[0110] (4) Symmetric query interactive collaborative classification

[0111] In the embodiment of the present invention, in order to enable both the mask classification and fine-grained sentence description tasks to identify the fine-grained categories of objects in the image, a collaborative classification module based on symmetric query interaction is designed. The intermediate layer features extracted from the sentence description generation module are fused with the features used for mask classification, and the fused features are used for the final mask classification. Specifically, for the mask embedding Q output by the image encoder, m and text embedding Q t, two symmetrical interactive attention modules are used to perform fusion interaction between features. The calculation method is:

[0112] Q′ m =ψ(Q m ,Q t )

[0113] Q t ′=ψ(Q t ,Q m )

[0114] In the formula, ψ represents the interactive attention calculation process, Q′ m represents the fused mask embedding, Q t ′ represents the fused text embedding. The interactive attention module consists of a multi-head attention module and a feedforward neural network. Finally, the two symmetrical fused features are concatenated and fused with a linear layer for mask classification.

[0115] 3. Model training method:

[0116] In the embodiment of the present invention, the model proposed in the present invention supports end-to-end training, and the optimization goal of the model is to comprehensively minimize the pixel classification loss L cls , image segmentation loss L seg , image description loss L cap and knowledge distillation loss L CIKD . The image classification loss L cls Defined using cross entropy loss:

[0117]

[0118] Where p i Represents the model's prediction result for pixel i, y i Indicates the dimension value.

[0119] The image segmentation loss is calculated as:

[0120]

[0121] Where I(·) is the indicator function, represents the label corresponding to category c, represents the empty set, L focal represents focal loss, L dice represents dice loss, η1 and η2 represent weight coefficients, represents the model prediction value, Represents the label value of step t.

[0122] The image description loss is calculated as:

[0123]

[0124] Where p cap Represents the image description prediction value output by the model, Indicates the text description annotation of step t. L indicates the length of the text description sentence.

[0125] In summary, the optimization goal of the multimodal incremental panoramic perception model of remote sensing images proposed in this invention is to minimize the following joint loss function:

[0126] L total =L cls +λL seg +L cap +L CIKD

[0127] Where, L total represents the joint loss value, and λ represents the weight coefficient of the image segmentation loss.

[0128] 4. Model training:

[0129] In the embodiment of the present invention, the training steps of the remote sensing image multimodal incremental panoramic perception model specifically include:

[0130] ①. Generally, for the t-th learning step, the t-1-th step model M t-1 The parameters of the training dataset D are frozen using the t-th step training dataset D t , train and get the t-step model M t Parameters;

[0131] ② For the image data x∈D used in the t-th step training t , first feed it into the cross-modal universal image encoder CUIE to generate image features f, and then feed f into the Transformer decoder to generate mask feature embedding Q m and text feature embedding Q t ;

[0132] ③. Embed the mask features generated by the model in step t-1 and text feature embedding Mask feature embedding generated by the model at step t and text feature embedding At the same time, it is sent to the cross-modal interactive knowledge distillation module CIKD to calculate the knowledge distillation loss function L CIKD ;

[0133] ④. Using the t-1 step model M t-1 The prediction results generate pseudo annotations, which are combined with the real annotations obtained from the training dataset in step t to generate the model M in step t t All annotations required for training;

[0134] ⑤. Based on the fusion annotation obtained in step ④, calculate the t-th step model M t The pixel classification loss L of the output cls , image segmentation loss L seg and image description loss L cap , and the knowledge distillation loss function L obtained in step ③ CIKD Conduct joint optimization;

[0135] ⑥ Repeat steps ② to ⑤, iteratively train the model until the model converges, and obtain the model M of the tth step t , used to interpret the newly added semantics in step t.

[0136] 5. Model application:

[0137] In the embodiment of the present invention, in response to the multi-task collaborative interpretation requirements of panoramic perception interpretation of remote sensing images, and the continuous interpretation requirements for a steady stream of new data, the remote sensing image multimodal incremental panoramic perception method constructed by the present invention can achieve multi-task interpretation at the pixel level, instance level and image level, and can generalize the new semantics for the continuously increasing interpretation requirements, and learn new knowledge without losing old knowledge. Figure 3 As shown, the multimodal incremental panoramic perception method for remote sensing images proposed in the present invention can utilize incremental training to achieve the interpretation of newly added semantics of remote sensing images, including cross-modal multi-task interpretation effects of pixel-level classification, instance-level segmentation and image-level description of images.

[0138] 6. Experimental verification:

[0139] In the embodiment of the present invention, the effect of the multimodal incremental panoramic perception method of remote sensing images proposed by the present invention is verified on a general remote sensing image panoramic perception dataset. The dataset is the FineGrip panoramic perception dataset, which contains 25 categories of semantics. The experiment sets three incremental learning settings, namely 20-5, 15-5 and 15-2. The connotation of the above incremental learning settings is: for 20-5, the first step is to learn 20 categories, and then each incremental learning step learns 5 categories, so there are 2 incremental learning steps in total; for 15-5, the first step is to learn 15 categories, and then each incremental learning step learns 5 categories, so there are 3 incremental learning steps in total; for 15-2, the first step is to learn 15 categories, and then each incremental learning step learns 2 categories, so there are 6 incremental learning steps in total. The method proposed by the present invention is subjected to an ablation experiment on the FineGrip dataset, and the panoramic quality score (PQ) is used as the evaluation index of the image segmentation result, and the Bilingual Evaluation Understudy (BLEU) is used as the evaluation index of the image description task. The results are shown in Table 1.

[0140] Table 1 Ablation experiment results

[0141]

[0142] In Table 1: fine-tuning represents the isolated learning method, that is, only the newly added category annotations are used for training; ICD represents the training of the model constrained by intermediate layer contrast distillation; CID represents the training of the model constrained by cross-guided instance distillation; TBPL represents the training of the model participating in the generation of task biased pseudo-annotations; all represents the average results of all categories.

[0143] Experimental results show that compared with the traditional single-task isolated learning method, the remote sensing image multimodal incremental panoramic perception model proposed in this invention can realize multi-task joint incremental learning tasks and effectively improve the performance of each subtask.

[0144] From the description of the above embodiments, those skilled in the art can know that the present invention proposes a multi-modal incremental panoramic perception method for remote sensing images; this method breaks through the limitation of existing remote sensing image interpretation methods that only realize single-task interpretation, and expands the interpretation product from a single form to a multi-modal mode at the pixel level, instance level and image level, which can realize brain-like perception of remote sensing images and reduce the gap between machine learning models and human perception patterns. It can reduce the dependence of deep learning models on large-scale data storage and annotation, break through the bottleneck of isolated learning, reduce the cost of model training and maintenance, and improve the intelligence of the model.

[0145] The multimodal incremental panoramic perception method of remote sensing images proposed in the present invention is inspired by the multimodal cognitive model of human beings, combined with the superior learning ability of artificial neural networks, and unifies cross-modal multi-task learning into a single model framework to achieve multi-level and multi-modal continuous interpretation of remote sensing images. Compared with the current remote sensing image interpretation method, the present invention has the following advantages: 1) The incremental learning model is extended from a single model to a multi-task model, which can support multi-task and cross-modal panoramic perception of remote sensing images, and realize multi-level interpretation capabilities from pixel level, instance level to image level; 2) It can realize the coordinated optimization of cross-modal features of text and image, and at the same time improve the interpretation efficiency of each single-modal task; 3) There is no need to retain old data, only use the old model to autonomously generate old knowledge, and realize the migration of old knowledge to the new model. In addition, the method of the present invention can better understand and imitate the learning mode and cognitive mechanism of the human brain from a system level, and realize the multi-level, multi-modal and adaptive perception ability of remote sensing image content.

[0146] Furthermore, the present invention also provides a remote sensing image multimodal incremental panoramic perception system, which is applied to a remote sensing image multimodal incremental panoramic perception method in the above embodiment, performs multi-level and multimodal continuous interpretation of remote sensing images, and completes cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images. The system includes:

[0147] A model building module, used to build a multimodal panoramic perception model of remote sensing images based on incremental learning; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal general image encoder module, a cross-modal interactive knowledge distillation module, a task bias pseudo-annotation generation module and a symmetric query interactive collaborative classification module;

[0148] A model training module, used for performing incremental training on the constructed multimodal panoramic perception model of remote sensing images;

[0149] The model application module is used to input remote sensing image data into the trained multimodal panoramic perception model of remote sensing images, perform multi-level and multimodal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images.

[0150] The implementation principle and technical effects of the system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0151] In addition, an embodiment of the present invention also provides a storage medium on which one or more programs readable by a computing device are stored, and the one or more programs include instructions, which, when executed by the computing device, enable the computing device to execute a multimodal incremental panoramic perception method for remote sensing images in the above-mentioned embodiment.

[0152] In the embodiment of the present invention, the storage medium may be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination thereof.

[0153] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0154] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0155] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing image multi-modal incremental panoramic perception method, characterized in that: The method comprises the following steps: S1. Constructing a multimodal panoramic perception model of remote sensing images based on incremental learning; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal general image encoder module, a cross-modal interactive knowledge distillation module, a task bias pseudo-annotation generation module and a symmetric query interactive collaborative classification module; S2. performing incremental training on the constructed multimodal panoramic perception model of remote sensing images; S3. Input the remote sensing image data into the trained multimodal panoramic perception model of remote sensing images, perform multi-level and multimodal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images.

2. A remote sensing image multi-modal incremental panoramic perception method according to claim 1, characterized in that: In the cross-modal universal image encoder module, the image features are divided into mask feature embedding and text feature embedding by sharing an encoder, wherein: the mask feature embedding is used for image segmentation tasks, and the text feature embedding is used for image description tasks, specifically: For the input image x at step t, the image feature f is defined as: f=Θ(ReLU(Θ(E(x))))+E(x) In the formula, Θ(·) represents linear mapping, ReLU represents activation function, and E(·) represents image feature encoder; After obtaining the image feature f, a Transformer decoder is used to generate an image mask, and another Transformer decoder is used to generate a text description. Specifically, semantic segmentation and instance segmentation are unified into the image segmentation task, and the image feature f is used to generate N mask feature embeddings. For the image description task, the image feature f is used to generate text feature embeddings.

3. A remote sensing image multi-modal incremental panoramic perception method according to claim 2, characterized in that: The cross-modal interactive knowledge distillation module includes two parts: intermediate layer contrast distillation and cross-guided instance distillation, where: In the middle layer contrast distillation part, the features output by the cross-modal universal image encoder module are used as input to integrate the learning tasks of the two modalities of image segmentation and image description into the contrastive learning framework. The calculation method is: Where, L ICD represents the intermediate layer contrast distillation loss, d(·) represents the similarity metric function, D CL (·) represents contrastive learning loss, |·| represents absolute value operation, and I(·) is the indicator function; and Respectively represent the t-1 step model M t-1 and the t-th step model M t The output segmentation mask, and Respectively represent the t-1 step model M t-1 and the t-th step model M t Output text description result, M t-1 and M t They represent the model of step t-1 and the model trained in step t respectively; and Respectively represent anchor feature, positive sample feature and negative sample feature; C 0:t-1 Indicates the category that has been learned at step t-1; C 0:t Indicates the category that has been learned at step t, i, j indicates that it belongs to C 0:t The category number, k means it belongs to C 0:t-1 The category number of In the cross-guided instance distillation part, an adaptive weight mechanism is used to improve the discrimination ability of fine-grained target instances. Specifically, the quality score q is constructed. r , which is used to guide the output of the new model in the distillation process to be closer to the output of the old model, so as to reduce the catastrophic forgetting of old knowledge; the quality score q r The calculation method is: In the formula, c r Represents the t-step model M t The output is the category prediction result for instance r, γ represents the parameter used to control the weight of classification and segmentation tasks, Represents the t-1 step model M t-1 The segmentation result of the output instance r is The features of the model output at step t are expressed by M t-1 The decoder outputs the segmentation result of instance r, and IoU represents the intersection over union calculation.

4. The remote sensing image multi-modal incremental panoramic perception method according to claim 3, characterized in that: The optimization goal of cross-guided instance distillation is: Where, L CID represents the cross-guided instance distillation loss, and They represent the prediction results of the model output at the t-1th step and the prediction results of the model output at the tth step, respectively, and d(·) represents the similarity measurement function.

5. A remote sensing image multi-modal incremental panoramic perception method according to claim 4, characterized in that: The optimization goal of the cross-modal interactive knowledge distillation module is to comprehensively calculate the intermediate layer contrast distillation loss and the cross-guided instance distillation loss: L TKD =L ICD +L CID Where, L TKD represents the knowledge distillation loss function.

6. A remote sensing image multi-modal incremental panoramic perception method according to claim 5, characterized in that: In the task bias pseudo-annotation generation module, for the image segmentation task, for any pixel i in the image x, its annotation calculation method in the tth step is: In the formula, represents the generated annotation for pixel i during the t-th step of training, Represents the t-1 step model M t-1 The output prediction result for pixel i, x i represents pixel i, c in image x b represents the background category, C 0:t-1 represents the category that has been learned at step t-1, C t represents the incremental category of the t-th step training, Γ is the threshold used to control the confidence of the generated pseudo-annotation, and ζ is a positive constant used to measure the stability of the model prediction results; Represents the output text description result of image x; For the image description task, at step t, the text description is annotated The calculation method is: in Represents the output text description of image x, represents the manually annotated text description that can be obtained in step t, Represents a concatenation operation.

7. A remote sensing image multi-modal incremental panoramic perception method according to claim 6, characterized in that: In the symmetric query interactive collaborative classification module, the mask feature Q output by the cross-modal universal image encoder is m and text features Q t , two symmetrical interactive attention models are used to perform feature fusion interaction, and the calculation method is: Q′ m =ψ(Q m ,Q t ) Q′ t =ψ(Q t ,Q m ) In the formula, ψ represents the interactive attention calculation process, Q′ m represents the fused mask embedding, Q′ t Represents the fused text embedding.

8. The remote sensing image multi-modal incremental panoramic perception method according to claim 7, characterized in that: The constructed multimodal panoramic perception model of remote sensing images supports end-to-end training, and the optimization goal is to comprehensively minimize the pixel classification loss L cls , image segmentation loss L seg , image description loss L cap and knowledge distillation loss L CIKD , calculated as: THE total =L cls +λL seg +L cap +L CIKD Where, L total represents the joint loss value, λ represents the weight coefficient of image segmentation loss; Where: Image classification loss L cls Defined using cross entropy loss: In the formula, p i Represents the model's prediction result for pixel i, y i Indicates the marked value; Image segmentation loss L seg The calculation method is: Where I(·) is the indicator function, represents the label corresponding to category c, represents the empty set, L focal represents focal loss, L dice represents dice loss, η1 and η2 represent weight coefficients, represents the model prediction value, represents the annotation value of step t; Image description loss L cap The calculation method is: In the formula, p cap Represents the image description prediction value output by the model, represents the text description annotation of the tth step, and L represents the length of the text description sentence.

9. A remote sensing image multi-modal incremental panoramic perception method according to claim 8, characterized in that: The constructed multimodal panoramic perception model of remote sensing images is incrementally trained. The training process includes: ①、The t-1 step model M t-1 The parameters of the training dataset D are frozen using the t-th step training dataset D t , train and get the t-step model M t Parameters; ② For the image x∈D used in the t-th step training t , first feed it into the cross-modal universal image encoder module to generate image features f, and then feed f into the Transformer decoder to generate mask feature embedding and text feature embedding; ③、The t-1 step model M t-1 Generated mask feature embedding and text feature embedding, model M at step t t The generated mask feature embedding and text feature embedding are simultaneously fed into the cross-modal interactive knowledge distillation module to calculate the knowledge distillation loss function L CIKD ; ④. Using the t-1 step model M t-1 The prediction results generate pseudo annotations, which are combined with the real annotations obtained from the training dataset in step t to generate the model M in step t t All annotations required for training; ⑤. Based on the fusion annotation obtained in step ④, calculate the t-th step model M t The pixel classification loss L of the output cls , image segmentation loss L seg and image description loss L cap , and the knowledge distillation loss function L obtained in step ③ CIKD Conduct joint optimization; ⑥ Repeat steps ② to ⑤, iteratively train the model until convergence, and obtain the model M of the tth step t , used to interpret the newly added semantics in step t.

10. A remote sensing image multi-modal incremental panoramic perception system, characterized in that: The method for multimodal panoramic perception of remote sensing images based on incremental learning as described in any one of claims 1 to 9 is applied to perform multi-level and multi-modal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images. The system comprises: A model building module, used to build a multimodal panoramic perception model of remote sensing images based on incremental learning; the constructed multimodal panoramic perception model of remote sensing images includes: a cross-modal general image encoder module, a cross-modal interactive knowledge distillation module, a task bias pseudo-annotation generation module and a symmetric query interactive collaborative classification module; A model training module, used for performing incremental training on the constructed multimodal panoramic perception model of remote sensing images; The model application module is used to input remote sensing image data into the trained multimodal panoramic perception model of remote sensing images, perform multi-level and multimodal continuous interpretation of remote sensing images, and complete cross-modal multi-tasks of pixel-level classification, instance-level segmentation and image-level description of remote sensing images.

Images