Change general model construction method for multi-task interpretation of remote sensing multi-temporal scene

Abstract

The invention belongs to the technical field of crossing of remote sensing and computer vision, and particularly relates to a change general model construction method for multi-task interpretation of a remote sensing multi-temporal scene. Comprising the steps that a CNN encoder and a Vi T encoder are adopted for encoding, and a double-time-phase visual token is generated by an input double-time-phase image; the LoRA technology is adopted to introduce learnable parameters on the basis of the frozen multi-modal large model to achieve visual language alignment in the remote sensing change field, and the multi-modal large model is further used for generating text output containing multi-modal information for input double-time-phase images and text instructions; a visual decoder is added for performing high-dimensional decoding on the obtained double-time-phase visual token and the obtained text output to output a change graph; adding learnable parameters by adopting an Adapter fine tuning mode according to the constructed variable general model, and realizing a multi-task unified processing function of the variable general model; and continuously training based on the text generation loss and the segmentation mask loss, and optimizing the change general model.

Description

Technical Field

[0001] The present invention belongs to the field of intersection of remote sensing and computer vision technology, and specifically relates to a method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes. Background Art

[0002] Changes in the Earth's surface, as a dynamic indicator of the Earth system, profoundly influence the evolution of the Earth and human survival. Observing and analyzing these changes is crucial for promoting sustainable human development. Remote sensing satellite imagery, with its unique "panoramic perspective," has become an effective tool for monitoring Earth's dynamic changes. The interpretation of changes in remote sensing imagery has become an important research focus. Intelligent interpretation of changes in multi-temporal remote sensing imagery aims to identify and describe changes reflected in images taken at different times within the same geographic area, providing decision makers with temporal and spatial information support. Intelligent change interpretation is a key tool for environmental analysis and monitoring, widely used in environmental protection, urban planning, resource management, military reconnaissance, and other fields.

[0003] Existing intelligent change interpretation algorithms are primarily categorized into two types: change detection algorithms and change description algorithms. Change detection algorithms can accurately locate the spatial location of surface changes with pixel-level accuracy. Typically, change detection employs a segmentation network with an encoder-decoder architecture to encode features in bi-temporal images, then refines the encoded features for decoding to obtain a change map. Change description, on the other hand, aims to describe the attributes and meaning of changes using natural language, emphasizing understanding the semantics of change. Change description algorithms involve the collaborative work of remote sensing image processing models and natural language generation models. While these algorithms have demonstrated effectiveness and achieved advanced performance in remote sensing intelligent interpretation tasks, they typically employ a one-model-per-task approach. This characteristic results in separate models, resulting in information separation and hindering the multi-task processing capabilities and open-world change analysis capabilities of intelligent change interpretation.

[0004] Therefore, it is necessary to develop a universal multi-task interpretation model for remote sensing multi-temporal scenes to unify multiple intelligent change interpretation tasks. This will help improve the efficiency and accuracy of remote sensing image change interpretation and provide more comprehensive and in-depth support for related fields. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a general change model construction method for multi-task interpretation of remote sensing multi-temporal scenes, which is used to solve the problems of each model being a system and information being separated in the existing technology, and the inability to unify multiple tasks such as change detection, change description, change visual question answering, single-temporal road / building extraction, and single-temporal image description.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for constructing a general change model for multi-task interpretation of remote sensing multi-temporal scenes includes the following steps:

[0008] Visual interaction understanding: CNN encoder and ViT encoder are used to generate bi-temporal visual tokens from the input bi-temporal images;

[0009] Cross-modal enhancement: LoRa technology is used to introduce learnable parameters on top of a frozen multimodal large model to achieve visual language alignment in the field of remote sensing changes. The multimodal large model is also used to generate text output containing multimodal information based on the input of bi-temporal images and text instructions.

[0010] Visual decoding: Add a visual decoder to perform high-dimensional decoding of the obtained bi-temporal visual tokens and the obtained text output to output a change map;

[0011] Multi-task unified perception: Targeting the building of a universal model for visual interaction understanding, cross-modal enhancement, and visual decoding, we use adapter fine-tuning to add learnable parameters to achieve unified multi-task processing within the universal model.

[0012] General variation model optimization: Continuous training based on text generation loss and segmentation mask loss optimizes the general variation model.

[0013] Optionally, the CNN encoder of the present invention is: a CNN encoder pre-trained by CLIP, the input of the CNN encoder is a dual-phase image I1I2, and the image I1 is passed through the CNN encoder to obtain multi-scale visual features And it is converted to the same spatial dimension through upsampling and spliced ​​in the channel dimension; image I2 is passed through the CNN encoder to obtain multi-scale visual features. And convert them to the same spatial dimension through upsampling and concatenate them in the channel dimension.

[0014] Optionally, the ViT encoder of the present invention is: a ViT encoder pre-trained by MAE, the input of the ViT encoder is a dual-phase image I1I2, and the image I1 is passed through the ViT encoder to obtain multi-level features Splicing in the channel dimension; image I2 passes through the ViT encoder to obtain multi-level visual features Stitching in the channel dimension.

[0015] Optionally, the present invention aligns the CNN visual features and ViT visual features output by the CNN encoder and the ViT encoder in the spatial dimension and splices them in the channel dimension.

[0016] Optionally, the present invention uses LoRA technology to introduce learnable parameters based on the frozen multimodal large model, specifically including the following steps:

[0017] Step 2.1 Model decomposition: Divide the multimodal large language model into multiple Transformer blocks;

[0018] Step 2.2 LoRA adaptation: Freeze the entire multimodal large language model weight matrix, and use LoRA technology to inject two learnable rank decomposition matrices into the multi-head attention of each Transformer block.

[0019] Optionally, the present invention assumes that for the mth Transformer block, the multi-head attention MAtt m And introduced the adaptive multi-head attention LoRA_MAtt with learnable rank decomposition matrix m Respectively expressed as:

[0020]

[0021] Where H represents the number of attention heads. In the m-th Transformer block, is the weight matrix of the h∈[0,H]th attention head, represents a learnable low-rank weight matrix, and d represents a scaling constant.

[0022] Optionally, the visual decoding of the present invention includes the following steps:

[0023] Extract the multimodal embedding P from the penultimate layer of the multimodal large language model optimized by LoRA and the text output prediction in <mask>The multimodal embedding P corresponding to the token seg , and the bi-temporal visual token V output in step 1 p Together they are fed into the visual decoder F dec , to obtain pixel-level change detection mask described <mask>Token is a special character introduced to distinguish the corresponding change detection mask from other text

[0024] Optionally, the specific process of adding learnable parameters by using the adapter fine-tuning method described in the present invention is as follows:

[0025] All parameters in the change general model except the LoRA learnable parameters and visual decoder are frozen, and adapter fine-tuning is used to unify the change general model for multi-temporal change detection, multi-temporal change description, multi-temporal change visual question answering, single-temporal image description, single-temporal building / road extraction, and single-temporal visual question answering tasks.

[0026] Optionally, the specific process of fine-tuning the adapter according to the present invention is as follows:

[0027] In the variational general model, two new learnable parameters are inserted into all linear layers: the bias parameter β and the offset parameter α. For the linear layer output y(x) = Wx, the bias parameter β is initially set to 0, and the offset parameter α is initialized to a random Gaussian distribution. The conversion is performed by combining the bias factor and the offset factor:

[0028] u(x)=α·(Wx+β)

[0029] α~N(μ,σ 2 ), β=Init(0)

[0030] Among them, W represents the original parameter matrix of the linear function, N(μ, σ 2 ) is Gaussian distribution, Init() is Init function.

[0031] Optionally, when the present invention continuously trains based on text generation loss and segmentation mask loss, the training loss function is:

[0032] L=λ txt L txt +λ mask L mask

[0033] Among them, L txt To introduce text generation loss, L mask is the segmentation mask loss, λ txt and λ mask Used to weighted balance text generation loss and segmentation mask loss.

[0034] Beneficial effects:

[0035] This paper proposes a method for constructing a universal change model for multi-task interpretation of multi-temporal remote sensing scenes. This method can understand and uniformly handle multiple change interpretation-related tasks, such as change detection, change description, change visual question answering, single-temporal road / building extraction, and single-temporal image description. Technically, we have constructed a novel visual interaction understanding method and designed cross-modal enhancements to adapt it to the field of remote sensing change interpretation. This method uses visual decoding to obtain spatial change information and utilizes a unified multi-task perception stage to achieve a universal paradigm for intelligent interpretation of remote sensing changes. We also continuously optimize and enhance its open-world intelligent interaction capabilities and multi-task compliance capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart for constructing a general model for the present invention;

[0038] Figure 2 This is a framework diagram of the method for constructing a universal model for changing the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of this disclosure.

[0041] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0042] like Figure 1-2 As shown, the embodiment of the present application provides a method for constructing a general change model for multi-task interpretation of remote sensing multi-temporal scenes. The following step sequence is not the only sequence of the embodiment of the present invention. For the sake of convenience, this embodiment adopts the following execution sequence description. The specific process is as follows:

[0043] Step 1, visual interaction understanding: Use CNN encoder and ViT encoder to encode the bi-temporal image I1I2 to obtain the bi-temporal visual token V p .

[0044] This embodiment provides a specific implementation method, including the following steps:

[0045] Step 1.1 Multi-scale Modeling: Perform multi-scale feature extraction on the bi-temporal image using a CNN encoder such as ResNet50×16 or ConvNeXt pre-trained with CLIP. The input of the CNN encoder is the bi-temporal image I1I2.

[0046] Image I1 passes through the CNN encoder to obtain multi-scale visual features And convert to the same spatial dimension by upsampling, and concatenate in the channel dimension:

[0047]

[0048] Image I2 passes through the CNN encoder to obtain multi-scale visual features And convert to the same spatial dimension by upsampling, and concatenate in the channel dimension:

[0049]

[0050] in, Represents the visual features extracted by the first coding layer in the image I1CNN encoder, ConvBlock1() represents the first coding layer algorithm in the CNN encoder, Represents the visual features extracted by the jth encoding layer in the image I1CNN encoder, ConvBlock j () represents the j-th encoding layer algorithm in the CNN encoder, V cnn,1 Represents the CNN visual features extracted by the CNN encoder for image I1, represents the visual features extracted by the first encoding layer in the CNN encoder for image I2, Denotes the visual features extracted by the jth encoding layer in CNN for image I2, V cnn,2 Represents the CNN visual features extracted by the CNN encoder for image I2, and Concat[] represents the splicing process.

[0051] Step 1.2 Multi-level modeling: Perform multi-level feature extraction on the bi-temporal image using a ViT encoder such as ViT-S, ViT-H, or ViT-G pre-trained with MAE. The input of the ViT encoder is the bi-temporal image I1I2.

[0052] Image I1 passes through the ViT encoder to obtain multi-level features Concatenate in the channel dimension:

[0053]

[0054] Image I2 passes through the ViT encoder to obtain multi-level visual features Concatenate in the channel dimension:

[0055]

[0056] in, represents the visual features extracted by the first coding layer in the ViT encoder for image I1, ViTBlock1() represents the first coding layer algorithm in the ViT encoder, Represents the visual features extracted by the i-th encoding layer in the image I1, ViT encoder, ViTBlock i () represents the i-th coding layer algorithm in the ViT encoder, V vit,1 represents the ViT visual features extracted by the ViT encoder for image I1, represents the visual features extracted by the first coding layer in the ViT encoder for image I2, Represents the visual features extracted from the i-th coding layer in image I2, ViT, V vit,2 Represents the ViT visual features extracted by the ViT encoder for image I2, and Concat[] represents the splicing process.

[0057] Step 1.3 Visual feature alignment: Align the CNN multi-scale visual features and ViT multi-level visual features output by the CNN encoder and ViT encoder in the spatial dimension and concatenate them in the channel dimension. The visual token after dimension alignment is represented as V p,1 ,V p,2 Then the visual tokens are concatenated in the channel dimension to obtain the bi-temporal visual token V p The operation is as follows:

[0058] V p,1 =Concat[V cnn,1 , V vit,1 ]

[0059] V p,2 =Concat[V cnn,2 , V vit,2 ]

[0060] V p =Concat[V p,1 , V p,2 ]

[0061] Step 2, cross-modal enhancement: LoRA technology is used to introduce learnable parameters on the basis of the frozen multimodal large language model to achieve visual language alignment in the field of remote sensing changes, and then encode the dual-phase images I1, I2 and the text command T to generate a text output containing multimodal information

[0062] Existing large multimodal language models for natural scenes lack domain knowledge for remote sensing change interpretation, and the generated text output is not accurate enough, making it difficult to apply to actual change interpretation scenarios.

[0063] To address this issue, LoRA technology introduces learnable parameters to a frozen large multimodal language model for natural scenes, enabling visual-language alignment training for remote sensing variations. The learnable parameters introduced by LoRA imbue the large multimodal language model with specialized knowledge for intelligent interpretation of remote sensing variations, while avoiding the traditional, expensive full-parameter fine-tuning and overfitting risks.

[0064] This embodiment provides a specific implementation method, including the following steps:

[0065] Step 2.1 Model decomposition: Select Qwen-VL-Chat as the basic multimodal large language model and divide the multimodal large language model into multiple Transformer blocks.

[0066] Step 2.2 LoRA Adaptation: LoRA technology freezes all weight matrices of Qwen-VL-Chat during training and injects the learnable rank decomposition matrix into the Transformer block, thereby introducing domain-specific knowledge of remote sensing changes.

[0067] For the m-th Transformer block of Qwen-VL-Chat, multi-head attention MAtt m The calculations are as follows:

[0068]

[0069] Where H represents the number of attention heads. In the mth Transformer block, the weight matrix of the h∈[0,H]th attention head is:

[0070] In the mth layer Transformer block, after LoRA adaptation, the multi-head attention with learnable low-rank weight matrix is ​​introduced with LoRA_MAtt m It is defined as follows:

[0071]

[0072] In general, this process starts with the frozen Qwen-VL-Chat as the starting point, and then achieves visual and language alignment in the field of remote sensing change by optimizing two smaller matrices, adapting knowledge to the field of remote sensing change interpretation.

[0073] Step 2.3 Multimodal modeling: Input the dual-phase images I1, I2 and the text command T into the LoRA-adapted Qwen-VL-Chat to generate a text output containing multimodal information.

[0074]

[0075] Where MLLM represents the multimodal large language model Qwen-VL-Chat.

[0076] Step 3: Visual decoding: Add a visual decoder to the bi-temporal visual token V obtained in step 1 p and the text output obtained in step 2 Perform high-dimensional decoding and output a change map.

[0077] This embodiment provides a specific implementation method, including the following steps:

[0078] Extract the multimodal embedding P from the penultimate layer of Qwen-VL-Chat after LoRA optimization in step 2 and the text prediction in <mask>The multimodal embedding P corresponding to the token seg , and the bi-temporal visual token V output in step 1 p Together they are fed into the visual decoder F composed of Uperhead dec , get pixel-level change detection mask Intuitively obtain the changes of buildings and roads in multi-temporal images, <mask>Token is a special character introduced to distinguish the corresponding change detection mask from other text The specific expressions are as follows:

[0079]

[0080] Step 4: Multi-task unified perception: Use the adapter fine-tuning method to add learnable parameters to achieve the multi-task unified processing function of the changing general model.

[0081] In this step, all parameters in the variation general model except the LoRA learnable parameters and the visual decoder are frozen. Adapter fine-tuning is used to unify the variation general model for tasks such as multi-temporal change detection, multi-temporal change description, multi-temporal change visual question answering, single-temporal image description, single-temporal building / road extraction, and single-temporal visual question answering.

[0082] This embodiment provides a specific implementation method, including the following steps:

[0083] The general model involves a CNN encoder, a ViT decoder, and a multimodal large model; these three parts contain a large number of linear layers. Two new learnable parameters are inserted into all linear layers in the model: the bias parameter β and the offset parameter α.

[0084] For the linear layer output y(x)=Wx, the bias parameter β is initialized to 0, the offset parameter α is initialized to a random Gaussian distribution, and the conversion is performed by combining the bias factor and the offset factor:

[0085] y(x)=α·(Wx+β)

[0086] α~N(μ,σ 2 ), β=Init(0)

[0087] Among them, W represents the original parameter matrix of the linear function, N(μ, σ 2 ) is Gaussian distribution, Init() is init function.

[0088] After fine-tuning with the adapter, the universal model has the ability to perform unified reasoning on multiple tasks, while also enhancing its ability to follow task instructions.

[0089] Step 5: Continue training through text loss and mask loss to optimize the general model.

[0090] The present invention provides a specific implementation method, which includes the following steps:

[0091] In order to continuously improve the intelligent interaction ability and multi-task following ability of the general model in the open world, the text generation loss L is introduced. txt and segmentation mask loss L mask The LoRA method in step 2 is tested on a large-scale synthetic / manually annotated multimodal instruction fine-tuning dataset in the field of remote sensing change.

[0092] The learnable parameters introduced by the Adapter method in step 5 and the visual decoder in step 4 are optimized and updated. The text loss function uses the autoregressive cross entropy loss function generated by the text, while the segmentation mask loss uses the cross entropy loss function and the Dice loss function. Given the manual annotation Y txt and mask M, the above loss function can be expressed as:

[0093] L=λ txt L txt +λ mask L mask

[0094]

[0095] where λ txt ,λ mask Used to weight the balance between text generation loss and segmentation mask loss. ce and λ dice Used to weighted balance the cross loss function CE() and the Dice loss function Dice().

[0096] Compared with the existing technology, this method has the following advantages:

[0097] This method enhances multitasking capabilities by integrating multiple learning processes. This avoids the complexity of training models for each task separately, as is often the case with traditional methods. This reduces repeated computations, improves efficiency and consistency, and maintains high interpretation accuracy. Furthermore, compared to other methods, this method enhances user-model interactivity through features like visual question answering, making it easier for non-expert users to obtain the required change information.

[0098] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description.

[0099] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / mask> < / mask> < / mask> < / mask>

Claims

1. A general change model construction method for multi-task interpretation of remote sensing multi-temporal scenes, characterized by: The following steps are involved: Visual interaction understanding: CNN encoder and ViT encoder are used to generate bi-temporal visual tokens from the input bi-temporal images; Cross-modal enhancement: LoRa technology is used to introduce learnable parameters on top of a frozen multimodal large model to achieve visual language alignment in the field of remote sensing changes. The multimodal large model is also used to generate text output containing multimodal information based on the input of bi-temporal images and text instructions. Visual decoding: Add a visual decoder to perform high-dimensional decoding of the obtained bi-temporal visual tokens and the obtained text output to output a change map; Multi-task unified perception: Targeting the building of a universal model for visual interaction understanding, cross-modal enhancement, and visual decoding, we use adapter fine-tuning to add learnable parameters to achieve unified multi-task processing within the universal model. General variation model optimization: Continuous training based on text generation loss and segmentation mask loss optimizes the general variation model.

2. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 1 is characterized in that: The CNN encoder is: a CNN encoder pre-trained by CLIP. The input of the CNN encoder is a dual-phase image I1I2. The image I1 is passed through the CNN encoder to obtain multi-scale visual features. And it is converted to the same spatial dimension through upsampling and spliced ​​in the channel dimension; image I2 is passed through the CNN encoder to obtain multi-scale visual features. And convert them to the same spatial dimension through upsampling and concatenate them in the channel dimension.

3. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 2, characterized in that: The ViT encoder is: a ViT encoder pre-trained by MAE. The input of the ViT encoder is a bi-phase image I1I2. The image I1 is passed through the ViT encoder to obtain multi-level features. Splicing in the channel dimension; image I2 passes through the ViT encoder to obtain multi-level visual features Stitching in the channel dimension.

4. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 3 is characterized in that: The CNN visual features and ViT visual features output by the CNN encoder and the ViT encoder are aligned in the spatial dimension and spliced ​​in the channel dimension.

5. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 1, characterized in that: The LoRA technology is used to introduce learnable parameters based on the frozen multimodal large model, specifically including the following steps: Step 2.1 Model decomposition: Divide the multimodal large language model into multiple Transformer blocks; Step 2.2 LoRA adaptation: Freeze the entire multimodal large language model weight matrix, and use LoRA technology to inject two learnable rank decomposition matrices into the multi-head attention of each Transformer block.

6. The method for constructing a universal model of change for multi-task interpretation of remote sensing multi-temporal scenes according to claim 5 is characterized in that For the mth Transformer block, multi-head attention MAtt m And introduced the adaptive multi-head attention LoRA_MAtt with learnable rank decomposition matrix m Respectively expressed as: Where H represents the number of attention heads. In the m-th Transformer block, is the weight matrix of the h∈[0,H]th attention head, represents a learnable low-rank weight matrix, and d represents a scaling constant.

7. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 1, characterized in that: The visual decoding includes the following processes: Extract the multimodal embedding P from the penultimate layer of the multimodal large language model optimized by LoRA and predict the text output in <mask>The multimodal embedding P corresponding to the token seg , and the bi-temporal visual token V p Together they are fed into the visual decoder F dec , to obtain pixel-level change detection mask described <mask>Token is a special character introduced to distinguish the corresponding change detection mask from other text < / mask> < / mask> 8. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 1, characterized in that: The specific process of adding learnable parameters using Adapter fine-tuning is as follows: All parameters in the change general model except the LoRA learnable parameters and visual decoder are frozen, and adapter fine-tuning is used to unify the change general model for multi-temporal change detection, multi-temporal change description, multi-temporal change visual question answering, single-temporal image description, single-temporal building / road extraction, and single-temporal visual question answering tasks.

9. The method for constructing a universal change model for multi-task interpretation of remote sensing multi-temporal scenes according to claim 8, characterized in that: The specific process of the Adapter fine-tuning is as follows: In the variational general model, two new learnable parameters are inserted into all linear layers: the bias parameter β and the offset parameter α. For the linear layer output y(x) = Wx, the bias parameter β is initially set to 0, and the offset parameter α is initialized to a random Gaussian distribution. The conversion is performed by combining the bias factor and the offset factor: y(x)=α·(Wx+β) a~N(μ,σ 2 ),β=Init(0) Among them, W represents the original parameter matrix of the linear function, N(μ, σ 2 ) is Gaussian distribution, Init() is Init function.

10. The method for constructing a universal change model for multi-task interpretation of multi-temporal remote sensing scenes according to claim 1, characterized in that: When the training is continued based on the text generation loss and the segmentation mask loss, the training loss function is: L=λ txt L txt +λ mask L mask Among them, L txt To introduce text generation loss, L mask is the segmentation mask loss, λ txt and λ mask Used to weighted balance text generation loss and segmentation mask loss.

Images