Semantic description and question-answering system establishment method and system for remote sensing disaster detection

Abstract

The invention relates to the field of computer vision language and image recognition, in particular to a remote sensing disaster detection-oriented semantic description and question-answering system establishment method and system.The method comprises the steps of generating text language description capable of accurately responding to task demand images on the premise of adding text constraints through model establishment, training and evaluation and utilizing image features, and establishing a semantic description and question-answering system for remote sensing disaster detection. On the basis of establishing a mapping relation between a pixel-level segmentation result and a question-answering system, high-level semantic information is extracted, the question-answering system is established, answers for disaster inquiry questions can be accurately output, three sub-tasks of semantic segmentation, statement description and the question-answering system can be completed by using one set of network model, and the method is simple and convenient to operate. The computer memory space is saved, the response speed is improved, and technical support is provided for improving the automation degree, the intelligence degree and the quick response capability of a remote sensing disaster detection system.

Description

Technical Field

[0001] The present invention relates to the field of computer vision language and image recognition, and in particular to a method and system for establishing a semantic description and question-answering system for remote sensing disaster detection. Background Art

[0002] Visual language problem is a cross-disciplinary problem that combines computer vision with natural language processing. The image-to-text generation task represented by sentence description and visual question answering is an important visual language cross-modal task. Its purpose is to use image features to generate a textual language description of the image under the premise of adding possible text constraints. In order to generate text that can accurately respond to task requirements, the model needs to understand and analyze the relationship between features while accurately extracting key features of the image, and extract potential high-level semantic information based on text constraints.

[0003] Image-text generation models can be divided into two types: single-task and multi-task pre-trained models. Single-task models usually only use one text generation task, such as sentence description or visual question answering, as the optimization target and use a single-task dataset for training. In recent years, with the rapid development of datasets and GPU computing power, multi-task visual language pre-training (VLP) models have emerged. By pre-training a large model with a general structure, the trained model adjusts its structure and enters the downstream task, performing parameter fine-tuning on a specific dataset.

[0004] Remote sensing images have the advantages of high resolution, multi-spectral, and large range, and are widely used and highly efficient. Through the change detection and processing methods of remote sensing images, the complex information in disaster remote sensing images can be sorted out, and the disaster information required by experts can be obtained, which is of great significance for assessing the extent of natural disasters such as floods and earthquakes. At present, semantic segmentation and change detection are commonly used on-site to process remote sensing images, using pixel-level or regional-level detection methods and outputting them as images. Due to the lack of high-level semantic information extraction and expression capabilities, it is impossible to detect the affected areas and semantically describe the information of key areas; it is also impossible to output answers to the questions raised about the disaster situation. Obviously, the existing detection methods still require professionals to make judgments based on the output images in order to give an analysis of the disaster situation. The detection methods have low automation and intelligence levels, which affects the rapid and accurate judgment of the extent of disasters in natural areas.

[0005] In recent years, the image text generation task has also received increasing attention in the field of remote sensing images due to its potential in disaster applications. Although technicians in the field have carried out many studies on remote sensing image text generation (application numbers; 202310339866.4, 202110982615.9, 201811584864.7), the studies only target one visual language task, or use target detection as an additional task to a visual language task. There is no network model that can effectively complete semantic segmentation, sentence description, and visual question answering multiple tasks.

[0006] In response to the problems in the above-mentioned technology, the present invention provides a method and system for establishing a semantic description and question-answering system for remote sensing disaster detection, which can use the same network model to complete the three subtasks of semantic segmentation, sentence description and question-answering system, and realize semantic segmentation of the disaster area while describing the disaster situation and establishing a question-answering system. It can accurately output answers to questions raised by users about the disaster situation, and provide technical support for improving the automation, intelligence and rapid response capabilities of remote sensing disaster detection systems. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for establishing a semantic description and question-answering system for remote sensing disaster detection, so as to solve the problems raised in the above-mentioned background technology.

[0008] A method for establishing a semantic description and question-answering system for remote sensing disaster detection, the method is used to solve three subtasks of semantic segmentation, sentence description and visual question-answering, and includes the following steps:

[0009] Step 1: Obtain a remote sensing image dataset of the disaster area and annotate it for network model training and evaluation, including annotating semantic segmentation data of the remote sensing image dataset to obtain remote sensing image segmentation maps of the disaster area, annotating sentence descriptions of the remote sensing image dataset to describe the situation in the disaster area, and annotating visual question and answer data of the remote sensing image dataset to obtain question and answer text of the disaster area.

[0010] Step 2: Based on the labeled disaster area remote sensing image dataset, obtain the network model parameters for solving the three subtasks by training the network model, including:

[0011] The classic semantic segmentation model is migrated to the semantic segmentation network model, and then semantic segmentation training is performed on the remote sensing image dataset with labeled semantic segmentation data to obtain the optimal semantic segmentation network model parameters; the semantic segmentation training adopts a common learning training method to divide the remote sensing image of the disaster area into small blocks, which are input into the semantic segmentation network model in batches to generate local area features, and then a local semantic segmentation map of the image is generated using the local area features, and finally a semantic segmentation result map is obtained by splicing.

[0012] The sentence description network model and the semantic segmentation network model are serialized, and then sentence description training is performed on the remote sensing image dataset labeled with sentence descriptions to obtain the optimal sentence description network model parameters, and the semantic segmentation model with the best performance in the semantic segmentation network model and sentence description network model training is screened out for subsequent visual question answering network model training, the optimal semantic segmentation network model parameters are frozen during the sentence description training process, and a training method combining ordinary learning and reinforcement learning is adopted to fuse the generated local area features to obtain global features, and finally the global features are decoded to obtain image text description;.

[0013] The visual question answering network model is added as a branch structure to the serial structure, and the screened semantic segmentation model is added to the visual question answering network model training, and then the visual question answering training is performed on the remote sensing image dataset labeled with visual question answering to obtain the optimal visual question answering network model parameters, the sentence description network model and the visual question answering network model are in a parallel relationship, and the sentence description is an upstream task of the visual question answering; during the visual question answering training process, the optimal semantic segmentation network model parameters and the optimal sentence description network model parameters are frozen, and a training method combining ordinary learning and reinforcement learning is adopted to generate local area features on the remote sensing image dataset, and then the prior knowledge accumulated in the process of generating sentence description text from the image is used to guide the generation of text features of the input question, and the image features and text features are cross-modally fused to obtain fused features, and finally the fused features are decoded to obtain the answer text of the question.

[0014] Step 3: Use the network model connection switching method to solve the three sub-tasks of semantic segmentation, sentence description and visual question answering. When it is necessary to solve the sentence description task, connect the semantic segmentation network model with the sentence description network model to obtain the image description text; when it is necessary to solve the visual question answering task, switch to the semantic segmentation network model and connect it with the visual question answering network model to obtain the question answering text.

[0015] Furthermore, the step of fusing the generated local area features to obtain global features includes: stacking all the generated local feature maps using a local feature joint coding structure and sending them to a flattening layer to obtain output features, and linearly projecting and position encoding the output features to obtain a global feature map.

[0016] Furthermore, the step of cross-modal fusing the image features and the text features includes: using a cross-modal fusing device to splice the image features and the text features after they have passed through n dual-path feature generation layers, inputting the joint attention layer for feature fusion, and outputting fused features; the cross-modal fusing device cross-modally fuses and encodes the image features and the text features based on the prior knowledge accumulated in the upstream sentence description; n is the number of dual-path features.

[0017] The two-way feature includes text self-attention, image self-attention and image-guided text attention, wherein the image self-attention uses the network model parameters obtained by training in the upstream sentence description, and the image-guided text attention process is:

[0018]

[0019] head=A(x i W Q ,YW K ,YW V ),

[0020] Z=(head 1 ,head 2 ,…head n )W O ,

[0021] O=FFN(Z)

[0022] in, is a text feature, is the image feature, is the projection matrix of multi-head attention, is the dimensionality reduction matrix of multi-head attention, and FFN is the feed-forward network.

[0023] Furthermore, the ordinary learning process uses cross entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses Beam Search algorithm to infer and generate word sequences, and uses sequence loss function to judge the quality of sequence generation.

[0024] Furthermore, the sequence loss function in the sentence description training process adopts CIDEr-D (w s ,w * ) is used as the sequence evaluation index, and the reinforcement learning loss function in the sentence description training process is:

[0025]

[0026] Among them, Y represents the predicted sequence; Z represents the corresponding target sequence; k is the total number of sequences in each batch; b is the baseline function, that is, the average of the reward function values ​​of all sequences in this batch; reward(w i ) is the reward function indicator; CIDEr-D(w s ,w * ) is a sequence evaluation index.

[0027] Furthermore, the sequence loss function in the visual question answering training uses WupLus Score as the sequence judgment indicator, and the WupLus Score is:

[0028]

[0029] Among them, c A ,c T is the sequence length, n a ,n tmax are the number of elements a in sequence A and the number of elements t in sequence T that are most similar to a.

[0030] The reinforcement learning loss function in the visual question answering training process is:

[0031]

[0032] Among them, c Y ,c Z represents the original length of the generated sequence and the corresponding target sequence without padding value; Y = {y 0 ,y 1 ,...,y T}, Z={z 0 ,z 1 ,...,z T} represents the sequence elements contained in the generated sequence and the corresponding target sequence; Represents the sequence element y t The number of in sequence Y; Represents the element y in the sequence Z t , the number of elements with the highest Wup Score.

[0033] A semantic description and question-answering system for remote sensing disaster detection, including:

[0034] Image data collection and annotation module, used for network model training and evaluation. The image data set collection and annotation module includes annotating semantic segmentation data of remote sensing image data sets to obtain remote sensing image segmentation maps of disaster areas, annotating sentence descriptions of remote sensing image data sets to describe the situation in disaster areas, and annotating visual question and answer of remote sensing image data sets to obtain question and answer texts of disaster areas.

[0035] A semantic segmentation network model module is used to solve the semantic segmentation task. The semantic segmentation network model module includes a remote sensing image input structure, an image segmentor, an image feature extraction model, and a segmentation head. The remote sensing image of the disaster area generates local image features through the image feature extraction model.

[0036] A sentence description network model module is used to solve the semantic segmentation task, and the sentence description network model module includes a local feature joint encoding structure and a text decoder; the local feature joint encoding structure is used to generate global image features from local image features, and the local feature joint encoding structure includes a flattening layer, a linear projection layer, and a position encoding layer;

[0037] A visual question answering task network model module is used to solve the visual question answering task. The visual question answering task network model module includes a question answering text input structure, a text encoder, a local feature joint encoding structure, a cross-modal fuser and the text decoder; the cross-modal fuser is used to fuse local image features and text features of the question to generate fusion features. The cross-modal fuser is an encoder based on the prior knowledge accumulated in the sentence description to guide the generation and encoding of fusion features. The cross-modal fuser includes n two-way feature generation layers and a joint attention layer.

[0038] The network model connection module is used to switch the network model connection to solve the three sub-tasks of semantic segmentation, sentence description and visual question and answer. When it is necessary to solve the sentence description task, the semantic segmentation network model is connected to the sentence description network model to obtain the description text of the image; when it is necessary to solve the visual question and answer task, it is switched to connect the semantic segmentation network model with the visual question and answer network model to obtain the question and answer text.

[0039] The network model training learning evaluation module is used to obtain the optimal network model parameters for solving the three subtasks. The training learning includes ordinary learning and reinforcement learning. The ordinary learning process uses cross entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses Beam Search algorithm to infer and generate word sequences, and uses sequence loss function to judge the quality of sequence generation; the sequence loss function in the visual question answering training uses WupLus Score as the sequence evaluation indicator.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) Designing a local feature joint encoding structure to extract image features solves the problem of how to efficiently and accurately extract global features for high-resolution and large-size images;

[0042] (2) Visual question answering is regarded as a constrained sentence description task. Semantic segmentation and sentence description are used as the allocation strategy of the upstream tasks of visual question answering. A visual language cross-modal fusion based on upstream accumulation is designed. The large amount of prior knowledge accumulated in sentence description is used to guide feature extraction and encoding, thereby reducing computing power and memory space. At the same time, the WupLusScore sequence evaluation indicator is proposed to solve the problem of punishment for cheating high scores by repeating words in the evaluation process and unreasonable reward setting due to insensitivity to numerical answers.

[0043] (3) By establishing a mapping relationship between pixel-level segmentation results and a question-answering system, high-level semantic information is extracted and a question-answering system is established, which can accurately output answers to disaster-related questions. The test results show that the design of the present invention can achieve good performance indicators in the three subtasks of semantic segmentation, sentence description index and visual question-answering. It can be seen that a set of network models designed by the present invention can complete the three subtasks of semantic segmentation, sentence description and question-answering system, saving computer memory space and improving response speed, providing technical support for improving the automation, intelligence and rapid response capabilities of remote sensing disaster detection systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Steps of establishing a semantic description and question-answering system for remote sensing disaster detection provided in the embodiment of the present application Figure 2 The network model framework structure provided for the embodiment of the present application

[0045] Figure 3 The cross-modal fusion structure provided in the embodiment of the present application

[0046] Figure 4 The miou index change curve for semantic segmentation training provided in the embodiment of the present application

[0047] Figure 5 CIDEr-D indicator change curve for sentence description training provided in the embodiment of this application

[0048] Figure 6 The Top-1 Accuracy indicator change curve for visual question answering training provided in the embodiment of this application

[0049] In the figure: 1-remote sensing image input structure; 2-question and answer text input structure; 3-image segmenter; 4-image feature extraction model; 5-segmentation head; 6-local feature joint encoding structure; 7-text encoder; 8-cross-modal fuser; 9-text decoder; 10-word sequence. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] like Figure 1 As shown, the steps of the method for establishing a semantic description and question-answering system for remote sensing disaster detection provided by the embodiment of the present application include:

[0052] S1: Obtain remote sensing image datasets of disaster areas and annotate them for network model training and evaluation, including annotating semantic segmentation data of remote sensing image datasets to obtain remote sensing image segmentation maps of disaster areas, annotating sentence descriptions of remote sensing image datasets to describe the conditions of disaster areas, and annotating visual question and answer data of remote sensing image datasets to obtain question and answer texts of disaster areas;

[0053] S2: Based on the labeled disaster area remote sensing image dataset, the network model parameters for solving three subtasks are obtained by training the network model, including:

[0054] The classic semantic segmentation model is migrated to the semantic segmentation network model, and then semantic segmentation training is performed on the remote sensing image data set with annotated semantic segmentation data to obtain the optimal semantic segmentation network model parameters; the semantic segmentation training adopts a common learning training method to divide the remote sensing image of the disaster area into small blocks, which are input into the semantic segmentation network model in batches to generate local area features, and then the local area features are used to generate a semantic segmentation map of the local area of ​​the image, and finally the semantic segmentation result map is obtained by splicing;

[0055] The sentence description network model and the semantic segmentation network model are serialized, and then sentence description training is performed on a remote sensing image dataset labeled with sentence descriptions to obtain optimal sentence description network model parameters, and the semantic segmentation model with the best performance in the semantic segmentation network model and sentence description network model training is screened out for subsequent visual question answering network model training, and the optimal semantic segmentation network model parameters are frozen during the sentence description training process, and a training method combining ordinary learning and reinforcement learning is adopted to perform feature fusion on the generated local area features to obtain global features, and finally the global features are decoded to obtain image text description;

[0056] The visual question answering network model is added as a branch structure to the serial structure, and the selected semantic segmentation model is added to the visual question answering network model training, and then the visual question answering training is performed on the remote sensing image dataset labeled with visual question answering to obtain the optimal visual question answering network model parameters, the sentence description network model and the visual question answering network model are in a parallel relationship, and the sentence description is an upstream task of the visual question answering; during the visual question answering training process, the optimal semantic segmentation network model parameters and the optimal sentence description network model parameters are frozen, and a training method combining ordinary learning and reinforcement learning is adopted to generate local area features on the remote sensing image dataset, and the prior knowledge accumulated in the process of generating sentence description text from the image is used to guide the generation of text features of the input question, and the image features and text features are cross-modally fused to obtain fused features, and finally the fused features are decoded to obtain the answer text of the question;

[0057] S3: Use network model connection switching to solve the three subtasks of semantic segmentation, sentence description and visual question answering. When it is necessary to solve the sentence description task, connect the semantic segmentation network model with the sentence description network model to obtain the image description text; when it is necessary to solve the visual question answering task, switch to the semantic segmentation network model and connect it with the visual question answering network model to obtain the question answering text.

[0058] This embodiment establishes a network model for solving three subtasks of semantic segmentation, sentence description and visual question answering. For the sentence description subtask, in view of the problem that high-resolution images require global features in sentence description and visual question answering, and the global features are subject to video memory conditions, local area features are fused to obtain global features, and finally the global features are decoded to obtain image text description; for the visual question answering subtask, the computing power and memory space are reduced, and the prior knowledge accumulated in the process of generating sentence description text from images is used to guide the generation of text features of the input question, and the image features and text features are cross-modally fused to obtain fused features, and finally the fused features are decoded to obtain the answer text of the question; the remote sensing image dataset of the disaster area annotated with semantic segmentation data, sentence description and visual question answering is used to train and evaluate the network model, and the parameters of the network model are optimized.

[0059] As another preferred embodiment of the present invention, step S1 includes the following:

[0060] 2343 disaster remote sensing images with a size of 4000*3000 were collected, and the images were semantically segmented and annotated to obtain image segmentation maps; the images were annotated with sentence descriptions to obtain sentences describing the situation in the disaster area; the images were annotated with visual question and answering to obtain questions and answers about the situation in the disaster area. At the same time, incorrectly annotated images were removed, and the sentence description dataset was supplemented to supplement the visual question and answer verification set and test set. Finally, a dataset consisting of 1405 training images and their text annotations, 436 verification images and their text annotations, 448 test images and their text annotations, and 7107 questions and answers was obtained.

[0061] like Figure 2 As shown, as another preferred embodiment of the present invention, step S2 includes the following: the network model includes an image data set collection and annotation module, a semantic segmentation network model module, a sentence description network model module, a visual question answering network model module and a switching module; the semantic segmentation network model module includes a remote sensing image input structure 1, an image segmentor 3, an image feature extraction model 4 and a segmentation head 5 for solving semantic segmentation tasks; the sentence description network model module includes a local feature joint encoding structure 6 and a text decoder 9 for solving sentence description tasks; the visual question answering task network model module includes a question answering text input structure 2, a text encoder 7, a local feature joint encoding structure 6, a cross-modal fusion device 8 and the text decoder 9 for solving visual question answering tasks;

[0062] The working principle is as follows: the disaster remote sensing image is divided into small blocks by the image segmenter 3, and sent to the corresponding image feature extraction model 4 in batches to generate local image features. The local image features are spliced ​​through one of the branches and the connected segmentation head 5 to produce a segmented image; when the subtask is a sentence description task, the local image features 5 are sent to the local feature joint encoding structure 6 through another branch to generate global image features for language description content, and the global image features are then used to generate corresponding description sentences through the text decoder 9; when the subtask is visual question answering, the question and answer text is input through the question and answer text input structure 2, and then encoded through the text encoder 7 to extract text features. The text features and the global image features are fused and encoded through the cross-modal fuser 8 to obtain the fused features, and the fused features are then used to obtain the answer to the question through the text decoder 9.

[0063] As another preferred embodiment of the present invention: the step of fusing the generated local area features to obtain global features includes: using the local feature joint coding structure 6 to stack all the generated local feature maps and send them to the flattening layer to obtain output features, and linearly projecting and position encoding the output features to obtain a global feature map.

[0064] In this embodiment, in order to solve the problem that remote sensing images require global features in sentence description and visual question answering tasks, but global features are restricted by video memory conditions, a local feature joint encoding structure 6 is designed, which imitates ViT to generate a global feature map based on the local features of the image, and the global feature map is then encoded and decoded to obtain the sentence description text.

[0065] like Figure 3 As shown, as another preferred embodiment of the present invention: the step of cross-modal fusion of image features and text features includes: using a cross-modal fusion device 8 to splice the image features and text features after passing through n dual-path feature generation layers, inputting a joint attention layer for feature fusion, and outputting fusion features; the cross-modal fusion device 8 cross-modally fuses and encodes the image features and text features based on the prior knowledge accumulated in the upstream sentence description; n is the number of dual-path features, and the dual-path features include text self-attention, image self-attention, and image-guided text attention, wherein the image self-attention uses the network model parameters trained in the upstream sentence description task, and the image-guided text attention process is:

[0066]

[0067] head=A(x i W Q ,YW K ,YW V ),

[0068] Z=(head 1 ,head 2 ,...head n )W O ,

[0069] O=FFN(Z)

[0070] in, is a text feature, is the image feature, is the projection matrix of multi-head attention, is the dimensionality reduction matrix of multi-head attention, and FFN is the feed-forward network.

[0071] In this embodiment, after extracting image features and text features, the network needs to cross-modally fuse the two features to promote information interaction. Since the image feature extraction part is pre-trained in the upstream task, its accumulated data set knowledge is much higher than the text feature extraction branch added later. At the same time, since the image exists in a two-dimensional space, the amount of information it contains is much larger than the text. Compared with existing designs, the image feature extraction model 4 of the present application can provide a large amount of prior knowledge, guide the generation of text features of the input question, and cross-modally fuse the image features and text features to obtain fused features, and finally decode the fused features to generate the answer text.

[0072] As another preferred embodiment of the present invention: the ordinary learning process uses cross entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses Beam Search algorithm to infer and generate word sequences, and uses a sequence loss function to judge the quality of sequence generation. The weights and sequence indicators generated based on Beam Search are introduced into the loss function, which is more in line with the actual generation situation.

[0073] As another preferred embodiment of the present invention: the sequence loss function in the sentence description task training process adopts CIDEr-D (w s ,w * ) is used as the sequence evaluation index, and the reinforcement learning loss function in the sentence description training process is:

[0074]

[0075] Among them, Y represents the predicted sequence; Z represents the corresponding target sequence; k is the total number of sequences in each batch; b is the baseline function, that is, the average of the reward function values ​​of all sequences in this batch; reward(w i ) is the reward function indicator; CIDEr-D(w s ,w * ) is a sequence evaluation index.

[0076] As another preferred embodiment of the present invention: the sequence loss function in the visual question answering task training uses WupLus Score as the sequence evaluation index, and the WupLus Score is:

[0077]

[0078] Among them, c A ,c T is the sequence length, n a ,n tmax are the number of elements a in sequence A and the number of elements t in sequence T that are most similar to a.

[0079] The reinforcement learning loss function in the visual question answering training process is:

[0080]

[0081] Among them, c Y ,c Z represents the original length of the generated sequence and the corresponding target sequence without padding value; Y = {y 0 ,y 1 ,...,y T}, Z={z 0 ,z 1 ,...,z T} represents the sequence elements contained in the generated sequence and the corresponding target sequence; Represents the sequence element y t The number of in sequence Y; Represents the element y in the sequence Z t , the number of elements with the highest Wup Score.

[0082] In this embodiment, for the existing sequence evaluation indicators, Wup Score can only measure the similarity between words; it is not sensitive to numerical answers; Wups Score lacks the penalty for cheating high scores with repeated words and is still insensitive to numerical answers. The WupLus Score designed in this application introduces sequence length penalty and repeated word penalty on the basis of existing sequence evaluation indicators, and redefines the measurement method for the similarity of numerical answers, which can solve the problem of unreasonable reward setting due to the penalty for cheating high scores with repeated words and insensitivity to numerical answers in the evaluation process.

[0083] As another preferred embodiment of the present invention: the training network model in step S2 specifically includes:

[0084] Evaluation indicators: For semantic segmentation tasks, miou is selected as the evaluation indicator; for sentence description tasks, BLEU, METEOR, ROUGE-L and CIDEr-D are selected as the evaluation indicators; for visual question answering tasks, Top-1 Accuracy is selected as the evaluation indicator; these evaluation indicators are commonly used indicators in various sub-fields, and the higher the score, the better the effect.

[0085] The following training is optimized using the SGD method. The initial learning rate is set to 0.01, and the learning rate adjustment strategy uses the poly method. After each round of training, the evaluation index is obtained by comparing and evaluating with the corresponding annotations on the remote sensing disaster dataset.

[0086] 1. Semantic segmentation network model training

[0087] Training program:

[0088] The seven classic semantic segmentation models, including UNet, DeepLabv3+, PSPNet, CCNet, FarSeg, STDC-Seg, and Swin Transformer+uperNet (Super for short), are migrated to the semantic segmentation task network model as the image feature extraction model 4, and trained on the remote sensing image dataset with labeled semantic segmentation data to obtain the optimal sentence description network model parameters. Ordinary learning training is adopted, cross entropy loss is used for optimization, and the mean intersection-over-union (MIOU) is used as the performance indicator.

[0089] Training results:

[0090] like Figure 4 As shown in the figure, after 5 rounds of normal learning training, the improvement of the miou index slowed down. After 35 rounds of training, the improvement of the miou index was very small. At the end of 38 rounds of training, the miou index reached its peak, and the network model parameters of the semantic segmentation task were optimal.

[0091] It can be seen from Table 5-1 that compared with the other six semantic segmentation models, the miou index obtained by using CCNet as the image feature extraction model 4 is the highest. However, in order to show its performance in subsequent sentence descriptions, PSPNet, CCNet, and STDC-Seg with relatively high indicators are used as image feature extraction models 4 to enter the training of subsequent sentence descriptions.

[0092] Table 5-1 Semantic segmentation index results (normal learning training 40 rounds)

[0093] category 1 iou2 iou4 iou5 6 iou7 iou8 9i miou UNet 0.0430 0.4259 0.6651 0.3390 0.6438 0.0036 0.3531 0.7517 0.4730 DeepLabv3+ 0.5949 0.7269 0.7622 0.5921 0.7741 0.4197 0.4995 0.8390 0.6246 PSPNet 0.6240 0.7743 0.7916 0.6763 0.7881 0.4846 0.5667 0.8470 0.6720 CCNet 0.6515 0.7801 0.8075 0.6797 0.7947 0.5426 0.5932 0.8684 0.6880 FarSeg 0.6417 0.7519 0.7760 0.5911 0.7724 0.3964 0.5573 0.8483 0.6418 STDC-Seg 0.6342 0.7755 0.8012 0.6408 0.7800 0.3771 0.5181 0.8578 0.6509 Super 0.6084 0.6736 0.7488 0.6426 0.7673 0.3610 0.4141 0.8417 0.6075

[0094] 2. Sentence description task network model training

[0095] Training program:

[0096] After the sentence description network model and the semantic segmentation network model are serialized, ordinary learning training is first used, and then reinforcement learning training is used to fine-tune the parameters. The Beam Search algorithm is used to select appropriate elements, and a three-layer Transformer is used for both the encoding layer and the semantic decoding layer. After the training, the optimal semantic segmentation task network model parameters are obtained, and the semantic segmentation model with the best performance in completing the two tasks of semantic segmentation and sentence description is selected as the image feature extraction model 4 to enter the subsequent visual question answering task network model training.

[0097] Training process and results:

[0098] As shown in Table 5-2, the indicator reaches its peak after 5 rounds of normal learning training. At this time, the sentence description indicator is obtained. From Table 5-2 combined with Table 5-1, it can be concluded that:

[0099] ① Compared with other semantic segmentation models, CCNet is used as the image feature extraction model 4, and the performance indicators are the best in completing the training of semantic segmentation and sentence description tasks respectively.

[0100] ②Compared with the untrained CCNet, the trained CCNet as an image feature extraction model 4 has better performance indicators in completing the two tasks of semantic segmentation and sentence description respectively.

[0101] Table 5-2 Statement description indicator results

[0102]

[0103] Based on the above comparative evaluation, the model parameters obtained by CCNet and the above 5 rounds of ordinary learning (XE) were selected to enter the subsequent reinforcement learning (SCST) training. The results are as follows:

[0104] ① As shown in Table 5-3: After ordinary learning, reinforcement learning can improve the sequence indicators of the introduced reward function to a certain extent. If reinforcement learning is used directly, the model will become unstable after one round. It can be seen that reinforcement learning can only be used for the fine-tuning process after ordinary learning training. When ordinary learning training is used first and the indicators converge to near the peak, reinforcement learning should be used instead.

[0105] ③ Such as Figure 5 As shown in the figure, the index reaches the peak after 5 rounds of ordinary learning. At this time, the training method is switched to reinforcement learning. The overall index of reinforcement learning improves rapidly between 1-5 rounds, and the index improves very slowly after 5 rounds. 5 rounds of ordinary learning + 8 rounds of reinforcement learning training are performed to obtain the optimal parameters.

[0106] Table 5-3 Statement description indicators

[0107] General learning and reinforcement learning strategies CIDEr-D BLEU-4 METEOR ROUGE Remark 15SCST 0.0021 0.0010 0.0043 0.0002 Instability after 1 round 15XE 2.6807 0.3836 0.3679 0.6109 5XE+10SCST 2.7263 0.3902 0.3595 0.6031

[0108] (3) Network model training for visual question answering tasks

[0109] Training program:

[0110] The visual question answering task network model is added as a branch structure to the serial structure, and the screened CCNet semantic segmentation model is added to the visual question answering task network model for training. First, the performance of the designed cross-modal fuser 8 (abbreviated as upAble) is evaluated by ordinary learning training; next, ordinary learning training + reinforcement learning training method is adopted, and the Beam Search algorithm is used to select appropriate elements during fine-tuning; at the same time, considering that the local feature joint encoding structure 6 and the text decoder 9 are used for the question answering system module and the semantic description module, the optimal parameters obtained by freezing and not freezing the sentence description task network model training are designed for training. For the non-freezing strategy, the model parameters obtained in each round of training are used for the sentence description task to obtain the sentence description index, and the optimal question answering system and semantic description task network model parameters are obtained by comprehensive evaluation of the visual question answering and sentence description indicators.

[0111] Training experiment results:

[0112] Evaluating upAble performance

[0113] The performance indicators of the existing Co-attention, Connected-attention, Cross-modal Skip-connectedNetwork, MCAN and the designed cross-modal fuser 8 (abbreviated as upAble) are evaluated. The same experimental parameters are used. The fusion layer of the cross-modal fuser adopts a 4-layer network structure and 5 rounds of ordinary learning.

[0114] As can be seen from Table 5-5, upAble outperforms the other four existing design structures in terms of performance. In addition, for some difficult questions under complex images, it can still perform well when other designs give incorrect answers. This proves the effectiveness of sentence description as an upstream task and using prior knowledge to guide text feature generation. Therefore, a cross-modal fuser 8 is used in subsequent training.

[0115] Table 5-5 Visual question answering Top-1 Accuracy indicators under different fusion methods

[0116] Cross-modal fusion methods Top-1 Accuracy Co-attention 0.7602 Connected-attention 0.3801 Cross-modal Skip-connected Network 0.7558 MCAN 0.7675 upAble 0.7712

[0117] Evaluate the effects of two training strategies on the optimal parameters obtained by training the network model for the unfrozen and frozen sentence description tasks

[0118] From Table 5-6 we can conclude that:

[0119] ① The visual question answering indicators obtained under the two strategies are close, but the optimal semantic description parameters obtained by freezing converge faster. When the visual question answering model parameters obtained by the unfrozen strategy are used for the sentence description task, the sentence description task indicators are lower than those obtained by the frozen strategy. Therefore, the frozen strategy is used for subsequent training.

[0120] ②Evaluate the effect of reward function 5WupLus Score loss function

[0121] From Table 5-6 and Figure 6 It can be concluded that:

[0122] ① After 5 rounds of ordinary XE learning, the index reaches the peak. At this time, reinforcement learning is used. After 5 rounds of reinforcement learning, the index improves very slowly, and the designed reward function WupLus Score is higher than the index obtained by the existing Accuracy reward function. After 5 rounds of ordinary learning + 5 rounds of WupLus Score reinforcement learning, the parameters obtained are optimal.

[0123] Table 5-6 Visual language and sentence description dual task indicators

[0124]

[0125] (4) The network model is used to solve the three subtasks of semantic segmentation, sentence description index and visual question answering.

[0126] The optimal model parameters are obtained according to the network model training process, and the model is tested on the labeled disaster area remote sensing image dataset to verify the effectiveness of the designed network model. According to the application test results, as shown in Tables 5-7, the model designed by the present invention can achieve good performance indicators in the three subtasks of semantic segmentation, sentence description index and visual question answering, and compared with the existing single-task model, the indicators are improved to a certain extent.

[0127] Table 5-7 Model application test results

[0128]

[0129] The above description is only a preferred embodiment of the invention and is not intended to limit the invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the invention should be included in the scope of protection of the invention.

Claims

1. A method for establishing a semantic description and question-answering system for remote sensing disaster detection, characterized in that: The method is used to solve three subtasks of semantic segmentation, sentence description and visual question answering, and includes the following steps: Step 1: Obtain a remote sensing image dataset of the disaster area and annotate it for network model training and evaluation, including annotating semantic segmentation data on the remote sensing image dataset to obtain remote sensing image segmentation maps of the disaster area, annotating sentence descriptions on the remote sensing image dataset to describe the situation in the disaster area, and annotating visual question and answer on the remote sensing image dataset to obtain question and answer texts of the disaster area; Step 2: Based on the labeled disaster area remote sensing image dataset, obtain the network model parameters for solving the three subtasks by training the network model, including: The classic semantic segmentation model is migrated to the semantic segmentation network model, and then semantic segmentation training is performed on the remote sensing image data set with annotated semantic segmentation data to obtain the optimal semantic segmentation network model parameters; the semantic segmentation training adopts a common learning training method to divide the remote sensing image of the disaster area into small blocks, which are input into the semantic segmentation network model in batches to generate local area features, and then the local area features are used to generate a semantic segmentation map of the local area of ​​the image, and finally the semantic segmentation result map is obtained by splicing; The sentence description network model and the semantic segmentation network model are serialized, and then sentence description training is performed on a remote sensing image dataset labeled with sentence descriptions to obtain optimal sentence description network model parameters, and the semantic segmentation model with the best performance in the semantic segmentation network model and sentence description network model training is screened out for subsequent visual question answering network model training, and the optimal semantic segmentation network model parameters are frozen during the sentence description training process, and a training method combining ordinary learning and reinforcement learning is adopted to perform feature fusion on the generated local area features to obtain global features, and finally the global features are decoded to obtain image text description; The visual question answering network model is added as a branch structure to the serial structure, and the selected semantic segmentation model is added to the visual question answering network model training, and then the visual question answering training is performed on the remote sensing image dataset labeled with visual question answering to obtain the optimal visual question answering network model parameters, the sentence description network model and the visual question answering network model are in a parallel relationship, and the sentence description is an upstream task of the visual question answering; during the visual question answering training process, the optimal semantic segmentation network model parameters and the optimal sentence description network model parameters are frozen, and a training method combining ordinary learning and reinforcement learning is adopted to generate local area features on the remote sensing image dataset, and the prior knowledge accumulated in the process of generating sentence description text from the image is used to guide the generation of text features of the input question, and the image features and text features are cross-modally fused to obtain fused features, and finally the fused features are decoded to obtain the answer text of the question; Step 3: Use the network model connection switching method to solve the three sub-tasks of semantic segmentation, sentence description and visual question answering. When it is necessary to solve the sentence description task, connect the semantic segmentation network model with the sentence description network model to obtain the image description text; when it is necessary to solve the visual question answering task, switch to the semantic segmentation network model and connect it with the visual question answering network model to obtain the question answering text.

2. The method for establishing a semantic description and question-answering system for remote sensing disaster detection according to claim 1, characterized in that: The step of fusing the generated local area features to obtain global features includes: stacking all the generated local feature maps using a local feature joint coding structure and sending them to a flattening layer to obtain output features, and linearly projecting and position encoding the output features to obtain a global feature map.

3. The text description method based on disaster remote sensing image detection according to claim 1 is characterized in that: The step of cross-modal fusing the image features and the text features comprises: using a cross-modal fusing device to splice the image features and the text features after they have passed through n dual-path feature generation layers, inputting a joint attention layer for feature fusion, and outputting fused features; the cross-modal fusing device cross-modally fuses and encodes the image features and the text features based on the prior knowledge accumulated in the upstream sentence description; n is the number of dual-path features.

4. The method for establishing a semantic description and question-answering system for remote sensing disaster detection according to claim 3, characterized in that: The two-way feature includes text self-attention, image self-attention and image-guided text attention, wherein the image self-attention uses the network model parameters obtained by training in the upstream sentence description, and the image-guided text attention process is: head=A(x i W Q , IS K , IS V ), Z=(head1,head2,...he...headad n )W O , O=FFN(Z) in, is a text feature, is the image feature, is the projection matrix of multi-head attention, is the dimensionality reduction matrix of multi-head attention, and FFN is the feed-forward network.

5. The method for establishing a semantic description and question-answering system for remote sensing disaster detection according to claim 1, characterized in that: The ordinary learning process uses cross entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses the Beam Search algorithm to infer and generate word sequences, and uses a sequence loss function to judge the quality of sequence generation.

6. A method for establishing a semantic description and question-answering system for remote sensing disaster detection according to claim 5, characterized in that: The sequence loss function in the sentence description training process adopts CIDEr-D (w s ,w * ) is used as the sequence evaluation index, and the reinforcement learning loss function in the sentence description training process is: Among them, Y represents the predicted sequence; Z represents the corresponding target sequence; k is the total number of sequences in each batch; b is the baseline function, that is, the average of the reward function values ​​of all sequences in this batch; reward(w i ) is the reward function indicator; CIDEr-D(w s ,w * ) is a sequence evaluation index.

7. The text description method for remote sensing image detection in disaster areas according to claim 5 is characterized in that: The sequence loss function in visual question answering training uses WupLus Score as the sequence judgment indicator, and the WupLus Score is: Among them, c A ,c T is the sequence length, n a ,n tmax are the number of elements a in sequence A and the number of elements t in sequence T that are most similar to a; The reinforcement learning loss function in the visual question answering training process is: Among them, c Y ,c Z represents the original length of the generated sequence and the corresponding target sequence without padding value; Y = {y 0 ,y 1 ,...,y T }, Z={z 0 ,z 1 ,...,z T } represents the sequence elements contained in the generated sequence and the corresponding target sequence; Represents the sequence element y t The number of in sequence Y; Represents the element y in the sequence Z t , the number of elements with the highest Wup Score.

8. A semantic description and question-answering system for remote sensing disaster detection, characterized in that: include: Image data collection and annotation module, used for network model training and evaluation. The image data set collection and annotation module includes annotating semantic segmentation data of remote sensing image data sets to obtain remote sensing image segmentation maps of disaster areas, annotating sentence descriptions of remote sensing image data sets to describe the situation in disaster areas, and annotating visual question and answer of remote sensing image data sets to obtain question and answer texts of disaster areas. A semantic segmentation network model module is used to solve the semantic segmentation task. The semantic segmentation network model module includes a remote sensing image input structure, an image segmentor, an image feature extraction model, and a segmentation head. The remote sensing image of the disaster area generates local image features through the image feature extraction model. A sentence description network model module is used to solve the semantic segmentation task, and the sentence description network model module includes a local feature joint encoding structure and a text decoder; the local feature joint encoding structure is used to generate global image features from local image features, and the local feature joint encoding structure includes a flattening layer, a linear projection layer, and a position encoding layer; A visual question answering task network model module is used to solve the visual question answering task. The visual question answering task network model module includes a question answering text input structure, a text encoder, a local feature joint encoding structure, a cross-modal fuser and the text decoder; the cross-modal fuser is used to fuse local image features and text features of the question to generate fusion features. The cross-modal fuser is an encoder based on the prior knowledge accumulated in the sentence description to guide the generation and encoding of fusion features. The cross-modal fuser includes n two-way feature generation layers and a joint attention layer. The network model connection module is used to switch the network model connection to solve the three subtasks of semantic segmentation, sentence description and visual question answering. When it is necessary to solve the sentence description task, the semantic segmentation network model is connected with the sentence description network model to obtain the image description text; when it is necessary to solve the visual question answering task, it is switched to connect the semantic segmentation network model with the visual question answering network model to obtain the question answering text; The network model training learning evaluation module is used to obtain the optimal network model parameters for solving the three subtasks. The training learning includes ordinary learning and reinforcement learning. The ordinary learning process uses cross entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses Beam Search algorithm to infer and generate word sequences, and uses sequence loss function to judge the quality of sequence generation; the sequence loss function in the visual question answering training uses the modified WupLus Score as the sequence evaluation indicator.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for establishing a semantic description and question-answering system for remote sensing disaster detection as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for establishing a semantic description and question-answering system for remote sensing disaster detection as described in any one of claims 1 to 7.

Images