A method and system for establishing a semantic description and question-answering system for remote sensing disaster detection

Abstract

The present application relates to the field of computer vision language and image recognition, and particularly relates to a semantic description and question-answering system establishment method and system for remote sensing disaster detection, which generates text language description capable of accurately responding to task requirements image under the premise of adding text constraints by using image features, and establishes a mapping relationship between the pixel-level segmentation result and the question-answering system, and extracts high-level semantic information, and establishes a question-answering system, which can accurately output answers to disaster inquiry questions. The present application can complete three sub-tasks of semantic segmentation, sentence description and question-answering system by using a set of network models, thereby saving computer memory space and improving response speed, and providing technical support for improving the automation, intelligent level and rapid response capability of the remote sensing disaster detection system.

Description

Technical Field

[0001] This invention relates to the fields of computer vision language and image recognition, specifically a method and system for establishing a semantic description and question-answering system for remote sensing disaster detection. Background Technology

[0002] Visual language problems are interdisciplinary problems that combine computer vision and natural language processing. Image-to-text generation tasks, such as sentence description and visual question answering, are important cross-modal visual language tasks. Their purpose is to generate textual descriptions of images using image features, with the addition of possible textual constraints. In order to generate text that can accurately respond to the task requirements, the model needs to not only accurately extract key image features, but also understand and analyze the relationships between features and extract potential high-level semantic information based on textual constraints.

[0003] Image text generation models can be divided into two types: single-task and multi-task pre-trained models. Single-task models typically use only one text generation task, such as sentence description or visual question answering, as the optimization objective and are trained using a single-task dataset. In recent years, with the rapid development of datasets and GPU computing power, multi-task visual language pre-trained (VLP) models have emerged. By pre-training a large model with a general structure, the trained model is adjusted in structure and then used for downstream tasks, where parameters are fine-tuned on specific datasets.

[0004] Remote sensing images possess advantages such as high resolution, multispectral density, and wide coverage, making them widely applicable and highly efficient. Change detection and processing methods for remote sensing images can sift through the complex information within disaster remote sensing images, extracting the disaster information needed by experts. This is of great significance for assessing the severity of natural disasters such as floods and earthquakes. Currently, on-site processing commonly utilizes semantic segmentation and change detection methods to process remote sensing images, employing pixel-level or region-level detection methods to output images. However, due to a lack of high-level semantic information extraction and expression capabilities, these methods cannot achieve semantic description of disaster-affected areas or key regions. Furthermore, they cannot provide answers to questions raised about the disaster situation. Clearly, existing detection methods still require experts to make judgments based on the output images to provide a disaster analysis. The low level of automation and intelligence in these methods hinders the rapid and accurate assessment of the severity of disasters in natural areas.

[0005] In recent years, image text generation has received increasing attention in the field of remote sensing imagery due to its potential for disaster applications. Although researchers in this field have conducted numerous studies on remote sensing image text generation (application numbers: 202310339866.4, 202110982615.9, 201811584864.7), these studies only focus on a single visual language task or treat object detection as an additional task to a visual language task. No network model has yet emerged that can effectively perform multiple tasks such as semantic segmentation, sentence description, and visual question answering.

[0006] To address the problems in the aforementioned technologies, this invention provides a method and system for establishing a semantic description and question-answering system for remote sensing disaster detection. This system can use the same network model to complete three sub-tasks: semantic segmentation, statement description, and a question-answering system. It enables semantic segmentation of disaster areas, description of the disaster situation, and establishment of a question-answering system. This system can accurately output answers to user inquiries about the disaster situation, providing technical support for improving the automation, intelligence, and rapid response capabilities of remote sensing disaster detection systems. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for establishing a semantic description and question-answering system for remote sensing disaster detection, so as to solve the problems mentioned in the background art.

[0008] A method for establishing a semantic description and question-answering system for remote sensing disaster detection, the method is used to solve three sub-tasks: semantic segmentation, sentence description, and visual question answering, and includes the following steps:

[0009] Step 1: Obtain and annotate remote sensing image datasets of disaster areas for network model training and evaluation. This includes annotating semantic segmentation data in the remote sensing image datasets to obtain remote sensing image segmentation maps of disaster areas, annotating descriptive statements in the remote sensing image datasets to describe the situation in disaster areas, and annotating visual question-and-answer data in the remote sensing image datasets to obtain question-and-answer text for disaster areas.

[0010] Step Two: Based on the labeled remote sensing image dataset of disaster areas, obtain the network model parameters to solve the three sub-tasks by training the network model, specifically including:

[0011] The classic semantic segmentation model is transferred to the semantic segmentation network model. Then, semantic segmentation is trained on a remote sensing image dataset labeled with semantic segmentation data to obtain the optimal semantic segmentation network model parameters. The semantic segmentation training adopts the ordinary learning training method, which segments the remote sensing image of the disaster area into small blocks, inputs them into the semantic segmentation network model in batches to generate local regional features, and then generates local semantic segmentation maps of the image based on the local regional features. Finally, the semantic segmentation result map is obtained by stitching together the segments.

[0012] The sentence description network model and the semantic segmentation network model are sequentially trained. Then, the sentence description is trained on a remote sensing image dataset labeled with sentence descriptions to obtain the optimal parameters of the sentence description network model. The semantic segmentation model, which performs best in both the semantic segmentation network model and the sentence description network model training, is selected for subsequent visual question answering network model training. During the sentence description training process, the optimal parameters of the semantic segmentation network model are frozen. A training method combining ordinary learning and reinforcement learning is used to fuse the generated local region features to obtain global features. Finally, the global features are decoded to obtain the image text description.

[0013] A visual question answering (VQA) network model is added as a branch structure to the serial structure, and the selected semantic segmentation model is added to the training of the VQA network model. Then, VQA training is performed on a remote sensing image dataset labeled with visual questions and answers to obtain the optimal VQA network model parameters. The statement description network model and the VQA network model are parallel, and the statement description is the upstream task of visual question answering. During the VQA training process, the optimal semantic segmentation network model parameters and the optimal statement description network model parameters are frozen. A training method combining ordinary learning and reinforcement learning is adopted. After generating local region features on the remote sensing image dataset, the prior knowledge accumulated in the process of generating statement description text in the image is used to guide the generation of text features of the input question. Cross-modal fusion of image features and text features is performed to obtain fused features. Finally, the fused features are decoded to obtain the answer text of the question.

[0014] Step 3: Solve the three sub-tasks of semantic segmentation, sentence description, and visual question answering by switching network model connections. When the sentence description task needs to be solved, connect the semantic segmentation network model with the sentence description network model to obtain the image description text; when the visual question answering task needs to be solved, switch to connect the semantic segmentation network model with the visual question answering network model to obtain the question and answer text.

[0015] Furthermore, the step of fusing the generated local region features to obtain global features includes: stacking all the generated local feature maps using a local feature joint coding structure and sending them into a flattening layer to obtain output features; and performing linear projection and position encoding on the output features to obtain a global feature map.

[0016] Furthermore, the step of performing cross-modal fusion of image features and text features includes: using a cross-modal fusion device to concatenate the image features and text features after passing through n dual-path feature generation layers and inputting them into a joint attention layer for feature fusion, and outputting fused features; the cross-modal fusion device performs cross-modal fusion and encoding of image features and text features based on prior knowledge accumulated in the upstream sentence description; where n is the number of dual-path features.

[0017] The dual-path features include text self-attention, image self-attention, and image-guided text attention. The image self-attention uses the network model parameters obtained during training based on the upstream sentence description. The image-guided text attention process is as follows:

[0018]

[0019] head = A(x i W Q YW K YW V ),

[0020] Z = (head1, head2, ... head) n W O ,

[0021] O = FFN(Z)

[0022] in, It is a text feature. It is an image feature. It is the projection matrix of multi-head attention. It is a dimensionality reduction matrix for multi-head attention, and FFN is a feedforward network.

[0023] Furthermore, the ordinary learning process uses cross-entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses the Beam Search algorithm to infer and generate word sequences, and uses a sequence loss function to evaluate the quality of sequence generation.

[0024] Furthermore, the sequence loss function used in the training process is described by CIDEr-D(w s ,w * As a sequence evaluation metric, the statement describes the reinforcement learning loss function during the training process as follows:

[0025]

[0026] Where Y represents the predicted sequence; Z represents the corresponding target sequence; k is the total number of sequences in each batch; b is the baseline function, i.e., the average reward function value of all sequences in this batch; reward(w i) is the reward function metric; CIDEr-D(w s ,w * () is a sequence evaluation index.

[0027] Furthermore, the sequence loss function in visual question answering training uses the WupLus Score as the sequence evaluation metric, where the WupLus Score is:

[0028]

[0029] Among them, c A ,c T It is the sequence length, n a ,n tmax These are the number of elements a in sequence A and the number of elements t in sequence T that are most similar to a.

[0030] The reinforcement learning loss function in the visual question answering training process is:

[0031]

[0032] Among them, c Y ,c Z Y represents the original length of the generated sequence and the corresponding target sequence without padding values; Y = {y 0 ,y 1 ,...,y T}, Z={z 0 ,z 1 ,...,z T} represents the sequence elements contained in the generated sequence and the corresponding target sequence; Represents the sequence element y t The number of occurrences in sequence Y; This represents the value of element y in sequence Z. t The number of elements with the highest Wup Score.

[0033] A semantic description and question-answering system for remote sensing disaster detection, comprising:

[0034] The image data collection and annotation module is used for network model training and evaluation. The image dataset collection and annotation module includes annotating semantic segmentation data of remote sensing image datasets to obtain remote sensing image segmentation maps of disaster areas, annotating sentence descriptions of remote sensing image datasets to describe the situation in disaster areas, and annotating visual question-and-answer data of remote sensing image datasets to obtain question-and-answer text for disaster areas.

[0035] A semantic segmentation network model module is used to solve semantic segmentation tasks. The semantic segmentation network model module includes a remote sensing image input structure, an image segmenter, an image feature extraction model, and a segmentation head. Remote sensing images of disaster areas generate local image features through the image feature extraction model.

[0036] The statement description network model module is used to solve semantic segmentation tasks. The statement description network model module includes a local feature joint encoding structure and a text decoder. The local feature joint encoding structure is used to generate global image features from local image features. The local feature joint encoding structure includes a flattening layer, a linear projection layer and a position encoding layer.

[0037] A visual question answering task network model module is used to solve visual question answering tasks. The visual question answering task network model module includes a question answering text input structure, a text encoder, a local feature joint encoding structure, a cross-modal fusion device, and the text decoder. The cross-modal fusion device is used to fuse local image features and question text features to generate fused features. The cross-modal fusion device is an encoder that uses prior knowledge accumulated in the statement description to guide the generation and encoding of fused features. The cross-modal fusion device includes n dual-path feature generation layers and a joint attention layer.

[0038] The network model connection module is used to switch network model connections to solve three sub-tasks: semantic segmentation, sentence description, and visual question answering. When the sentence description task needs to be solved, the semantic segmentation network model is connected with the sentence description network model to obtain the image description text; when the visual question answering task needs to be solved, the connection is switched between the semantic segmentation network model and the visual question answering network model to obtain the question and answer text.

[0039] The network model training and evaluation module is used to obtain the optimal network model parameters for solving three sub-tasks. The training and learning includes ordinary learning and reinforcement learning. The ordinary learning process uses cross-entropy loss to evaluate the quality of sequence generation. The reinforcement learning process uses the Beam Search algorithm to infer and generate word sequences, and uses a sequence loss function to evaluate the quality of sequence generation. The sequence loss function in the visual question answering training uses WupLus Score as the sequence evaluation index.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] (1) A local feature joint coding structure was designed to extract image features, which solved the problem of how to efficiently and accurately extract global features for high-resolution, large-size images;

[0042] (2) Visual question answering is regarded as a constrained sentence description task. Semantic segmentation and sentence description are used as the allocation strategy for upstream tasks of visual question answering. A visual-language cross-modal fusion fusion based on upstream accumulation is designed. The feature extraction and encoding are guided by a large amount of prior knowledge accumulated in sentence description, which reduces computing power and memory space. At the same time, the WupLusScore sequence evaluation index is proposed to solve the problem of the penalty for obtaining high scores by repeating words and the unreasonable reward setting caused by the insensitivity to numerical answers during the evaluation process.

[0043] (3) By establishing a mapping relationship between pixel-level segmentation results and question-answering system, high-level semantic information is extracted and a question-answering system is established, which can accurately output answers to questions about disaster situation. Test results show that the design of this invention can achieve good performance indicators in the three sub-tasks of semantic segmentation, sentence description indicators and visual question answering. It can be seen that a network model designed by this invention can complete the three sub-tasks of semantic segmentation, sentence description and question-answering system, saving computer memory space and improving response speed, providing technical support for improving the automation, intelligence and rapid response capability of remote sensing disaster detection system. Attached Figure Description

[0044] Figure 1 The steps of the method for establishing a semantic description and question-answering system for remote sensing disaster detection provided in the embodiments of this application

[0045] Figure 2 The network model framework structure provided in the embodiments of this application

[0046] Figure 3 The cross-modal fusion structure provided in the embodiments of this application

[0047] Figure 4 The semantic segmentation training miou index change curve provided in the embodiments of this application

[0048] Figure 5 The statement provided in this application describes the training CIDEr-D index change curve.

[0049] Figure 6 The Top-1 Accuracy index change curve for visual question answering training provided in the embodiments of this application

[0050] In the figure: 1-Remote sensing image input structure; 2-Question-answer text input structure; 3-Image segmenter; 4-Image feature extraction model; 5-Segmentation head; 6-Local feature joint encoding structure; 7-Text encoder; 8-Cross-modal fusion unit; 9-Text decoder; 10-Word sequence. Detailed Implementation

[0051] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0052] like Figure 1 The diagram illustrates the steps of a method for establishing a semantic description and question-answering system for remote sensing disaster detection, as provided in an embodiment of this application. The method includes:

[0053] S1: Obtain and annotate remote sensing image datasets of disaster areas for network model training and evaluation. This includes annotating semantic segmentation data of the remote sensing image datasets to obtain remote sensing image segmentation maps of disaster areas, annotating statement descriptions of the remote sensing image datasets to describe the situation in disaster areas, and annotating visual question-and-answer data of the remote sensing image datasets to obtain question-and-answer text for disaster areas.

[0054] S2: Based on annotated remote sensing image datasets of disaster areas, network model parameters are obtained by training a network model to solve three sub-tasks, specifically including:

[0055] The classic semantic segmentation model is transferred to the semantic segmentation network model. Then, semantic segmentation training is performed on a remote sensing image dataset labeled with semantic segmentation data to obtain the optimal semantic segmentation network model parameters. The semantic segmentation training adopts the ordinary learning training method, which segments the remote sensing image of the disaster area into small blocks, inputs them into the semantic segmentation network model in batches to generate local region features, and then generates local semantic segmentation maps of the image based on the local region features. Finally, the semantic segmentation result map is obtained by stitching together the images.

[0056] The sentence description network model and the semantic segmentation network model are sequentially trained. Then, the sentence description is trained on a remote sensing image dataset labeled with sentence descriptions to obtain the optimal parameters of the sentence description network model. The semantic segmentation model with the best performance in the training of the semantic segmentation network model and the sentence description network model is selected for subsequent training of the visual question answering network model. During the sentence description training process, the optimal parameters of the semantic segmentation network model are frozen. A training method combining ordinary learning and reinforcement learning is adopted to fuse the generated local region features to obtain global features. Finally, the global features are decoded to obtain the image text description.

[0057] A visual question answering (VQA) network model is added as a branch structure to the serial structure, and the selected semantic segmentation model is added to the training of the VQA network model. Then, VQA training is performed on a remote sensing image dataset labeled with visual questions and answers to obtain the optimal VQA network model parameters. The statement description network model and the VQA network model are parallel, and the statement description is the upstream task of visual question answering. During the VQA training process, the optimal semantic segmentation network model parameters and the optimal statement description network model parameters are frozen. A training method combining ordinary learning and reinforcement learning is adopted. After generating local region features on the remote sensing image dataset, the prior knowledge accumulated in the process of generating statement description text in the image is used to guide the generation of text features of the input question. Cross-modal fusion of image features and text features is performed to obtain fused features. Finally, the fused features are decoded to obtain the answer text of the question.

[0058] S3: Solve the three sub-tasks of semantic segmentation, sentence description, and visual question answering by switching network model connections. When the sentence description task needs to be solved, the semantic segmentation network model is connected with the sentence description network model to obtain the image description text; when the visual question answering task needs to be solved, the connection between the semantic segmentation network model and the visual question answering network model is switched to obtain the question and answer text.

[0059] This embodiment establishes a network model to solve three sub-tasks: semantic segmentation, sentence description, and visual question answering. For the sentence description sub-task, to address the issue that high-resolution images require global features for sentence description and visual question answering, but global features are limited by GPU memory, local region features are fused to obtain global features, and finally, the global features are decoded to obtain the image text description. For the visual question answering sub-task, to reduce computing power and memory space, the prior knowledge accumulated during the image-to-sentence description training process is used to guide the generation of text features for the input question, and image features and text features are fused across modally to obtain fused features, and finally, the fused features are decoded to obtain the answer text for the question. The network model is trained and evaluated using a disaster area remote sensing image dataset labeled with semantic segmentation data, sentence description, and visual question answering data, and the parameters of the network model are optimized.

[0060] In another preferred embodiment of the present invention, step S1 includes the following:

[0061] We collected 2343 existing disaster remote sensing images with a size of 4000*3000. We performed semantic segmentation and annotation on the images to obtain image segmentation maps; we performed text description annotation on the images to obtain descriptions of the disaster area; we performed visual question-and-answer annotation on the images to obtain questions and answers about the disaster area. At the same time, we removed incorrectly annotated images and supplemented the text description dataset to supplement the visual question-and-answer validation and test sets. Finally, we obtained a dataset consisting of 1405 training images and their text annotations, 436 validation images and their text annotations, 448 test images and their text annotations, and 7107 questions and answers.

[0062] like Figure 2 As shown, in another preferred embodiment of the present invention, step S2 includes the following: the network model includes an image dataset collection and annotation module, a semantic segmentation network model module, a sentence description network model module, a visual question answering network model module, and a switching module; the semantic segmentation network model module includes a remote sensing image input structure 1, an image segmenter 3, an image feature extraction model 4, and a segmentation head 5 for solving semantic segmentation tasks; the sentence description network model module includes a local feature joint encoding structure 6 and a text decoder 9 for solving sentence description tasks; the visual question answering task network model module includes a question answering text input structure 2, a text encoder 7, a local feature joint encoding structure 6, a cross-modal fusion device 8, and the text decoder 9 for solving visual question answering tasks;

[0063] The working principle is as follows: Disaster remote sensing images are segmented into small blocks by image segmenter 3, and then fed into the corresponding image feature extraction model 4 in batches to generate local image features. The local image features are then stitched together by a branch connected to segmentation head 5 to produce segmented images. When the subtask is a sentence description task, the local image features 5 are fed into the local feature joint encoding structure 6 through another branch to generate global image features for the language description content. The global image features are then fed into the text decoder 9 to generate the corresponding description sentences. When the subtask is a visual question answering task, the question and answer text is input through question and answer text input structure 2 and then encoded by text encoder 7 to extract text features. The text features and global image features are then fused and encoded by cross-modal fusion unit 8 to obtain fused features. The fused features are then fed into the text decoder 9 to obtain the question answer.

[0064] As another preferred embodiment of the present invention, the step of performing feature fusion on the generated local region features to obtain global features includes: stacking all the generated local feature maps using the local feature joint coding structure 6 and sending them into the flattening layer to obtain output features, and performing linear projection and position encoding on the output features to obtain a global feature map.

[0065] In this embodiment, in order to address the problem that remote sensing images require global features in sentence description and visual question answering tasks, but global features are limited by video memory conditions, a local feature joint encoding structure 6 is designed. It imitates ViT to generate a global feature map based on the local features of the image. The global feature map is then successively encoded and decoded to obtain the sentence description text.

[0066] like Figure 3 As shown, in another preferred embodiment of the present invention: the step of cross-modal fusion of image features and text features includes: using a cross-modal fusion processor 8 to concatenate the image features and text features after passing through n dual-path feature generation layers and inputting them into a joint attention layer for feature fusion, outputting fused features; the cross-modal fusion processor 8 performs cross-modal fusion and encoding of image features and text features based on prior knowledge accumulated in the upstream sentence description task; n is the number of dual-path features, the dual-path features include text self-attention, image self-attention, and image-guided text attention, wherein the image self-attention uses network model parameters trained in the upstream sentence description task, and the image-guided text attention process is as follows:

[0067]

[0068] head = A(x i W Q YW K YW V ),

[0069] Z = (head1, head2, ... head) n W O ,

[0070] O = FFN(Z)

[0071] in, It is a text feature. It is an image feature. It is the projection matrix of multi-head attention. It is a dimensionality reduction matrix for multi-head attention, and FFN is a feedforward network.

[0072] In this embodiment, after the network extracts image features and text features, it needs to perform cross-modal fusion of the two features to promote information interaction. Since the image feature extraction part is pre-trained in the upstream task, its accumulated dataset knowledge is much higher than that of the text feature extraction branch added later. At the same time, since the image exists in two-dimensional space, the amount of information it contains is much greater than that of the text. Compared with existing designs, the image feature extraction model 4 of this application can provide a large amount of prior knowledge to guide the generation of text features of the input question, and perform cross-modal fusion of image features and text features to obtain fused features. Finally, the fused features are decoded to generate the answer text.

[0073] As another preferred embodiment of the present invention: the ordinary learning process uses cross-entropy loss to evaluate the quality of sequence generation; the reinforcement learning process uses the Beam Search algorithm to infer and generate word sequences, and uses a sequence loss function to evaluate the quality of sequence generation. The weights and sequence indicators based on Beam Search are introduced into the loss function, which is more consistent with the actual generation situation.

[0074] As another preferred embodiment of the present invention: the sequence loss function used in the statement describing the task training process adopts CIDEr-D(w s ,w * As a sequence evaluation metric, the statement describes the reinforcement learning loss function during the training process as follows:

[0075]

[0076] Where Y represents the predicted sequence; Z represents the corresponding target sequence; k is the total number of sequences in each batch; b is the baseline function, i.e., the average reward function value of all sequences in this batch; reward(w i ) is the reward function metric; CIDEr-D(w s ,w * () is a sequence evaluation index.

[0077] As another preferred embodiment of the present invention: the sequence loss function in the visual question answering task training adopts WupLus Score as the sequence evaluation metric, wherein WupLus Score is:

[0078]

[0079] Among them, c A ,c T It is the sequence length, n a ,n tmax These are the number of elements a in sequence A and the number of elements t in sequence T that are most similar to a.

[0080] The reinforcement learning loss function in the visual question answering training process is:

[0081]

[0082] Among them, c Y ,c Z Y represents the original length of the generated sequence and the corresponding target sequence without padding values; Y = {y 0 ,y 1 ,...,y T}, Z={z 0 ,z 1 ,...,z T} represents the sequence elements contained in the generated sequence and the corresponding target sequence; Represents the sequence element y t The number of occurrences in sequence Y; This represents the value of element y in sequence Z. t The number of elements with the highest Wup Score.

[0083] In this embodiment, for existing sequence evaluation metrics, Wup Score can only measure the similarity between words; it is not sensitive to numerical answers. Wup Score lacks penalties for using repeated words to obtain high scores and is still not sensitive to numerical answers. The WupLus Score designed in this application introduces sequence length penalties and repeated word penalties on the basis of existing sequence evaluation metrics, and redefines the measurement method of numerical answer similarity. It can solve the problem of unreasonable reward settings caused by penalties for using repeated words to obtain high scores and the insensitivity to numerical answers during the evaluation process.

[0084] As another preferred embodiment of the present invention: the trained network model in step S2 specifically includes:

[0085] Evaluation metrics: For semantic segmentation tasks, miou is selected as the evaluation metric; for sentence description tasks, BLEU, METEOR, ROUGE-L, and CIDEr-D are selected as the evaluation metrics; for visual question answering tasks, Top-1 Accuracy is selected as the evaluation metric. These evaluation metrics are commonly used in various subdomains, and the higher the score, the better the performance.

[0086] The following training uses the SGD method for optimization, with an initial learning rate of 0.01. The learning rate adjustment strategy uses the poly method. After each training round, the evaluation index is obtained by comparing with the corresponding annotations on the remote sensing disaster dataset.

[0087] 1. Training of semantic segmentation network model

[0088] Training plan:

[0089] Seven classic semantic segmentation models—UNet, DeepLabv3+, PSPNet, CCNet, FarSeg, STDC-Seg, and Swin Transformer+uperNet (Super for short)—were used as image feature extraction models and transferred to the semantic segmentation task network model. The model was trained on a remote sensing image dataset with labeled semantic segmentation data to obtain the optimal sentence description network model parameters. Ordinary learning training was used, cross-entropy loss was used for optimization, and the average intersection-union ratio (MIOU) was used as the performance metric.

[0090] Training results:

[0091] like Figure 4 As shown, after 5 rounds of training in normal learning, the improvement of the miou index slows down. After 35 rounds of training, the improvement of the miou index is very small. At the end of 38 rounds of training, the miou index reaches its peak, at which point the network model parameters for the semantic segmentation task are optimal.

[0092] As can be seen from Table 5-1, compared with the other six semantic segmentation models, CCNet as the image feature extraction model 4 has the highest miou index. However, in order to show its performance in subsequent sentence descriptions, PSPNet, CCNet, and STDC-Seg, which have relatively high indexes, were used as image feature extraction models 4 for training in subsequent sentence descriptions.

[0093] Table 5-1 Semantic segmentation index results (after 40 rounds of normal learning and training)

[0094] category iou1 iou2 iou4 iou5 iou6 iou7 iou8 iou9 miou UNet 0.0430 0.4259 0.6651 0.3390 0.6438 0.0036 0.3531 0.7517 0.4730 DeepLabv3+ 0.5949 0.7269 0.7622 0.5921 0.7741 0.4197 0.4995 0.8390 0.6246 PSPNet 0.6240 0.7743 0.7916 0.6763 0.7881 0.4846 0.5667 0.8470 0.6720 CCNet 0.6515 0.7801 0.8075 0.6797 0.7947 0.5426 0.5932 0.8684 0.6880 FarSeg 0.6417 0.7519 0.7760 0.5911 0.7724 0.3964 0.5573 0.8483 0.6418 STDC-Seg 0.6342 0.7755 0.8012 0.6408 0.7800 0.3771 0.5181 0.8578 0.6509 Super 0.6084 0.6736 0.7488 0.6426 0.7673 0.3610 0.4141 0.8417 0.6075

[0095] 2. Training the network model for the statement description task

[0096] Training plan:

[0097] After sequentially training the sentence description network model and the semantic segmentation network model, ordinary learning is used for initial training, followed by reinforcement learning training to fine-tune the parameters. The Beam Search algorithm is used to select suitable elements, and both the encoding layer and the semantic decoding layer use 3-layer Transformers. After training, the optimal parameters of the semantic segmentation task network model are obtained, and the semantic segmentation model with the best performance in both semantic segmentation and sentence description tasks is selected as the image feature extraction model 4 and entered into the subsequent visual question answering task network model training.

[0098] Training process and results:

[0099] As shown in Table 5-2, the index reaches its peak after 5 rounds of normal learning and training. At this point, the sentence description index is obtained. From Table 5-2 and Table 5-1, we can conclude that:

[0100] ① Compared to other semantic segmentation models, CCNet as the image feature extraction model 4 achieved the best performance in both semantic segmentation and sentence description tasks during training.

[0101] ② Compared to the untrained CCNet, the trained CCNet, as an image feature extraction model 4, performs better in both semantic segmentation and sentence description tasks, with superior performance metrics.

[0102] Table 5-2 Statement Description Indicator Results

[0103]

[0104] Based on the above comparative evaluation, the model parameters obtained from CCNet and the above 5 rounds of ordinary learning (XE) were selected for subsequent reinforcement learning (SCST) training, and the results are as follows:

[0105] ① As shown in Table 5-3: After ordinary learning, reinforcement learning can improve the sequence index of the reward function. If reinforcement learning is used directly, the model will become unstable after one round. It can be seen that reinforcement learning can only be used for fine-tuning after ordinary learning training. When ordinary learning training is used first and convergence is obtained to the point where the index is close to the peak, reinforcement learning should be used instead.

[0106] ③ For example Figure 5 As shown, the index reaches its peak after 5 rounds of normal learning. At this point, the training method is switched to reinforcement learning. Reinforcement learning improves the overall index rapidly between 1 and 5 rounds, but the index improves very slowly after 5 rounds. The optimal parameters are obtained by performing 5 rounds of normal learning + 8 rounds of reinforcement learning training.

[0107] Table 5-3 describes the indicators.

[0108] Ordinary learning and reinforcement learning strategies CIDEr-D BLEU-4 METEOR ROUGE Remark 15SCST 0.0021 0.0010 0.0043 0.0002 Lost stability after 1 round 15XE 2.6807 0.3836 0.3679 0.6109 5XE+10SCST 2.7263 0.3902 0.3595 0.6031

[0109] (3) Training of network models for visual question answering tasks

[0110] Training plan:

[0111] The visual question answering task network model is added as a branch structure to the serial structure, and the selected CCNet semantic segmentation model is added to the visual question answering task network model for training. First, the performance of the designed cross-modal fusion generator 8 (upAble) is evaluated using ordinary learning training. Next, ordinary learning training + reinforcement learning training is adopted, and the Beam Search algorithm is used to select suitable elements during fine-tuning. At the same time, considering that the local feature joint encoding structure 6 and the text decoder 9 are used for both the question answering system module and the semantic description module, two strategies are designed for training: freezing and not freezing the optimal parameters obtained from the sentence description task network model training. For the non-freezing strategy, the model parameters obtained from each round of training are used again for the sentence description task to obtain the sentence description index. The optimal question answering system and semantic description task network model parameters are obtained by comprehensively evaluating the visual question answering and sentence description index.

[0112] Training experiment results:

[0113] Evaluation of upAble performance

[0114] The performance metrics of five existing modal fusion methods—Co-attention, Connected-attention, Cross-modal Skip-connectedNetwork, MCAN, and the designed cross-modal fusion generator 8 (upAble)—were evaluated. The same experimental parameters were used, and the fusion layer of the cross-modal fusion generator adopted a 4-layer network structure and 5 rounds of normal learning.

[0115] As shown in Table 5-5, upAble outperforms the other four existing design structures in terms of performance. In addition, it still performs well for some difficult problems in complex images when other designs answer incorrectly. This proves the effectiveness of using sentence description as an upstream task to guide text feature generation with prior knowledge. Therefore, cross-modal fusion 8 is used for subsequent training.

[0116] Table 5-5 Top-1 Accuracy Index of Visual Question Answering under Different Fusion Methods

[0117] Cross-modal fusion method Top-1 Accuracy Co-attention 0.7602 Connected attention 0.3801 Cross-modal Skip-connected Network 0.7558 MCAN 0.7675 upAble 0.7712

[0118] The effectiveness of two training strategies—optimal parameters obtained from training network models for unfrozen and frozen statement description tasks—is evaluated.

[0119] From Table 5-6, we can conclude that:

[0120] ① The visual question answering metrics obtained under the two strategies are similar, but the convergence is faster after freezing the optimal semantic description parameters. Furthermore, the visual question answering model parameters obtained under the unfrozen strategy are used for the sentence description task, and the resulting sentence description task metrics are lower than those obtained under the frozen strategy. Therefore, the frozen strategy is adopted for subsequent training.

[0121] ② Evaluate the effectiveness of the reward function 5WupLus Score loss function.

[0122] From Tables 5-6 and Figure 6 We can conclude that:

[0123] ① After 5 rounds of ordinary XE learning, the index reaches its peak. At this point, reinforcement learning is used. After 5 rounds of reinforcement learning, the index improves very slowly. Moreover, the index obtained by the designed reward function WupLus Score is higher than that obtained by the existing Accuracy reward function. The parameters obtained after 5 rounds of ordinary learning + 5 rounds of WupLus Score reinforcement learning are optimal.

[0124] Table 5-6 Dual-task indicators for visual language and sentence description

[0125]

[0126] (4) The network model is used to solve the testing of the three sub-tasks: semantic segmentation, sentence description metrics, and visual question answering.

[0127] The optimal model parameters were obtained through the network model training process. The model was then tested on a labeled remote sensing image dataset of disaster areas to verify the effectiveness of the designed network model. According to the application test results, as shown in Tables 5-7, the model designed in this invention achieves good performance indicators in all three sub-tasks: semantic segmentation, sentence description metrics, and visual question answering. Furthermore, compared to existing single-task models, the indicators show a certain degree of improvement.

[0128] Table 5-7 Model Application Test Results

[0129]

[0130] The above description is merely a preferred embodiment of the invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for establishing a semantic description and question-answering system for remote sensing disaster detection, characterized in that, The method is used to solve three subtasks of semantic segmentation, sentence description and visual question answering, and comprises the following steps: Step 1: Obtain disaster area remote sensing image dataset and label it for network model training and evaluation, including labeling semantic segmentation data of the remote sensing image dataset for obtaining disaster area remote sensing image segmentation map, labeling sentence description of the remote sensing image dataset for describing disaster area situation, and labeling visual question answering of the remote sensing image dataset for obtaining disaster area question and answer text; Step 2: Based on the labeled disaster area remote sensing image dataset, obtain network model parameters for solving three subtasks by training network model, specifically comprising: Migrate the classic semantic segmentation model to the semantic segmentation network model, and then perform semantic segmentation training on the remote sensing image dataset labeled with semantic segmentation data to obtain the optimal semantic segmentation network model parameters; the semantic segmentation training adopts the ordinary learning training method, divides the disaster area remote sensing image into small blocks, inputs the local area features generated by the semantic segmentation network model in batches, generates the semantic segmentation map of the local area of the image, and finally obtains the semantic segmentation result map through splicing; Serially connect the sentence description network model and the semantic segmentation network model, then perform sentence description training on the remote sensing image dataset labeled with sentence description to obtain the optimal sentence description network model parameters, and select the semantic segmentation model with the optimal performance in the semantic segmentation network model and the sentence description network model training for subsequent visual question answering network model training; the optimal semantic segmentation network model parameters are frozen in the sentence description training process, and the ordinary learning and reinforcement learning combined training method is adopted to fuse the generated local area features to obtain global features, and finally the global features are decoded to obtain image text description; Add the selected semantic segmentation model to the visual question answering network model training, and then perform visual question answering training on the remote sensing image dataset labeled with visual question answering to obtain the optimal visual question answering network model parameters; the sentence description network model and the visual question answering network model are in parallel relationship, and the sentence description is the upstream task of the visual question answering; the optimal semantic segmentation network model parameters and the optimal sentence description network model parameters are frozen in the visual question answering training process, and the ordinary learning and reinforcement learning combined training method is adopted; after generating the local area features on the remote sensing image dataset, the prior knowledge accumulated in the image generation sentence description text process of the sentence description training is used to guide the generation of the text features of the input question, and the image features and the text features are cross-modal fused to obtain the fused features, and finally the fused features are decoded to obtain the answer text of the question; Step three: the three sub-tasks of semantic segmentation, sentence description and visual question answering are solved by connecting and switching the network model, when the sentence description task needs to be solved, the semantic segmentation network model is connected with the sentence description network model to obtain the description text of the image; when the visual question answering task needs to be solved, the semantic segmentation network model is switched to be connected with the visual question answering network model to obtain the question and answer text.

2. The method of claim 1, wherein the method further comprises: The step of fusing the generated local region features to obtain global features includes: stacking all the generated local feature maps into a flat layer using a local feature joint coding structure to obtain output features, and performing linear projection and position coding on the output features to obtain a global feature map.

3. The method of claim 1, wherein the method further comprises: The step of cross-modal fusion of image features and text features includes: using a cross-modal fusioner to input the image features and text features into a joint attention layer for feature fusion after passing through n double-path feature generation layers, and outputting fused features; the cross-modal fusioner fuses and encodes the image features and text features based on the prior knowledge accumulated in the upstream sentence description; n is the number of double-path features.

4. The method of claim 3, wherein the method further comprises: The double-path features include text self-attention, image self-attention and image-guided text attention, wherein the image self-attention uses network model parameters obtained by upstream sentence description training, and the image-guided text attention process is: wherein, is a text feature, is an image feature, is a projection matrix for multi-head attention, is a reduced dimension matrix for multi-head attention, is a feed-forward network.

5. The method of claim 1, wherein the method further comprises: The cross-entropy loss is used to evaluate the pros and cons of sequence generation in the general learning process; the Beam Search algorithm is used to infer the generation of word sequences, and a sequence loss function is used to judge the pros and cons of sequence generation.

6. The method of claim 5, wherein the method further comprises: The sequence loss function in the sentence description training process is described as As a sequence evaluation index, the reinforcement learning loss function in the sentence description training process is: wherein, represents a predicted sequence; represents a corresponding target sequence; is the total number of sequences per batch; is the baseline function, i.e. the average of all sequence reward function values of the current batch; is the reward function indicator; is the sequence evaluation indicator.

7. The method of claim 5, wherein the method further comprises: The sequence loss function in visual question answering training uses WupLus Score as a sequence evaluation index, and the WupLus Score is: wherein, is the sequence length, is the number of elements in the sequence and the number of elements in the sequence most similar to in the sequence ; The reinforcement learning loss function in the visual question answering training process is: wherein, represents the original length of the generation sequence and the corresponding target sequence without padding values; represents the sequence elements contained in the generation sequence and the corresponding target sequence; represents the sequence element in the sequence ; represents the number of elements in the sequence ; represents the number of elements with the highest Wup Score for the element 8. A semantic description and question-answering system for remote sensing disaster detection, characterized in that, It includes: An image data collection and labeling module for network model training and evaluation, the image data set collection and labeling module includes labeling semantic segmentation data for remote sensing image data sets to obtain disaster area remote sensing image segmentation maps, labeling sentence descriptions for remote sensing image data sets to describe disaster area situations, and labeling visual question answering for remote sensing image data sets to obtain disaster area question and answer texts; A semantic segmentation network model module for solving the semantic segmentation task, the semantic segmentation network model module includes a remote sensing image input structure, an image segmenter, an image feature extraction model, and a segmentation head; a disaster area remote sensing image generates local image features through an image feature extraction model; A sentence description network model module for solving the semantic segmentation task, the sentence description network model module includes a local feature joint coding structure and a text decoder; the local feature joint coding structure is used to generate global image features from local image features, and the local feature joint coding structure includes a flattening layer, a linear projection layer, and a position coding layer; The visual question answering task network model module is used to solve the visual question answering task, and includes a question and answer text input structure, a text encoder, a local feature joint encoding structure, a cross-modal fusioner, and the text decoder; the cross-modal fusioner is used to fuse the local image features and the text features of the question to generate fusion features, and is an encoder based on accumulated prior knowledge in the sentence description to guide the generation and encoding of the fusion features, and includes n double-path feature generation layers and a joint attention layer; The network model connection module is used to switch the connection of the network models to solve the three subtasks of semantic segmentation, sentence description, and visual question answering; when the sentence description task needs to be solved, the semantic segmentation network model is connected with the sentence description network model to obtain the description text of the image; when the visual question answering task needs to be solved, the semantic segmentation network model is switched to be connected with the visual question answering network model to obtain the question and answer text. The network model training, learning, and evaluation module is used to obtain the optimal network model parameters for solving the three subtasks, and the training and learning includes ordinary learning and reinforcement learning; the ordinary learning process uses cross-entropy loss to evaluate the pros and cons of sequence generation; the reinforcement learning process uses the Beam Search algorithm to infer the generation of word sequences, and uses a sequence loss function to judge the pros and cons of sequence generation; the sequence loss function in the visual question answering training uses the modified WupLus Score as a sequence evaluation index.

9. A storage medium, characterized by The storage medium has a computer program stored thereon, and the computer program, when executed by a processor, causes the processor to perform the steps of the semantic description and question answering system establishment method for remote sensing disaster detection according to any one of claims 1 to 7.

10. A computer device, comprising: The computer device includes a memory and a processor, and the memory has a computer program stored thereon, and the computer program, when executed by the processor, causes the processor to perform the steps of the semantic description and question answering system establishment method for remote sensing disaster detection according to any one of claims 1 to 7.

Images