A method, device and medium for unified multi-modal terrain semantic segmentation and change detection

Abstract

The application discloses a method and device for unified multi-modal ground surface semantic segmentation and change detection, and a medium, and relates to the technical field of remote sensing earth observation and intelligent geographic information processing. The method uses an encoder based on a Siamese architecture to extract multi-scale features in each time phase and each mode; uses a multi-scale multi-modal fusion module to respectively perform cross-modal interactive fusion on deep heterogeneous features in the multi-scale features to generate fusion features in each time phase; uses a time correlation semantic segmentation decoder to model and decode the fusion features to obtain intermediate features and a ground surface semantic segmentation map; according to the fusion features and the intermediate features, uses a hierarchical difference aggregation module to generate change feature representation, combines the ground surface semantic segmentation map, and uses a change detection classifier to output a binary change detection map and a semantic change detection map. The application can realize parallel processing and feature sharing of multiple tasks in a unified technical framework, and improve interpretation accuracy in complex scenes.

Description

Technical Field

[0001] This application relates to the fields of remote sensing Earth observation and intelligent geographic information processing technology, and in particular to a unified multimodal surface semantic segmentation and change detection method, device and medium. Background Technology

[0002] Land surface semantic segmentation and change detection are two fundamental and crucial intelligent interpretation tasks in the field of remote sensing Earth observation. With the development of sensors and remote sensing technology, data acquisition has shown a significant multimodal trend (such as optical imagery, SAR imagery, and DSM). Different modalities of data are complementary in terms of imaging mechanisms, spatial resolution, and physical characteristics. However, most existing technologies suffer from the following shortcomings: First, Land Surface Semantic Segmentation (LSS), Binary Change Detection (BCD), and Semantic Change Detection (SCD) typically employ independent models, leading to significant duplication of computation; second, there is a lack of a unified multimodal fusion modeling approach, making it difficult to fully utilize the complementary information from heterogeneous data; and third, existing methods struggle to simultaneously address single-temporal semantic understanding and cross-temporal semantic change modeling. Therefore, there is an urgent need for a processing framework for multimodal land surface semantic segmentation and change detection, capable of collaboratively processing multiple tasks within a unified technical framework. Summary of the Invention

[0003] The purpose of this application is to provide a unified method, device and medium for multimodal surface semantic segmentation and change detection, which can realize parallel processing and feature sharing of multiple tasks under a unified technical framework, while improving the interpretation accuracy in complex scenarios.

[0004] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a unified method for multimodal surface semantic segmentation and change detection, including: Acquire dual-temporal, multimodal remote sensing data of the target area; Based on dual-temporal multimodal remote sensing data, an encoder based on the Siamese architecture was used to extract multi-scale features of each modality of remote sensing data in each temporal phase. Two multi-scale multimodal fusion modules are employed to perform cross-modal interactive fusion of deep heterogeneous features in the multi-scale features at each time phase, generating fused features for each time phase; the multi-scale multimodal fusion module adopts a self-query cross-attention mechanism; Using a temporally correlated semantic segmentation decoder, the fusion features under two temporal phases are modeled and decoded temporally correlated, and a semantic segmentation classifier is used to obtain intermediate features and surface semantic segmentation maps of the target area under each temporal phase. Based on the fusion features and intermediate features at each time phase, a hierarchical difference aggregation module is used to generate a representation of the change features; Based on the change feature representation and the surface semantic segmentation map at each time phase, a change detection classifier is used to output a binary change detection map and a semantic change detection map of the target region.

[0005] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the above-described unified multimodal surface semantic segmentation and change detection method.

[0006] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned unified multimodal surface semantic segmentation and change detection method.

[0007] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a unified method, device, and medium for multimodal surface semantic segmentation and change detection. It supports LSS, BCD, and SCD tasks simultaneously within a single framework, sharing the encoding and fusion stages, which significantly reduces computational redundancy. The SCB mechanism in the multi-scale multimodal fusion module adopts an attention strategy of same-modal query / key and dissimilar value, which is more suitable for multimodal data with low correlation but similar spatial relationships, achieving efficient fusion. The temporal correlation semantic segmentation decoder and hierarchical difference aggregation module are specifically designed for temporal correlation semantics and hierarchical differences, respectively, which significantly improves the interpretation accuracy in complex scenarios. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating a unified multimodal surface semantic segmentation and change detection method provided in this application embodiment; Figure 2 A schematic diagram of the overall structure of a unified multimodal surface semantic segmentation and change detection method provided in this application embodiment; Figure 3 A schematic diagram of the structure of the multi-scale multimodal fusion (MMF) module provided in the embodiments of this application; Figure 4A schematic diagram of the structure of the Cross Fusion Block (CFB) provided in the embodiments of this application; Figure 5 A schematic diagram of the processing flow of the Temporally Correlated Semantic Segmentation Decoder (TC-SS Decoder) provided in the embodiments of this application; Figure 6 A schematic diagram of the structure of the Hierarchy Discrepancy Aggregation (HDA) module and Discrepancy Aggregation (DA) block provided in the embodiments of this application; Figure 7 Example visualizations of input data on the SMARS-SCD dataset provided in this application embodiment; Figure 8 This application provides SCD result images for the SParis-50 scene in the SMARS-SCD dataset. Figure 9 This application provides SCD result images for the SMARS-SCD dataset SVenice-30 scene in this embodiment. Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] In one exemplary embodiment, such as Figure 1 As shown, a unified multimodal surface semantic segmentation and change detection method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method includes steps 101 to 106.

[0013] Step 101: Acquire dual-temporal multimodal remote sensing data of the target area.

[0014] Step 102: Based on the dual-temporal multimodal remote sensing data, use an encoder based on the Siamese architecture to extract multi-scale features of each modality of remote sensing data in each temporal phase.

[0015] Step 103: Two multi-scale multimodal fusion modules are used to perform cross-modal interactive fusion of deep heterogeneous features in the multi-scale features at each time phase to generate fused features at each time phase; the multi-scale multimodal fusion module adopts a self-query cross-attention mechanism.

[0016] Step 104: Utilize the temporal correlation semantic segmentation decoder to perform temporal correlation modeling and decoding on the fused features under dual temporal phases, and jointly obtain the intermediate features and the surface semantic segmentation map of the target area under each temporal phase using the semantic segmentation classifier.

[0017] Step 105: Based on the fusion features and intermediate features at each time phase, a change feature representation is generated using the hierarchical difference aggregation module.

[0018] Step 106: Based on the change feature representation and the surface semantic segmentation map at each time phase, use the change detection classifier to output the binary change detection map and semantic change detection map of the target area.

[0019] Steps 101 to 106 above form an end-to-end processing framework called DeepFuseNet, such as... Figure 2 As shown, by utilizing multi-scale multimodal fusion and hierarchical difference aggregation mechanisms, end-to-end unified processing of multiple tasks is achieved, solving the technical problems of model fragmentation, low accuracy, and computational redundancy in current multimodal remote sensing interpretation.

[0020] In another exemplary embodiment of this application, step 101 above involves the acquisition and preprocessing of multimodal and multitemporal data: acquiring multimodal remote sensing data of the target area at two different time points; performing spatial registration and normalization processing on the multimodal remote sensing data to obtain multimodal remote sensing data aligned in a unified coordinate system, and using it as dual-temporal multimodal remote sensing data of the target area.

[0021] Reference Figure 2 To obtain the same target area in time T 1 and T 2 Multimodal data (e.g., optical images) X and DSM / radar imagery Y The data undergoes spatial registration and normalization to align it in a unified coordinate system.

[0022] In another exemplary embodiment of this application, step 102 above involves feature extraction based on a Siamese encoder: using a weight-shared Siamese network to extract multi-scale features for each modality at different times. Assume... and These are data in different modalities, with their channel count × height × width being respectively... and Then for time t Input mode at time and Through encoding function Encoded as Time t Time scale s Embedded features and Encoding function It can be most mainstream architectures, such as CNN or Transformer.

[0023] In another exemplary embodiment of this application, the multi-scale multimodal fusion in step 103 above is used to fuse deep multi-scale multimodal features, and its structure is as follows: Figure 3 As shown. The multi-scale multimodal fusion module includes a first-level structure, a second-level structure, and a third-level structure connected in sequence; the first-level structure includes a first cross-fusion block, a first convolutional block, and a first upsampling block connected in sequence; the second-level structure includes a first normalization block, a second cross-fusion block, a second convolutional block, and a second upsampling block connected in sequence; both the first and second cross-fusion blocks employ a self-query cross-attention mechanism; the third-level structure includes a second normalization block, a third cross-fusion block, and a third convolutional block connected in sequence. (During each time phase...) The deep heterogeneous features in the scale features are processed by the first cross-fusion block, the first convolution block, and the first upsampling block, and then compared with each time phase. The deep heterogeneous features in the scale features undergo skip connections, addition, and normalization operations within the first normalization block to obtain the first intermediate fused features. These first intermediate fused features are then processed by the second cross-fusion block, the second convolutional block, and the second upsampling block, and then combined with features from each time phase. The deep heterogeneous features in the scale features are subjected to skip connections, addition and normalization operations in the second normalization block to obtain the second intermediate fusion features; the second intermediate fusion features are passed through the third cross fusion block and the third convolution block to output the fusion features at each time phase.

[0024] MMF can be divided into three stages, corresponding to three feature scales: , and Each stage contains a cross-fusion block and a convolutional block (Conv Block), specifically for the interaction and fusion of multimodal features. The general formula for each stage is: ; in AN ( ), CFB ( ), CB ( )and ups ( ) represent addition and normalization, cross-merging block, convolution block, and upsampling function, respectively; and This represents the interaction features at time scale s under time t. After the encoder generates multi-scale features, the three deepest scales are selected ( arrive The features of ) are passed to the MMF module, and arrive The multimodal features at these three scales are passed to the TC-SS decoder and HDA.

[0025] The structure of CFB is as follows: Figure 4 As shown, it consists of two self-query cross-attention blocks (SCBs) and a self-attention block between them. That is, both the first and second cross-fusion blocks include: a first self-query cross-attention block, a self-attention block, and a second self-query cross-attention block connected sequentially. The interaction fusion function of CFB and MMF is mainly contributed by the SCB. The self-attention block is used to stabilize and digest the multimodal features after interaction. Intermediate features and Output using the following formula: ; in, SCB ( )and SAB ( The symbols ) represent the self-query cross-attention block and the self-attention block, respectively. It should be noted that the attention mechanism in SCB differs from traditional cross-attention; the query in this structure... q s and keys k From the same modality, and the value v It comes from another modality and aims to retrieve information from another modality by leveraging the correlation of the modality itself. This is more suitable for multimodal data fusion with similar spatial relationships but low correlation.

[0026] In the MMF module, the first two stages conclude with an upsampling (UPS) operation, amplifying deep features by one level. The latter two stages begin with skip connections and addition & normalization (Add & Normal) operations to preserve and fuse the scale-specific features generated by the encoder. In the final stage, the multimodal features are concatenated into a single feature after interaction via CFB, and then refined through convolutional blocks to output the fused feature. The final stage can be represented as: ; in CAT ( ) indicates the concatenation operator.

[0027] After the encoder generates multi-scale features, this application only uses the three deepest scales ( arrive The features of ) are passed to MMF, and the three scales ( arrive The multimodal features are passed to the LSS and CD feature refinement modules (TC-SS decoder and HDA).

[0028] In another exemplary embodiment of this application, the time-related semantic segmentation decoder in step 104 above is as follows: Figure 5 As shown, this design aims to provide high-quality, temporally relevant semantic features for CD tasks. The temporally relevant semantic decoder inputs the fused features into the TC-SS decoder, enhances the semantic feature representation through temporal correlation modeling, and outputs a semantic segmentation map. The temporally relevant semantic segmentation decoder consists of a first cascaded layer and a second cascaded layer connected in sequence. The first cascaded layer performs the H1 stage, and the second cascaded layer performs the H0 stage.

[0029] In phase H1, fusion characteristics under dual temporal phases and The first output feature is then output through SCB and convolutional blocks, and then output in two temporal phases. and : .

[0030] Phase H0 includes the operations of phase H1, and features and Perform more convolutional processing: ; in, DC (·) indicates a dilated convolution block. , This represents the second output feature under dual-phase conditions. , , and All of these are used as intermediate features in the temporally correlated semantic segmentation decoder.

[0031] In the TC-SS decoder, bi-temporal features are modeled for temporal correlation using SCB, and then convolution is used to enhance the temporally related semantic features. In the H0 stage, considering the large feature size, dilated convolution blocks are used to expand the receptive field of the convolution and enhance multi-scale contextual information. The interaction of different temporal features also plays a role in data augmentation and feature space expansion to some extent. Bi-temporal semantic features ( and A semantic segmentation map of the land surface is obtained by convolutional blocks and upsampling back to the original image size. The structure of the Hierarchy Discrepancy Aggregation (HDA) module and the Discrepancy Aggregation (DA) block. In another exemplary embodiment of this application, the hierarchical difference aggregation structure in step 105 above is as follows: Figure 6 As shown in part (a), it absorbs multi-level features generated by the MMF and TC-SS decoders, and then enhances and aggregates semantic and differential features at different times and scales through multi-level DA blocks. The structure of the DA block is as follows: Figure 6 As shown in part (b), the general formula for the DA block is: ; in, and These represent the differential features and aggregate features, respectively, which are derived from... and The difference and connection are obtained. The designed HDA has three levels: H2, H1, and H0. (Scale) Dual-temporal fusion features and Initially, the input is fed into the first DA block after differencing and concatenation. The output is fused with multi-level features through convolutions in subsequent DA blocks, and the final difference features are output for BCD. . Obtaining the BCD image using regular convolutional blocks. It should be noted that SCB is not used in the HDA decoder, for the following reasons: 1) From a scale perspective... arrive 1) The feature size is large, leading to memory explosion due to cross-attention; 2) The main purpose of HDA is bi-temporal multi-scale feature aggregation, no longer focusing on feature refinement; 3) The feature dimensions are different. Even when using SCB, additional bi-directional feature transformation must be added, which will seriously reduce efficiency.

[0032] Therefore, the hierarchical difference aggregation module includes: a first multi-level difference aggregation block, a second multi-level difference aggregation block, and a third multi-level difference aggregation block. (Dual-phase) The fusion features at different scales are input into the first multi-level difference aggregation block after differencing and concatenation. After enhancement and aggregation, the first aggregated features are obtained and then input into the second multi-level difference aggregation block. (Dual-temporal) The scale-based fusion features, after differencing and concatenation, are input into the second multi-level differential aggregation block. Together with the first aggregation features, they are enhanced and aggregated to obtain the second aggregation feature, which is then input into the third multi-level differential aggregation block. (Dual-temporal) The scale-based fusion features are input into the third multi-level difference aggregation block after differencing and concatenation. Together with the second aggregation features, they are enhanced and aggregated to generate a variation feature representation.

[0033] In another exemplary embodiment of this application, Figure 2 In this context, SS Classifier represents the SS classifier, which is responsible for generating the predicted semantic map. CD Classifier represents the CD classifier, which is used to output a binary transformation map.

[0034] Multi-task result output: LSS output: Output from TC-SS decoder and SS classifier (semantic segmentation classifier) ; BCD output: Input CD classifier, output binary transformation map ; SCD Output: Combining the LSS and BCD results, the semantic change region (SCA) is obtained through masking operations, and the semantic change graph from class A to class B is determined. That is, in obtaining and back, By using mask Obtain, and then compare and Sure .

[0035] Therefore, the method for determining the semantic change detection map includes: obtaining the semantic change region through masking operation based on the surface semantic segmentation map and the binary change detection map under two temporal phases; and determining the semantic change detection map under two temporal phases based on the semantic change region.

[0036] In another exemplary embodiment of this application, after constructing an encoder based on the Siamese architecture, a multi-scale multimodal fusion module, a temporally correlated semantic segmentation decoder, a semantic segmentation classifier, a hierarchical difference aggregation module, and a change detection classifier, the above model is trained, and four types of loss functions are designed, including semantic segmentation loss. Binary change detection loss Semantic consistency loss and semantic change detection loss The network is trained using a joint loss function that includes four loss components. The total loss function is: ; In the formula, This indicates the total loss.

[0037] Semantic segmentation loss The cross-entropy loss between the temporally predicted semantic segmentation map and the labeled target semantic segmentation map can be estimated. That is: ; in This indicates the expected calculation. , This represents the cross-entropy loss under two time phases. , This represents a semantic segmentation map of the land surface predicted under two temporal phases. , This represents a semantic segmentation map of the land surface labeled under two temporal phases.

[0038] Binary change detection loss The binary change plots of the prediction and the target can be used. and The binary cross-entropy loss yields: ; in, This represents a binary change detection map indicating the prediction. This represents the labeled binary change detection map.

[0039] Semantic consistency loss It can be done and estimate: ; By making bi-temporal semantic predictions as consistent as possible in unchanged regions, but minimizing the similarity of bi-temporal semantic predictions in changed regions, LSS and BCD are synergistically driven to achieve high accuracy.

[0040] Semantic change detection loss Through predicted semantic change graphs and target calculate, ,in: ; This represents the semantic change detection loss value for all pixels in the entire image. This represents the semantic change detection map for prediction. This represents the semantic change detection graph of the annotation. γ It is a balance coefficient with a range of (0, +∞).

[0041] These four types of loss functions can provide comprehensive supervision signals and guide DeepFuseNet to achieve excellent results in LSS, BCD, and SCD tasks.

[0042] The steps of this application method are as follows: (1) acquiring dual-temporal multimodal remote sensing data and preprocessing it; (2) extracting multi-scale features of each temporal phase and each modality using a Siamese encoder; (3) constructing a multi-scale multimodal fusion module to perform cross-modal interactive fusion of deep heterogeneous features; (4) constructing a temporal-related semantic segmentation decoder to output dual-temporal surface semantic segmentation maps; (5) constructing a hierarchical difference aggregation module to extract and aggregate cross-temporal difference features; and (6) collaboratively outputting binary change detection maps and semantic change detection maps. This application realizes parallel processing and feature sharing of multiple tasks through a unified framework, improves the efficiency of multimodal fusion by using a self-query cross-attention mechanism, and solves the problems of task fragmentation, computational redundancy, and low change detection accuracy in complex scenarios in current remote sensing interpretation models.

[0043] The performance of DeepFuseNet is then validated using the SMARS-SCD dataset.

[0044] Dataset description: To overcome the lack of multimodal SCD datasets and the absence of datasets that simultaneously support multimodal SCD, LSS, and BCD tasks, this example presents a high-quality SMARS-SCD dataset based on the SMARS dataset. This dataset offers two different spatial resolutions: 30 cm and 50 cm. It categorizes urban land cover into five classes: Buildings, Streets, Trees, Lawns, and Others. The 30 cm resolution raster size for both SParis and Svenice is 5600×5600 pixels, while the 50 cm resolution raster sizes are 4500×3560 pixels and 5600×5600 pixels, respectively.

[0045] Experimental setup: HRNet-W30 was used as the encoder, the input image size was 512×512, and SGD was used as the optimizer.

[0046] Experimental results: Multi-task performance: DeepFuseNet achieved the highest accuracy on SCD, BCD, and LSS tasks. For example, in the SParis-30 scenario, the Fscd metric for the SCD task reached 93.23%, significantly outperforming the single-modal method SAAN (89.28%).

[0047] like Figure 7 As shown, DeepFuseNet can clearly identify small objects (such as trees), and the boundaries of changing regions are more precise, with accurate semantic change judgment.

[0048] The proposed DeepFuseNet achieves the highest accuracy across all scenarios, and its outputs for SCD, BCD, and LSS outperform other methods. Overall, it effectively identifies small objects, refines linear features, accurately defines the boundaries of changing regions, and precisely identifies semantic changes. Results are as follows... Figure 8 As shown in Tables 1 and 2.

[0049] Table 1. Accuracy statistics of SCD results in the SParis-50 scenario of the SMARS-SCD dataset.

[0050] Table 2. Accuracy statistics of SCD results in the SMARS-SCD dataset Svenice-30 scenario.

[0051] Next, this example evaluates the performance of the BCD task on the SMARS-SCD dataset. Similarly, for DeepFuseNet, this example only retains the binary change detection loss. DeepFuseNet achieves the highest BCD accuracy without using other losses; the comparison method is also specifically designed for BCD tasks. Accuracy on the SParis-30 and Svenice-30 datasets is listed in Table 3. It can be observed that DeepFuseNet's cIoU is 0.48 and 1.94 higher than the best metrics of other methods on the SParis-30 and Svenice-30 scenarios, respectively, indicating that DeepFuseNet significantly outperforms existing multimodal and unimodal BCD methods.

[0052] Table 3. Accuracy statistics of BCD results in SParis-30 and SVenice-30 scenarios of the SMARS-SCD dataset.

[0053] Finally, this example tests the performance of the LSS task on the SMARS-SCD dataset. It's worth noting that for DeepFuseNet, this example only retains the semantic segmentation loss. Without using other losses, optimal LSS performance is achieved; the comparison method is also specifically designed for LSS tasks. Representative results on SParis-30 and Svenice-30 are listed in Table 4. Figure 9 As shown in Table 4, DeepFuseNet performs best, with its mIoU exceeding that of other methods by 1.87 and 0.19 in the SParis-30 and Svenice-30 scenarios, respectively.

[0054] Table 4. Accuracy statistics of LSS results in SParis-30 and SVenice-30 scenarios of the SMARS-SCD dataset.

[0055] Through the above implementation methods, this application can effectively reduce system complexity while ensuring interpretation accuracy, and is suitable for multimodal remote sensing surface semantic segmentation and change detection applications in a wide range and multiple scenarios.

[0056] Compared with the prior art, this application has the following beneficial effects: 1. Unified framework: DeepFuseNet supports LSS, BCD and SCD tasks simultaneously within a single framework, sharing the encoding and fusion stages, which significantly reduces computational redundancy.

[0057] 2. Efficient Fusion: The SCB mechanism in the MMF module adopts an attention strategy of same-modal query / key and different-modal value, which is more suitable for the fusion of multimodal data with low correlation but similar spatial relationships.

[0058] 3. Refined Modeling: The TC-SS decoder and HDA module are specifically designed for temporal semantics and hierarchical differences, which significantly improves the interpretation accuracy in complex scenarios.

[0059] Based on the same inventive concept, this application also provides a unified multimodal land surface semantic segmentation and change detection system for implementing the unified multimodal land surface semantic segmentation and change detection method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more unified multimodal land surface semantic segmentation and change detection system embodiments provided below can be found in the limitations of the unified multimodal land surface semantic segmentation and change detection method described above, and will not be repeated here.

[0060] In one exemplary embodiment, a unified multimodal land surface semantic segmentation and change detection system is provided, comprising: a data acquisition module, an encoding module, a multi-scale multimodal fusion module, a semantic segmentation module, a differential feature modeling module, and a classification output module.

[0061] The data acquisition module is used to acquire dual-temporal multimodal remote sensing data of the target area.

[0062] The encoding module is used to extract multi-scale features of each modality of remote sensing data in each temporal phase using an encoder based on the Siamese architecture, based on dual-temporal multimodal remote sensing data.

[0063] The multi-scale multimodal fusion module is used to perform cross-modal interactive fusion of deep heterogeneous features in the multi-scale features at each time phase using two multi-scale multimodal fusion modules to generate fused features at each time phase; the multi-scale multimodal fusion module adopts a self-query cross-attention mechanism.

[0064] The semantic segmentation module is used to perform temporal correlation modeling and decoding of fused features under two temporal phases using a temporal correlation semantic segmentation decoder, and to obtain intermediate features and surface semantic segmentation maps of the target region under each temporal phase in conjunction with a semantic segmentation classifier.

[0065] The difference feature modeling module is used to generate a change feature representation based on the fusion features and intermediate features at each time phase using the hierarchical difference aggregation module.

[0066] The classification output module is used to output a binary change detection map and a semantic change detection map of the target area based on the change feature representation and the surface semantic segmentation map at each time phase, using a change detection classifier.

[0067] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores binary change detection maps and semantic change detection maps of the target region. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a unified multimodal surface semantic segmentation and change detection method.

[0068] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 10 The examples show more or fewer components, combinations of certain components, or different component arrangements. For example, a computer device is provided, including a memory and a processor, the memory storing a computer program, which the processor executes to implement the steps in the above-described method embodiments.

[0069] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0070] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0071] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0073] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0074] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A unified multimodal surface semantic segmentation and change detection method, characterized in that, include: Acquire dual-temporal, multimodal remote sensing data of the target area; Based on dual-temporal multimodal remote sensing data, an encoder based on the Siamese architecture was used to extract multi-scale features of each modality of remote sensing data in each temporal phase. Two multi-scale multimodal fusion modules are used to perform cross-modal interactive fusion of deep heterogeneous features in the multi-scale features at each time phase, generating fused features at each time phase; The multi-scale multimodal fusion module employs a self-query cross-attention mechanism; Using a temporally correlated semantic segmentation decoder, the fusion features under two temporal phases are modeled and decoded temporally correlated, and a semantic segmentation classifier is used to obtain intermediate features and surface semantic segmentation maps of the target area under each temporal phase. Based on the fusion features and intermediate features at each time phase, a hierarchical difference aggregation module is used to generate a representation of the change features; Based on the change feature representation and the surface semantic segmentation map at each time phase, a change detection classifier is used to output a binary change detection map and a semantic change detection map of the target region.

2. The method for unified multimodal surface semantic segmentation and change detection according to claim 1, characterized in that, Acquire dual-temporal, multimodal remote sensing data of the target area, including: Acquire multimodal remote sensing data of the target area at two different time points; Spatial registration and normalization are performed on multimodal remote sensing data to obtain multimodal remote sensing data aligned in a unified coordinate system, which is then used as bi-temporal multimodal remote sensing data for the target area.

3. The method for unified multimodal surface semantic segmentation and change detection according to claim 1, characterized in that, The multi-scale multimodal fusion module includes a first-level structure, a second-level structure, and a third-level structure connected in sequence. The first-level structure includes a first cross-fusion block, a first convolution block, and a first upsampling block connected in sequence; The second-level structure includes a first normalization block, a second cross-fusion block, a second convolution block, and a second upsampling block connected in sequence; both the first cross-fusion block and the second cross-fusion block adopt a self-query cross-attention mechanism. The third-level structure consists of a second normalized block, a third cross-fusion block, and a third convolutional block connected in sequence; Each time phase The deep heterogeneous features in the scale features are processed by the first cross-fusion block, the first convolution block, and the first upsampling block, and then compared with each time phase. The deep heterogeneous features in the scale features undergo skip connections, addition, and normalization operations within the first normalization block to obtain the first intermediate fused features. These first intermediate fused features are then processed by the second cross-fusion block, the second convolutional block, and the second upsampling block, and then combined with features from each time phase. The deep heterogeneous features in the scale features are subjected to skip connections, addition and normalization operations in the second normalization block to obtain the second intermediate fusion features; the second intermediate fusion features are passed through the third cross fusion block and the third convolution block to output the fusion features at each time phase.

4. The method for unified multimodal surface semantic segmentation and change detection according to claim 3, characterized in that, Both the first cross-fusion block and the second cross-fusion block include: a first self-query cross-attention block, a self-attention block and a second self-query cross-attention block connected in sequence; The query in the first self-query cross-attention block and the second self-query cross-attention block q s and keys k From the same modality, and the value v From another modality.

5. The method for unified multimodal surface semantic segmentation and change detection according to claim 1, characterized in that, The temporal-related semantic segmentation decoder includes a first cascaded layer and a second cascaded layer connected in sequence; The execution formula for the first cascade layer is: ； In the formula, and This indicates the fusion characteristics under two time phases. and This represents the first output feature under dual-phase conditions. SCB ( ) indicates a self-query cross attention block. CB ( ) represents a convolution block. ups ( ) indicates an upsampling block; The execution formula for the second cascade layer is: ； In the formula, , This represents the second output feature under dual-phase conditions. DC ( ) represents a dilated convolution block. , , and All of these are used as intermediate features in the temporally correlated semantic segmentation decoder; and A semantic segmentation map is obtained by convolutional blocks and upsampling back to the original image size.

6. The method for unified multimodal surface semantic segmentation and change detection according to claim 1, characterized in that, The hierarchical difference aggregation module includes: a first multi-level difference aggregation block, a second multi-level difference aggregation block, and a third multi-level difference aggregation block; Dual-phase The fusion features at different scales are input into the first multi-level difference aggregation block after difference and connection. After enhancement and aggregation, the first aggregated features are obtained and then input into the second multi-level difference aggregation block. Dual-phase The fusion features at different scales are input into the second multi-level difference aggregation block after difference and connection. Together with the first aggregation features, they are enhanced and aggregated to obtain the second aggregation features, which are then input into the third multi-level difference aggregation block. Dual-phase The scale-based fusion features are input into the third multi-level difference aggregation block after differencing and concatenation. Together with the second aggregation features, they are enhanced and aggregated to generate a variation feature representation.

7. The method for unified multimodal surface semantic segmentation and change detection according to claim 1, characterized in that, The method for determining the semantic change detection map includes: Based on the surface semantic segmentation map and binary change detection map under dual temporal phases, the semantic change region is obtained through masking operation; Based on the semantic change region, a semantic change detection map under dual temporal phases is determined.

8. The method for unified multimodal surface semantic segmentation and change detection according to claim 1, characterized in that, The total loss function used when training the encoder, multi-scale multimodal fusion module, temporally correlated semantic segmentation decoder, semantic segmentation classifier, hierarchical difference aggregation module, and change detection classifier based on the Siamese architecture is: ； In the formula, Indicates the total loss. Represents semantic segmentation loss. This represents the loss for binary change detection. This represents the semantic consistency loss. This represents the loss for semantic change detection; ； in, This indicates the expected calculation. , This represents the cross-entropy loss under two time phases. , This represents a semantic segmentation map of the land surface predicted under two temporal phases. , A semantic segmentation map of the land surface labeled with two temporal phases; ； in, This represents a binary change detection map indicating the predicted changes. This represents the labeled binary change detection map; ； ； ； in, This represents the semantic change detection loss value for all pixels in the entire image. This represents the semantic change detection map for prediction. This represents the semantic change detection map with annotations. This represents the balance coefficient.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the unified multimodal surface semantic segmentation and change detection method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the unified multimodal surface semantic segmentation and change detection method as described in any one of claims 1-8.

Images