A method, device and equipment for predicting urban downstream tasks

Abstract

The application discloses a city downstream task prediction method, device and equipment, the method comprises the steps of: acquiring multi-modal remote sensing data; the multi-modal remote sensing data is encoded to the same feature space, and the unified vectorization representation obtained is used to jointly pre-train the multi-modal pre-training model constructed in advance to determine the weight of the shared encoder; the shared encoder is used to extract features from target city multi-modal target remote sensing data to obtain a city feature set; the city feature set is fused into a fusion feature representation for a specified downstream task through a feature fusion network; and the fusion feature representation is input into an independent prediction head of the specified downstream task to obtain a prediction result. Thus, the multi-modal remote sensing data is encoded to the same feature space, the heterogeneity is eliminated, and the shared encoder weight is obtained through joint pre-training, so as to extract city features by using the shared encoder, fully mine the complementary information between multi-modal data, and improve the prediction accuracy of the downstream task.

Description

Technical Field

[0001] This application relates to the field of remote sensing data processing technology, and in particular to a method, apparatus and equipment for predicting downstream tasks in cities. Background Technology

[0002] As the core carrier of human activities, cities are highly coupled with resource consumption, environmental changes, and socio-economic evolution. Remote sensing technology, with its ability to observe the Earth's surface over large areas and periodically, has become a crucial data source for key applications such as urban planning, environmental monitoring, and carbon emission accounting. With the development of sensor technology, urban remote sensing data exhibits significant multimodal characteristics, encompassing heterogeneous information from multiple sources, including optical imagery, building heights, land cover maps, surface temperature grids, point-of-interest (POI) semantic maps, demographic grids, and GDP data. This allows for the characterization of urban space from multiple dimensions, including visual, structural, semantic, physical, and socio-economic perspectives.

[0003] However, current urban remote sensing data analysis primarily involves independent modeling of single or a few modalities, lacking a unified framework for processing multi-source heterogeneous data such as images, point sets, and statistics, thus failing to fully extract complementary information. Regarding remote sensing pre-training, pre-training mainly focuses on optical images of the natural surface, without designing mechanisms for non-image modalities (such as POIs and building heights) and macroscopic statistical attributes across resolutions and semantic granularities in urban scenes. This results in a semantic gap between pre-trained features and downstream tasks such as carbon emission estimation and population prediction, leading to low transfer efficiency.

[0004] Furthermore, multimodal fusion often employs fixed weight splicing or weighting, which makes it difficult to dynamically adjust according to task type (such as carbon emissions focusing on altitude characteristics, and heat island effect focusing on surface temperature) or regional characteristics (commercial areas and suburbs), thus limiting generalization ability and resulting in insufficient prediction accuracy.

[0005] Therefore, how to make full use of the complementary information between multi-source heterogeneous remote sensing data, avoid modal information fragmentation, and improve the prediction accuracy of downstream tasks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, one aspect of this application provides a method for predicting downstream tasks in cities, the method comprising: Acquire multimodal remote sensing data; The multimodal remote sensing data are encoded into the same feature space to obtain a unified vectorized representation; The pre-built multimodal pre-trained model is jointly pre-trained using the unified vectorized representation to determine the weights of the shared encoder in the multimodal pre-trained model. Using the shared encoder, feature extraction is performed on multimodal remote sensing data of the target city to obtain a city feature set; The urban feature set is fused into a fused feature representation for a specified downstream task through a feature fusion network; the specified downstream task includes at least carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. The fused feature representation is input into the independent prediction head of the specified downstream task to obtain the prediction result.

[0007] Optionally, the multimodal remote sensing data is encoded into the same feature space to obtain a unified vectorized representation, including: Based on preset classification rules, the multimodal remote sensing data is divided into multiple data categories, including category A, category B, and category C. Category A data has a higher spatial resolution than category B data, and category C data is global scalar data without spatial distribution information. The data of type A is segmented and embedded to obtain the first segmented label sequence; Based on the alignment to the first block label sequence, a cross-modal alignment embedding operation is performed on the B-type data to obtain the second block label sequence; Perform a global projection operation on the C-type data to obtain a global label sequence with the same feature dimensions as the first block label sequence; The global tag sequence is used as the head, and it is concatenated with the first block tag sequence and the second block tag sequence to obtain the unified vectorized representation.

[0008] Optionally, the B-type data includes semantic feature maps of interest points; a cross-modal aligned embedding operation is performed on the B-type data to obtain a second block label sequence, including: Map the point location data of the point of interest in the semantic feature map to a spatial grid of the same size as the downsampled data of class A. Extract the semantic embedding vector of each interest point in the semantic feature map of interest points, and perform mean pooling on the semantic embedding vectors within the same spatial grid to obtain the aggregated semantic vector; For target spatial grids in the spatial grid where the amount of interest point data is less than the threshold, a filling vector is generated and filled using the land cover category label at the corresponding location in the land cover segmentation map of the Class A data; The aggregated semantic vector and the padding vector are projected onto the same feature dimension and superimposed with the same positional encoding as the first block label sequence to obtain the second block label sequence corresponding to the interest point semantic feature map.

[0009] Optionally, the Class B data includes a land surface temperature grid; a cross-modal alignment embedding operation is performed on the Class B data to obtain a second block label sequence, including: The surface temperature grid is resampled to a spatial grid of the same size as the downsampled Class A data, and the average temperature within the corresponding ground area is recorded for each spatial grid. The average temperature is normalized to obtain a normalized temperature value. The normalized temperature value is projected onto the same feature dimension and superimposed with the same position code as the first block label sequence to obtain the second block label sequence corresponding to the surface temperature grid.

[0010] Optionally, joint pre-training of a pre-built multimodal pre-trained model may be performed, including: A first mask rate is applied to the first block marker sequence, and a second mask rate is applied to the second block marker sequence to obtain a masked block marker sequence; the first mask rate is greater than the second mask rate. The visible markers in the mask block marker sequence are concatenated with the global marker sequence, and then input into the shared encoder for joint encoding to obtain the output sequence. The output sequence is split according to the data types of the multiple data types, and then input into the decoder corresponding to each data type to perform loss reconstruction to obtain the reconstruction loss; Based on the reconstruction loss, the overall optimization objective is determined; and the multimodal pre-trained model is jointly trained with the goal of minimizing the overall optimization objective.

[0011] Optionally, the output sequence includes spatial label features corresponding to the A-class data and the B-class data; based on the reconstruction loss, the overall optimization objective is determined, including: Global average pooling is performed on the spatial label features to obtain the global mean of the spatial features; The global mean of the spatial features is input into the auxiliary prediction head of the C-class data to predict the corresponding scalar value, thereby obtaining the prediction loss. Based on the reconstruction loss and the prediction loss, the overall optimization objective is constructed.

[0012] Optionally, the B-type data includes an interest point semantic feature map, and the corresponding decoder is an interest point semantic map decoder; the reconstruction target of the interest point semantic map decoder is an aggregated semantic vector, and the corresponding reconstruction loss is a cosine similarity loss.

[0013] Optionally, the C-type data includes numerical scalars, categorical scalars, and text scalars; The prediction target of the auxiliary prediction head corresponding to the numerical scalar is the data normalized scalar value, and the corresponding prediction loss is the mean square error loss. The categorical scalar corresponds to the auxiliary prediction head whose prediction target is the category label, and the corresponding prediction loss is the cross-entropy loss. The prediction target of the text-type scalar corresponding auxiliary prediction head is the text semantic embedding vector, and the corresponding prediction loss is the cosine similarity loss.

[0014] Optionally, the city feature set includes a spatial feature grid and a semantic feature vector set; feature extraction is performed on the multimodal remote sensing data of the target city, including: Using the target remote sensing data as input, the step of encoding the multimodal remote sensing data into the same feature space to obtain a unified vectorized representation is performed to obtain a target vectorized representation; The target is vectorized and input into the shared encoder with the weights frozen; and the labeled feature vector output from the last layer of the shared encoder is extracted; the labeled feature vector includes the first labeled feature vector corresponding to the A-class data and the B-class data, and the second labeled feature vector corresponding to the C-class data; The first labeled feature vector is rearranged into a spatial feature map according to its spatial position, and the spatial feature maps are stacked into the spatial feature grid. Based on the second marker feature vector, the semantic feature vector group is constructed.

[0015] Optionally, the feature fusion network includes a gating network and an expert network group; the feature fusion network fuses the city feature set into a fused feature representation oriented towards a specified downstream task, including: Obtain the task type code of the specified downstream task, and extract target features from the city feature set; the target features include regional spatial query features and city global semantic features; the expert network group includes multiple expert networks that correspond one-to-one with the target features; The task type encoding, the regional spatial query features, and the city global semantic features are input into the gating network to determine the fusion weights corresponding to each data type. The corresponding target features are independently transformed by each of the expert networks to obtain the output features; Based on the fusion weights, the output features are weighted and summed to obtain the fusion feature representation.

[0016] Another aspect of this application provides a predictive apparatus for downstream urban tasks, the apparatus comprising: The remote sensing data acquisition module is used to acquire multimodal remote sensing data; The data unification processing module is used to encode the multimodal remote sensing data into the same feature space to obtain a unified vectorized representation; The joint training module is used to jointly pre-train a pre-built multimodal pre-trained model using the unified vectorized representation to determine the weights of the shared encoder in the multimodal pre-trained model. The feature extraction module is used to extract features from the multimodal remote sensing data of the target city using the shared encoder to obtain a city feature set. The feature fusion module is used to fuse the urban feature set into a fused feature representation for a specified downstream task through a feature fusion network; the specified downstream task includes at least carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. The task prediction module is used to input the fused feature representation into the independent prediction head of the specified downstream task to obtain the prediction result.

[0017] Another aspect of this application provides an electronic device including a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the computer program to implement the steps of the prediction method for the downstream urban task.

[0018] The method, apparatus, and equipment for predicting downstream tasks in cities provided in this application have the following beneficial effects: multimodal remote sensing data is encoded into the same feature space, fully utilizing the complementary information between multimodal data to eliminate heterogeneity, and shared encoder weights are obtained through joint pre-training to extract urban features using the shared encoder. By integrating the advantages of each modality, high-precision prediction of downstream tasks can be achieved. In other words, it fully leverages the complementary information between modal remote sensing data to improve the prediction accuracy, stability, and reliability of downstream tasks. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a method for predicting downstream urban tasks provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for predicting downstream urban tasks, provided as another embodiment of this application; Figure 3 A schematic diagram illustrating the principle of a method for predicting downstream urban tasks provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the principle of a method for predicting downstream urban tasks, provided as another embodiment of this application; Figure 5 A schematic diagram illustrating the principle of a method for predicting downstream urban tasks, provided as another embodiment of this application; Figure 6 A flowchart illustrating a method for predicting downstream urban tasks, provided as another embodiment of this application; Figure 7 A schematic diagram illustrating the principle of a method for predicting downstream urban tasks provided in another embodiment of this application; Figure 8 A schematic diagram of the structure of a predictive device for downstream urban tasks provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0020] The reference numerals in the attached diagram are as follows: 80 is the remote sensing data acquisition module, 81 is the unified data processing module, 82 is the joint training module, 83 is the feature extraction module, 84 is the feature fusion module, 85 is the task prediction module, 90 is the memory, 91 is the processor, 92 is the display screen, 93 is the input / output interface, 94 is the communication interface, 95 is the power supply, 96 is the communication bus, 901 is the computer program, 902 is the operating system, and 903 is the data. Detailed Implementation

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0023] Figure 1 This is a flowchart illustrating a method for predicting downstream urban tasks provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: S10: Acquire multimodal remote sensing data; In business scenarios such as urban planning, environmental monitoring, or carbon emission estimation, users need to use multi-source remote sensing data to perform quantitative analysis of target cities. Therefore, in this specific embodiment, multimodal remote sensing data is acquired. Multimodal remote sensing data may include, but is not limited to, optical RGB imagery, building height maps, semantic segmentation maps of ground features, semantic feature maps of points of interest, land surface temperature grids, total population, city GDP, climate zone type, and city name. This application does not limit the type and quantity of multimodal remote sensing data.

[0024] Furthermore, the acquisition of multimodal data can be achieved through publicly available remote sensing datasets, satellite imagery services, or urban statistics, and this application does not impose any restrictions on this method.

[0025] S11: Encode multimodal remote sensing data into the same feature space to obtain a unified vectorized representation; Because different modalities of data differ fundamentally in data structure, spatial resolution, and semantic granularity—for example, imagery is a regular raster, POIs are sparse point sets, and statistics are single-valued scalars—in order to achieve information complementarity between different modalities of data, the acquired multimodal remote sensing data is encoded into a unified feature space to obtain a unified vectorized representation.

[0026] Specifically, different processing methods can be used in parallel for different types of modal data, and the data can be split and aligned into a unified dimensional space to achieve alignment of different modal data. For example, for high-resolution spatial modalities, a patch token sequence can be obtained through a patch embedding operation. Furthermore, after preprocessing different types of data into token sequences, they are concatenated sequentially to obtain a unified vectorized representation, which can then be used as input for subsequent pre-training.

[0027] S12: Jointly pre-train the pre-built multimodal pre-trained model using a unified vectorized representation to determine the weights of the shared encoder in the multimodal pre-trained model; In one optional embodiment, a multimodal pre-trained model is pre-built, which includes a shared encoder and decoder. The model is then subjected to self-supervised joint pre-training using the unified vectorized representation output in step S11.

[0028] Specifically, during training, differential random masks are applied to each spatial modality. The encoder only processes visible tokens, while the decoder is responsible for reconstructing the masked parts. Simultaneously, a cross-modal auxiliary prediction task is designed for the global scalar modality, predicting each scalar value using the mean of the spatial features output by the encoder. By minimizing the weighted sum of the reconstruction loss and the auxiliary prediction loss, the encoder and decoder parameters are updated. After pre-training, the weights of the shared encoder are saved.

[0029] S13: Using a shared encoder, feature extraction is performed on the multimodal remote sensing data of the target city to obtain the city feature set; Based on the weight determination of the shared encoder described above, for the downstream target city to be analyzed, the original multimodal remote sensing data of the target city is first encoded into a unified vectorized representation according to the method in step S11. Then, the unified vectorized representation is input into the shared encoder after the weights are frozen, and forward inference is performed to extract all token features of the encoder's last layer output in order to obtain the city feature set related to the target description.

[0030] S14: Through a feature fusion network, the urban feature set is fused into a fused feature representation for a specified downstream task; wherein, the specified downstream task includes at least carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. In one alternative embodiment, the feature fusion network employs an adaptive fusion architecture based on a Mixture of Experts (MoE) network, comprising a gated network and a group of expert networks.

[0031] In a specific embodiment, when the feature fusion network performs feature fusion, the gating network simultaneously receives the task type encoding of the current downstream task, the regional spatial query features extracted from the city feature set (such as the global average pooling result of the RGB spatial feature map), and the city global semantic features (the mean of the global feature vector group), and dynamically calculates the fusion weights of each modality.

[0032] Each expert network in the expert network group independently transforms the features of its corresponding modality. Finally, the outputs of each expert network are weighted and summed according to the weights of the gating network output to obtain a multimodal fusion feature representation for the current task.

[0033] S15: Input the fused feature representation into the independent prediction head of the specified downstream task to obtain the prediction result.

[0034] Furthermore, each downstream task is configured with an independent task prediction head, which feeds the fused feature representation obtained in step S14 into the prediction head of the current task, outputting the final prediction result. In specific embodiments, downstream tasks may include, but are not limited to, urban carbon emission concentration estimation, population density prediction, and land surface temperature retrieval. In one optional embodiment, during downstream stage training, the pre-trained encoder weights are kept frozen, and only the feature fusion network and the task prediction head are trained, thereby reducing the required number of labeled samples.

[0035] Therefore, the urban downstream task prediction method provided in this application encodes multimodal remote sensing data into the same feature space, fully utilizes the complementary information between multimodal data, eliminates heterogeneity, and obtains shared encoder weights through joint pre-training. This allows the shared encoder to extract urban features, integrating the advantages of each modality to achieve high-precision prediction of downstream tasks. In other words, it fully leverages the complementary information between modal remote sensing data to improve the prediction accuracy, stability, and reliability of downstream tasks.

[0036] Figure 2 The flowchart illustrates a method for predicting downstream urban tasks according to another embodiment of this application. In an optional embodiment, such as... Figure 2 As shown, multimodal remote sensing data is encoded into the same feature space to obtain a unified vectorized representation, including: S20: Based on preset classification rules, multimodal remote sensing data is divided into multiple categories; the multiple categories include category A data, category B data, and category C data; category A data has a higher spatial resolution than category B data, and category C data is global scalar data without spatial distribution information; Figure 3 This is a schematic diagram illustrating the principle of a method for predicting downstream urban tasks provided in an embodiment of this application, as shown below. Figure 3 As shown, the prediction method for downstream urban tasks provided in this application can be divided into two stages. The first stage includes the preprocessing and pre-training of multimodal remote sensing data.

[0037] First, the first step is the preprocessing of the multimodal remote sensing data, which involves unifying the encoded values ​​of the multimodal remote sensing data into a unified feature space. Specifically, in one optional embodiment, the acquired multi-source heterogeneous remote sensing data is classified based on a preset classification rule, wherein the predicted classification rule can be based on the original spatial resolution and semantic granularity, resulting in... Figure 2 The data shown are classified into three categories: A, B, and C.

[0038] Type A data is high-resolution spatial modality, which may include, but is not limited to, RGB optical imagery, building height maps, and semantic segmentation maps of ground features. In one optional embodiment, the native spatial resolution of Type A data is sub-meter to meter level, and it has rich spatial detail information.

[0039] Type B data is a modality that includes spatial semantic information and spatial distribution information. It may include, but is not limited to, POI semantic feature maps and land surface temperature grids constructed based on Points of Interest (POI) data. The native spatial resolution of Type B data is from hundreds of meters to kilometers, and it has spatial distribution information, but the spatial resolution of Type B data is lower than that of Type A data.

[0040] Category C data includes global scalar data without spatial distribution information, which may include, but is not limited to, numerical scalars (such as continuous numerical data such as total regional population and city GDP), categorical scalars (such as discrete categorical data such as climate zone type and land use type), and textual scalars (such as free text data such as city name and regional function description). Category C data has no spatial distribution and describes the macroscopic attributes of a city or region in single-value form.

[0041] S21: Perform block embedding operation on the A-class data to obtain the first block label sequence; In one optional embodiment, the block embedding operation sequentially includes spatial registration, downsampling and cropping, patch segmentation, modality-specific linear projection, and position encoding. Specifically, during spatial registration, all Class A numbers are uniformly reprojected to the same coordinate reference system (such as UTM projection) and aligned to the same spatial grid at the target resolution, ensuring spatial consistency of different Class A numbers at the pixel level.

[0042] Then, downsampling and cropping are performed. Specifically, each modal data is cropped into pixel image blocks of a specified size (e.g., 1024×1024) that cover the same geographical area. It should be noted that this application does not limit the specified size and can be adjusted according to actual business needs. For ease of understanding, the specified size of 1024×1024 will be used as the description below.

[0043] Furthermore, the 1024×1024 pixel image patch is downsampled to 256×256 pixels using bilinear interpolation, making the ground resolution per pixel approximately four times that of the original resolution. Taking a 0.5-meter resolution RGB optical image as an example, after downsampling, each pixel corresponds to approximately 2 meters of ground, and each patch covers an area of ​​approximately 32 meters × 32 meters.

[0044] Then, patch segmentation is performed. Specifically, the 256×256 pixel image is uniformly divided into... There are 3 non-overlapping patches, each patch having a size of 1. The pixels yielded a total of 256 spatial patches. The spatial order of each patch was fully preserved, and the patch at position (i,j) naturally corresponded to the ground feature information of that spatial location.

[0045] In one optional embodiment of modal-specific linear projection, the pixel matrix of each patch is independently flattened into a one-dimensional vector and fed into a learnable linear projection layer specific to class A data, mapping it to a unified D-dimensional token space. It should be noted that the parameters of the projection layers for each class A data are not shared, in order to fully preserve the original feature distribution differences among the various class A data.

[0046] Specifically, the flattened dimension of a single patch in the RGB optical image is 16×16×3=768, and the flattened dimension of a single patch in the building height map and the feature segmentation map is 16×16=256. Each is independently linearly transformed and projected into D-dimensional space, see formula (1): (1) in, For the first The first A-class data The original data of each patch For the first The projection matrix of class A data. For the first The flattening dimension of Class A data. This corresponds to the bias term.

[0047] The above projection is performed independently on a single patch basis. Each of the 256 patches is processed by the same modal-specific projector, which outputs 256 dimensional tokens. The spatial arrangement order is completely consistent with the segmented patches, and the spatial position information is completely preserved.

[0048] Furthermore, a two-dimensional sine and cosine position code is superimposed after the Projector output to explicitly inject spatial position information, i.e., position code superposition is performed, see formula (2): (2) in, This refers to the row and column indices of the patch within a 16x16 grid. For encoding dimension index, This represents the total dimension of the token. The positional encoding is superimposed on the Projector output without undergoing linear transformation and does not affect the feature distribution of each modality.

[0049] After the above block embedding operation, the first Patch Token sequence of 256-dimensional space corresponding to each of the A-type data such as RGB optical image, building height map, and ground feature segmentation map can be output. Each A-type data is projected independently, and the spatial positions correspond one-to-one.

[0050] S22: Based on the alignment to the first block label sequence, perform cross-modal alignment embedding operation on the B-class data to obtain the second block label sequence; Based on the above-mentioned data processing for category A, such as Figure 3 As shown, based on the first Patch Token sequence, cross-modal alignment embedding operations are performed on Class B data such as POI semantic feature maps and surface temperature grids to obtain the second Patch Token sequence.

[0051] S23: Perform a global projection operation on the C-class data to obtain a global label sequence with the same feature dimensions as the first block label sequence; Simultaneously, a global projection operation is performed on the C-type global scalar data to obtain a global token sequence with the same feature dimensions as the first Patch Token sequence and the second Patch Token sequence. In a specific embodiment, such as... Figure 3 As shown, Class C data can be divided into numerical scalars, categorical scalars, and text scalars. Different global projection operations are performed on different types of Class C data.

[0052] Specifically, for C-type data of numerical scalars, such as continuous numerical scalars like total population and city GDP, Z-score normalization can be performed based on the training set statistics, see formula (3): (3) in, , These are the mean and standard deviation of this mode in the training set, respectively.

[0053] The normalized scalar value is processed through two independent MLP layers (linear layer - ReLU - linear layer). Projected onto a dimensional global token vector, where For hidden layer dimensions.

[0054] For C-class data of categorical scalars, such as discrete categorical scalars like climate zone type and land use type, a learnable categorical embedding matrix can be used. Map the category index to dimensional embedding vector, where This represents the total number of categories for this modality. After embedding, it passes through two independent MLP layers (...). Perform a nonlinear transformation to obtain A global token vector.

[0055] For Class C data, which consists of textual scalars such as city names and regional function descriptions, the same pre-trained text encoder as that used for Class B data is employed for extraction. 3D semantic embedding vectors, through independent linear domain adaptation layers ( Feature space alignment is performed to obtain a 3D global token sequence. The global tokens obtained from different C-class data do not have additional spatial location encoding, and the projection layer parameters of each C-class data are not shared.

[0056] S24: Using the global tag sequence as the head, concatenate it with the first block tag sequence and the second block tag sequence to obtain a unified vectorized representation.

[0057] Furthermore, such as Figure 3 As shown, the global token sequence corresponding to the C-class data is concatenated and placed at the beginning of the Patch Token sequence in the A-class and B-class data spaces. This serves as the city-level macro-semantic context injected into the attention calculations of each layer of the subsequent encoder, thereby obtaining a unified vectorized representation.

[0058] Table 1 is a schematic table of multimodal image data preprocessing results provided in the embodiments of this application. The following will be explained in conjunction with Table 1.

[0059] Table 1 is a schematic diagram of the preprocessing results of multimodal image data.

[0060] Table 1 shows the results of preprocessing multimodal influence data with a specified size of 1024×1024 as an example. Based on the above preprocessing, the concatenated unified vectorized representation can be obtained, that is, the unified token sequence, see formula (4): (4) Among them, the number of global token sequences The number of data items equals the number of Class C data items, and each Token is equal to the number of data items in Class C. 256 Patch Tokens from each modality of Class A and Class B data are perfectly aligned in spatial location and mapped to a unified vector through their respective modality-specific Linear Projectors. Token space.

[0061] Therefore, to address the heterogeneous characteristics of urban data across resolutions and semantic granularities, three parallel processing paths were designed to uniformly encode sub-meter-level optical imagery, 100-meter-level semantic grids, and city-level statistical scalars into a single... The 3D Token space enables efficient and unified representation of multi-source heterogeneous data, overcoming the limitations of existing methods that cannot simultaneously process data with different resolutions and semantic granularities.

[0062] In one alternative embodiment, such as Figure 3 As shown, class B data includes POI semantic feature maps. Therefore, a cross-modal aligned embedding operation is performed on class B data to obtain the second block label sequence, which includes: Map the point location data of the point of interest in the semantic feature map to a spatial grid of the same size as the downsampled data of class A; Extract the semantic embedding vector of each interest point in the semantic feature map, and perform mean pooling on the semantic embedding vectors within the same spatial grid to obtain the aggregated semantic vector; For target spatial grids where the amount of interest point data is less than the threshold, fill vectors are generated and filled using the land cover category labels at the corresponding locations in the land cover segmentation map of Class A data; The aggregated semantic vector and the padding vector are projected onto the same feature dimension and superimposed with the same positional encoding as the first block label sequence to obtain the second block label sequence corresponding to the interest point semantic feature map.

[0063] In a specific embodiment, after processing the POI semantic feature map, a 16×16 spatial grid identical to that of Class A data can be obtained. Specifically, in an optional embodiment, the POI semantic feature map is sequentially subjected to POI data spatialization, POI text encoding, semantic densening filling, and modality-specific linear domain adaptation projection.

[0064] In a specific embodiment of POI data spatialization, the POI point data within the city area corresponding to the multimodal image data are mapped to the same 16×16 spatial grid as the downsampled Class A data according to their geographic coordinates. Each grid cell covers the same ground area as one patch of the Class A data.

[0065] Further, POI text encoding is performed. Specifically, for the name and category text attributes of each POI, a pre-trained text encoder (such as BERT or its lightweight variant) is used to extract a semantic embedding vector. Then, mean pooling is performed on multiple semantic embedding vectors within the same grid cell to obtain the aggregated semantic vector of that grid cell.

[0066] Then, semantic desaturation is performed. Specifically, for target spatial grids with less than a threshold amount of POI data (i.e., for target spatial grids with sparse or missing POI data), the land cover category labels at corresponding locations in the land cover segmentation map of Class A data are read and downsampled to the same resolution. These land cover category labels are then input into the same text encoder to extract their semantic desaturation. The semantic embedding vector is used as the padding value at that position.

[0067] In a specific embodiment, this semantic dense padding strategy ensures that each position on the 16×16 spatial grid has an effective semantic vector, and uses the spatial semantic information of the segmentation map to make up for the uneven distribution of POI data, avoiding interference from zero padding to model training.

[0068] Finally, since each position in the POI semantic feature map is already... The text embedding vector does not require flattening, but the text embedding space and the feature space of the visual Patch Token have domain differences. Therefore, it needs to be aligned with the feature space by an independent learnable linear domain adaptation layer, that is, projecting the aggregated semantic vector and the padding vector to the same feature dimension, see formula (5): (5) in, For the first Aggregated semantic vectors (including dense padding) for each grid position. A domain-independent projection matrix is ​​adapted for the POI (Point of Interest). After projection, a two-dimensional positional encoding in the same form as that for Class A data is superimposed, resulting in 256 outputs for Class B data that are spatially perfectly aligned with Class A data. The POI semantic Patch Token sequence is the second Patch Token sequence corresponding to the POI semantic feature map.

[0069] In one alternative embodiment, such as Figure 3 As shown, Class B data includes surface temperature grids. Therefore, a cross-modal aligned embedding operation is performed on the Class B data to obtain a second block label sequence, including: The surface temperature grid was resampled to a spatial grid of the same size as the downsampled Class A data, and the average temperature over the corresponding ground area was recorded for each spatial grid. Normalize the average temperature to obtain the normalized temperature value; The normalized temperature values ​​are projected onto the same feature dimension and superimposed with the same location code as the first block label sequence to obtain the second block label sequence corresponding to the surface temperature grid.

[0070] In a specific embodiment, The surface temperature grid data was resampled to the same 16×16 spatial grid as the downsampled Class A data. Each grid cell recorded the average temperature over its corresponding surface area (where is in °C). Then, Z-score normalization was performed to obtain normalized temperature values, resulting in a 16×16 temperature scalar grid. It's understood that the temperature data is a continuous scalar, requiring no text encoding. The normalized scalar value at each grid location is then fed into a learnable linear projection layer specific to the temperature mode and mapped to... Dimensional Token space, see formula (6): (6) in, For the first Normalized temperature values ​​at each grid location. , These are temperature-modal independent projection parameters. After projection, a two-dimensional positional code in the same form as that of the Class A data is superimposed. Thus, the Class B data output has 256 parameters that are spatially perfectly aligned with the Class A data. The second-degree Patch Token sequence corresponds to the surface temperature grid.

[0071] In one optional embodiment, joint pre-training of a pre-built multimodal pre-trained model includes: A first mask rate is applied to the first block marker sequence, and a second mask rate is applied to the second block marker sequence to obtain a masked block marker sequence; the first mask rate is greater than the second mask rate. The visible markers in the mask block marker sequence are concatenated with the global marker sequence and then input into the shared encoder for joint encoding to obtain the output sequence. After splitting the output sequence according to the data types of multiple data, each data type is input into the decoder corresponding to the data type to perform loss reconstruction, and the reconstruction loss is obtained. Based on the reconstruction loss, the overall optimization objective is determined; and the multimodal pre-trained model is jointly trained with the goal of minimizing the overall optimization objective.

[0072] like Figure 2 As shown, based on the unified vectorized representation obtained in step 1, the token sequences of each modality output in step 1 are used as the input for step 2, that is, the multimodal pre-trained model is jointly trained through the unified vectorized representation. Specifically, differential random masks are first applied according to the modality type of the remote sensing data, and the masking processes for each modality are independent of each other.

[0073] For Class A high-resolution spatial modal data, a random mask is applied to each of the 256 spatial patch tokens in each modal (such as RGB imagery, building height maps, and feature segmentation maps), with the mask rate being a first mask rate. For example, in one optional embodiment, the first mask rate can be set between 40% and 60%, and as an optional embodiment, it can be set to 50%. Thus, approximately 128 visible tokens are randomly retained for each modality, while the remaining tokens are masked and not sent to the encoder.

[0074] For Class B spatial semantic / distributed modalities, each of the 256 Patch Tokens in the data (such as POI semantic maps and surface temperature grids) is also independently masked with a random mask rate of the second mask rate. In a specific embodiment, the relatively low mask rate is because the modal information density of Class B data is lower than that of Class A data. An excessively high mask rate would result in overly sparse visible tokens, which would be detrimental to the encoder learning effective semantic and spatial distribution patterns from the context. Therefore, as an optional embodiment, the first mask rate is greater than the second mask rate. For example, the second mask rate can be set to 20% to 40%, preferably 30%.

[0075] For the global scalar / administrative statistical modalities of type C, the global token is not masked and does not participate in mask reconstruction training.

[0076] It should be noted that, in specific embodiments, the mask position is selected by uniform random sampling without imposing any spatial or semantic constraints, so as to ensure that the encoder is forced to reconstruct the missing information from diverse context combinations during training, thereby improving the generalization ability of multimodal representation.

[0077] After random masking, the resulting masked Patch Token sequence consists of visible tokens. This sequence needs to be concatenated with the unmasked global token sequence, and the resulting unified sequence is then input into the shared encoder for joint encoding to obtain the output sequence. In an optional embodiment, the joint encoding is described in formula (7): (7) in, For the first A set of patch indices for spatial modalities that are not masked, and global tokens of class C are not masked and all participate in encoder calculation.

[0078] In an optional embodiment, the encoder employs a standard Vision Transformer (ViT) structure, consisting of stacked Transformer Blocks, each containing a multi-head self-attention (MHSA) mechanism, a feedforward neural network (FFN), and layer normalization (LayerNorm), as detailed in equations (8) and (9): (8) (9) in, For the first The output feature matrix of the layer, Visible tokens from all modalities share the same encoder weights. In each Self-Attention layer, tokens from different modalities can interact freely, achieving cross-modal information fusion. The encoder output sequence is of the same length as the input sequence, and each token corresponds to a one-dimensional semantic feature vector.

[0079] Understandably, the output sequence obtained through the above joint encoding contains token features from all modalities. It needs to be split according to different data types and modalities, and then sent to the dedicated decoder corresponding to each modal pair of data.

[0080] In one optional embodiment, for the spatial modalities of data types A and B, learnable Mask Tokens need to be inserted at the masked positions before being input into the decoder to complete the sequence to a full 256 positions before being sent to the decoder. However, for the global Patch Token sequence of data type C, since no masking is applied, the global Token features output by the encoder can be directly saved as city-level semantic features and are not sent to any decoder.

[0081] Specifically, for the first type of data and the second type of data... Each spatial modality, whose complete token sequence is constructed by position, is shown in formula (10): (10) in, For the encoder output of the first The modality of the first The feature vector of a visible token For the first Each modality-independent learnable Mask Token vector (not shared across modalities). For the first Two-dimensional sine and cosine positional encoding is used for each location. Overlay positional encoding enables the decoder to perceive the spatial coordinates of each masked location, thereby utilizing the contextual features of the visible tokens to perform targeted reconstruction of that location.

[0082] It is worth noting that when the complete token sequence after each modality is completed is sent to the corresponding dedicated decoder for reconstruction, each modality decoder is designed independently for its data type and the parameters are not shared between them.

[0083] Furthermore, after obtaining the reconstruction loss function, the overall optimization objective is determined, and the multimodal training model is jointly trained with the goal of minimizing the overall optimization objective. Thus, after training is complete, i.e., after the model training has converged, the weights of the shared encoder are obtained, and these weights are used for... Figure 3 The feature extraction in step 3 is shown.

[0084] Therefore, this application employs a shared Transformer encoder to jointly encode visible tokens from all modalities. Tokens from each modality fully interact during the self-attention computation at each layer of the encoder, achieving mutual enhancement of cross-modal semantic information. The differentiated mask rate design effectively adapts to the differences in information density between different modalities, avoiding problems such as excessive information loss or overly simplistic reconstruction tasks caused by a uniform mask rate.

[0085] Based on the above embodiments, as an optional embodiment, the output sequence includes spatial label features corresponding to class A data and class B data; based on the reconstruction loss, the overall optimization objective is determined, including: Global average pooling is performed on the spatial label features to obtain the global mean of the spatial features; The global mean of spatial features is input into the auxiliary prediction head of Class C data to predict the corresponding scalar value in order to obtain the prediction loss. Based on reconstruction loss and prediction loss, a total optimization objective is constructed.

[0086] It is understood that in the above embodiments, the global token sequence of class C data is not masked. However, for class C data, as an optional embodiment, a cross-modal assisted prediction task can be designed to predict the scalar values ​​of each class C by using the spatial token mean output by the encoder as input, thereby providing direct supervision signals for the representation learning of global tokens.

[0087] Specifically, a lightweight cross-modal auxiliary prediction head is configured for each Class C modality data to establish an explicit supervised association between spatial visual features and urban macro attributes. The input to the auxiliary prediction head is the global average pooling result of the spatial token features of all Class A and Class B data in the encoder output sequence, as shown in formula (11) to obtain the global mean of spatial features: (11) in, This represents the total number of spatial modal visible tokens in the current batch. For the corresponding index set. As input, the corresponding scalar value is predicted by a single-layer linear prediction head specific to each C-type modality to obtain the prediction loss.

[0088] Furthermore, the reconstruction loss corresponding to class A data and class B data, as well as the prediction loss corresponding to class C data, are weighted and summed, and the weighted summation is used as the overall optimization objective to achieve end-to-end joint training of the encoder, each dedicated decoder, and the auxiliary prediction head for class C data.

[0089] In one alternative embodiment, training may employ the AdamW optimizer, and the learning rate may use a warmup and cosine annealing scheduling strategy. The training data covers multimodal remote sensing data from multiple cities to improve the encoder's generalization ability to different urban scenarios.

[0090] After the model training converges, the weights of the shared Transformer encoder are saved for use. Figure 3In step 3, the weights of the modality-specific decoders and the auxiliary prediction heads for class C data are discarded after pre-training and do not participate in the downstream stages.

[0091] In one optional embodiment, the Class A data includes RDB optical imagery, building height maps, and feature segmentation maps, and the dedicated decoders for the corresponding Class A data include RDB optical imagery decoders, building height map decoders, and feature segmentation map decoders.

[0092] For the RDB optical image decoder, in one optional embodiment, a 4-layer lightweight Transformer decoder can be used to output 3D pixel prediction values ​​for the masked patch. The reconstruction target is the normalized value of the pixels in the original patch (the mean value minus the standard deviation for each pixel in the patch), and the reconstruction loss can be the normalized pixel mean square error, specifically, see formula (12): (12) in, The set of indices of the masked patch. The decoder's predicted value. This indicates the pixel normalization operation within the patch.

[0093] For the building height map decoder, in one optional embodiment, a 4-layer lightweight Transformer decoder can be used to output a single height regression value for the masked patch. The reconstruction target is the mean of the height values ​​of all pixels within the patch. The reconstruction loss can be the mean square error. For details, see formula (13): (13) in, The decoder's predicted value. This represents the average pixel height within the patch.

[0094] For the land cover segmentation map decoder, as an optional implementation, a 4-layer lightweight Transformer decoder can be used to predict the semantic category of the masked patch. The reconstruction target is the land cover category label (dominant category) with the largest proportion in the patch. The reconstruction loss can be the cross-entropy loss. For details, see formula (14): (14) in, For the first The dominant category label of the masked patch The decoder predicts the probability of this category.

[0095] In another alternative embodiment, the B-type data includes a POI semantic feature map and a land surface temperature grid, and the corresponding dedicated decoder for the B-type data includes a POI semantic feature map decoder and a land surface temperature grid decoder.

[0096] For the POI semantic feature map decoder, in one optional embodiment, a 2-layer MLP decoder can be used to predict the semantic vector output at the masked grid location and reconstruct the target. Figure 3 The aggregated POI semantic vector (including the dense padding result) at this position in step 1 is reconstructed using cosine similarity loss. For details, please refer to formula (15): (15) in, The semantic vector predicted by the decoder. The target is aggregated semantic vectors. Cosine similarity loss optimizes semantic direction consistency and is insensitive to the numerical scale of the vectors, making it suitable for semantic embedding space reconstruction tasks.

[0097] For a surface temperature grid decoder, in one optional embodiment, a two-layer MLP decoder can be used to output a single temperature regression value for the masked grid location, with the reconstruction target being... Figure 3 In step 1, the temperature scalar value at this location after Z-score normalization is used for reconstruction loss, which is calculated using mean square error. For details, please refer to formula (16). (16) in, The decoder's predicted value. The target normalized temperature value.

[0098] In one optional embodiment, Class C data includes numerical scalars, categorical scalars, and textual scalars. Therefore, in a specific embodiment, loss reconstruction needs to be performed separately for each type of Class C data.

[0099] Specifically, for numerical scalars, the prediction target of the auxiliary prediction head is the normalized scalar value of the data, and the corresponding prediction loss is the mean squared error loss. That is, the numerical scalar auxiliary prediction head uses a linear layer (…). The prediction target is Figure 3 The normalized scalar value corresponding to the mode in step 1 is shown. The prediction loss uses the mean square error. For details, please refer to formula (17): (17) in, To predict the output value of the head, For the target normalized scalar value, The number of modalities for numerical C-type data.

[0100] In one optional embodiment, the prediction target of the categorical scalar auxiliary prediction head is the category label, and the corresponding prediction loss is the cross-entropy loss. That is, the categorical scalar auxiliary prediction head uses a linear layer ( The prediction target is the class label, and the loss is cross-entropy. See formula (18) for details: (18) in, For category labels, To predict probabilities, This represents the number of modalities for categorical C-type data.

[0101] In another optional embodiment, the prediction target of the text-type scalar auxiliary prediction head is the text semantic embedding vector, and the corresponding prediction loss is the cosine similarity loss. That is, the text-type scalar auxiliary prediction head uses a linear layer ( The prediction target is Figure 3 The semantic embedding vectors corresponding to the textual scalars in step 1 are shown, and the loss is calculated using cosine similarity loss. For details, see formula (19): (19) in, To predict semantic vectors, For the target semantic vector, The number of modalities for text-based C-type data.

[0102] It should be noted that the physical meaning of the aforementioned auxiliary prediction task is that the encoder must extract discriminative representations from the spatial distribution of urban features, building morphology, and semantic features sufficient to infer macro-level urban attributes, which is highly consistent with the feature capabilities required for downstream urban analysis tasks (carbon emission estimation, population density prediction, etc.). Furthermore, it should be noted that the auxiliary prediction head has a very small number of parameters and is discarded along with the decoder after pre-training, not participating in downstream stages.

[0103] Table 2 is a schematic table of a loss reconstruction result provided in an embodiment of this application. The following will be explained in conjunction with Table 1.

[0104] Table 2 is a schematic diagram of one type of loss reconstruction result.

[0105] As shown in Table 2, different modal data in Class A data can be decoded using a 4-layer Transformer decoder, while different modal data in Class B data can be decoded using a 2-layer MLP decoder.

[0106] Based on this, the overall optimization objective is constructed by reconstructing the loss and predicting the loss. For details, please refer to formula (20): (20) in, These are the weighting coefficients for the reconstruction loss of each spatial mode. The unified weighting coefficients for the C-type auxiliary prediction loss can be set as fixed hyperparameters or dynamically adjusted using an automatic weighting strategy based on uncertainty, in order to avoid the loss of a certain mode dominating the overall optimization direction.

[0107] Therefore, in this embodiment, the multimodal masked autoencoder extends the MultiMAE framework. For multi-source heterogeneous remote sensing data, it adopts differentiated mask rates based on the semantic granularity and information density of each modality, rather than a uniform mask rate. Furthermore, no mask is applied to the global token sequence of Class C data; it participates in the attention calculation of each layer of the encoder as a complete global context throughout the process, fully leveraging the guiding role of urban macro-semantics in spatial representation learning.

[0108] Furthermore, this application introduces cosine similarity reconstruction loss for POI semantic modalities, using semantic vector direction rather than numerical precision for reconstruction. Addressing the limitations of global scalar modalities (population, GDP, climate zones, etc.) with their limited token quantity and unsuitability for mask reconstruction, a cross-modal auxiliary prediction task is designed. This task predicts scalar values ​​using the mean of spatial tokens output by the encoder, explicitly incorporating the semantic relationship between spatial visual features and urban macro-attributes into the pre-training objective. Compared to purely bypass-based implicit optimization, this design provides direct supervision for the representation learning of global tokens, and the learned features are highly consistent with the needs of downstream urban analysis tasks (carbon emission estimation, population density prediction, etc.).

[0109] Figure 4 This is a schematic diagram illustrating the principle of a prediction method for downstream urban tasks provided in another embodiment of this application. Based on the above embodiment, in an optional embodiment, the urban feature set includes a spatial feature grid and a semantic feature vector group; feature extraction is performed on the multimodal remote sensing data of the target city, including: Using target remote sensing data as input, the process involves encoding multimodal remote sensing data into the same feature space to obtain a unified vectorized representation, thus obtaining a target vectorized representation. The target is vectorized and input into the shared encoder with frozen weights; and the labeled feature vectors of the last layer output of the shared encoder are extracted; the labeled feature vectors include the first labeled feature vectors corresponding to class A data and class B data, and the second labeled feature vectors corresponding to class C data; The first labeled feature vectors are rearranged into spatial feature maps according to their spatial positions, and the spatial feature maps are stacked into a spatial feature grid. Based on the second labeled feature vector, construct a semantic feature vector group.

[0110] After jointly training the multimodal pre-trained model based on the above embodiments, the weights of the shared Transformer encoder can be obtained. Based on this, downstream task predictions for the target city can be performed. In a specific prediction embodiment, the weights of the shared Transformer encoder are fixed, and forward inference is performed on the multimodal remote sensing data of the target city to extract and store features.

[0111] like Figure 4 As shown, Figure 3 The first stage, as shown, is divided into multiple layers. First, in the input layer, the target remote sensing data of the target city is divided into three types: A, B, and C. These types are then preprocessed in parallel to obtain a vectorized representation of the target. Further, in the patch embedding and location coding layer, the A and B type data are encoded using modality-specific linear projection and two-dimensional sine and cosine location coding to output spatial patch tokens. The C type data is encoded using MLP or linear projection to output a global semantic token sequence.

[0112] Then, a shared Transformer encoder is used to jointly encode the visible Patch Token sequences of all modalities and the global Token sequence of Class C, enabling cross-modal semantic interaction. The output sequences are then split according to data type and modality and fed into dedicated decoders. An auxiliary prediction head is configured for Class C data.

[0113] Specifically, for Class A data, a lightweight Transformer decoder is used to reconstruct the pixel values ​​or category labels of the masked patch; for Class B data, a shallow MLP decoder is used to reconstruct the aggregate semantic vector or temperature regression value of the masked grid; and for Class C data, a cross-modal auxiliary prediction head is configured to predict each scalar value using the global mean of the spatial token as input. During the joint pre-training process, iterative training is continuously performed with the above formula (20) as the overall optimization objective.

[0114] After pre-training, the weights of each modality-specific decoder are discarded, and only the weights of the shared Transformer encoder are retained and completely frozen, not participating in any subsequent parameter updates. The purpose of freezing the encoder is twofold: first, to fully transfer the urban multimodal general semantic representation learned in the pre-training stage to downstream tasks, avoiding overfitting due to limited downstream labeled data; second, to significantly reduce the computational cost of the downstream training stage, enabling efficient training on labeled data with a smaller parameter scale during downstream task prediction.

[0115] It should be noted that, Figure 3The inference input in step 3 shown needs to go through the entire preprocessing process in step 1, including spatial registration, downsampling, patch segmentation, and overlay of Linear Projector projection and position encoding for each modality, so as to finally obtain a multimodal token sequence that is completely consistent with the training input format in step 2.

[0116] Unlike the training phase in step 2, no masking is applied during the inference phase. All 256 spatial PatchTokens for each modality and all global tokens of class C are fed into the encoder in their entirety. The reason for not applying masks is that the goal of the inference phase is to extract the most complete semantic features possible, rather than simulating a reconstruction task with missing information. The full token input enables the encoder to output the most complete semantic representation for each position under the full multimodal context.

[0117] After the decoder and the auxiliary prediction head layer, the token feature vectors output by the last layer of the shared encoder are extracted, including the first token feature vectors corresponding to class A and class B data, and the second token feature vector corresponding to class C data.

[0118] It is worth noting that, Figure 3 The features extracted in step 3 shown are from the last layer of the shared Transformer encoder (the... The output token feature vector of the encoder layer is the encoded feature corresponding to the input of each modality-specific decoder in step 2, not the reconstruction output of the decoder. The output of the last layer of the encoder has undergone sufficient interaction across modalities through Self-Attention, containing the richest multimodal fusion semantic information and serving as the standard source for feature extraction in downstream tasks. The decoder only serves as reconstruction supervision in the pre-training stage, and its output is reconstruction results such as pixel values ​​and class probabilities. Its semantic discriminative ability is weaker than that of the encoder output and it is not used as a feature source.

[0119] Specifically, the encoder's final output sequence contains token features from all modalities. These features are split according to modal identifiers, and feature vectors corresponding to each modality are extracted. For spatial modal feature extraction of class A and class B data, 256 token feature vectors corresponding to each spatial modality are extracted from the encoder output sequence according to their modal identifiers and rearranged according to their spatial index. Spatial feature map.

[0120] For RGB imagery, building height maps, feature segmentation maps (Type A data), POI semantic maps, and land surface temperature grids (Type B data), one image is obtained for each. Spatial feature map, total The feature maps are spatial. Notably, the feature vectors output by the encoder have incorporated cross-modal attention information from other modal tokens, and the feature maps of each modality have semantically enhanced each other, no longer being isolated representations of a single modality.

[0121] For global modality feature extraction of class C, the global token feature vector corresponding to each class C modality is extracted from the beginning of the encoder output sequence according to the modality identifier. Each class C modality corresponds to one global token feature vector. A global feature vector. The global token interacts fully with all spatial tokens in the Self-Attention of each layer of the encoder. Its output feature vector has integrated the macro-semantic and spatial distribution information of the city and can be directly used as a city-level global semantic representation.

[0122] Furthermore, after extraction, the modal features are organized and stored to construct a city feature set. Specifically, the first labeled feature vectors are rearranged according to their spatial location to form spatial feature maps, and these spatial feature maps are then stacked to form a spatial feature grid. .

[0123] Specifically, the spatial modalities of class A and class B data are... Feature maps are stacked along the modal dimension to construct a multimodal spatial feature grid. ,in, This represents the total number of spatial modes. In one optional implementation, This corresponds to five spatial modes: RGB optical imagery, building height map, land feature segmentation map, POI semantic map, and land surface temperature grid. Each location within the spatial feature grid... One corresponding to the urban geographic space The ground unit at this location indivual The 3D feature vectors jointly characterize the semantic information of the ground unit in all spatial modalities.

[0124] Furthermore, based on the second labeled feature vector, a semantic feature vector set is constructed. Specifically, the data of each C category... The global feature vectors are arranged in modal order to construct a city-level global semantic feature vector set. ,in, This represents the total number of modalities in class C data. Semantic feature vector group. The macroscopic attribute semantics describing the city will serve as auxiliary context during the feature fusion stage, participating in the dynamic calculation of modality weights.

[0125] In an alternative embodiment, a spatial feature grid and semantic feature vector group Each city is stored separately in tensor file format (such as ".npy" or ".pt"), with the city identifier and data phase as the index key.

[0126] It should be noted that, in specific embodiments, Figure 3 Step 3 shown, namely Figure 4 The steps shown can be performed independently for multiple cities, each with its own multimodal data. Figure 3 The first step of preprocessing, as shown, is fed into the same frozen encoder for forward inference, where the spatial feature meshes are extracted and stored. and semantic feature vector group The encoder weights remain completely consistent across all cities during the inference process, reflecting the cross-city generality of the feature representations learned by the pre-trained encoder. The inference processes in each city are independent and can be executed in parallel to improve processing efficiency.

[0127] Figure 5 A schematic diagram illustrating the principle of a method for predicting downstream urban tasks, as provided in another embodiment of this application, is shown below. Figure 5 As shown, based on the above embodiments, the second stage of the prediction method for downstream tasks in the city is entered, namely the downstream task adaptive fusion and prediction stage. That is, step 4, task adaptive modal fusion and downstream prediction based on a hybrid expert network, is performed.

[0128] In a specific embodiment, the spatial feature mesh obtained in the above embodiment is used. and semantic feature vector group As input to step 4, the multimodal urban features are dynamically fused into a fused feature representation for a specified downstream task through a task-adaptive fusion module (i.e., feature fusion network) based on a Mixture of Experts (MoE) network, and the prediction result is output by connecting to the corresponding task prediction head.

[0129] Figure 6 The flowchart illustrates a method for predicting downstream urban tasks, as provided in another embodiment of this application. As an optional embodiment, such as... Figure 6 As shown, the feature fusion network includes a gating network and an expert network group; through the feature fusion network, the city feature set is fused into a fused feature representation oriented towards a specified downstream task, including: S60: Obtain the task type code of the specified downstream task and extract the target features from the city feature set; the target features include regional spatial query features and city global semantic features; the expert network group includes multiple expert networks that correspond one-to-one with the target features; S61: Input the task type code, regional spatial query features and city global semantic features into the gating network to determine the fusion weights corresponding to each data type; Figure 7 This application also provides a schematic diagram of the principle of a method for predicting downstream urban tasks according to an embodiment. The overall structure of the feature fusion network is as follows: Figure 7 As shown, in step 4, the weights of the shared Transformer encoder obtained in step 2 are kept frozen, and the feature fusion network and the task prediction head are jointly trained end-to-end with the downstream labeled data as the supervision signal.

[0130] like Figure 7 As shown, the input for step 4 includes a spatial feature grid. Semantic feature vector group And the task type encoding of downstream tasks. In an optional embodiment, spatial feature grid. Each modality is perfectly aligned in spatial position, at each position One corresponding geographic space The ground unit, denoted as the first The feature map of each spatial mode is , .

[0131] semantic feature vector group This contains city-level global semantic vectors corresponding to each C-type data modality, denoted as the... The global feature vectors of the C-type modes are Its global mean is .

[0132] In one alternative embodiment, the task type encoding of the current downstream task is input into the gating network in the form of a learnable task embedding vector. Different tasks correspond to independent embedding vectors, which are jointly trained with the MoE fusion module.

[0133] For the gating network in the feature fusion network, used for three-dimensional adaptive weight calculation, in a specific embodiment, the task type encoding, regional spatial query features and city global semantic features are input into the gating network, and the fusion weights of each modality feature are dynamically calculated to achieve three-dimensional adaptation of task perception, regional perception and global semantic perception.

[0134] First, the regional spatial query features are constructed. In one optional embodiment, the feature map of the reference spatial modality (RGB optical image) is used. Perform global average pooling to obtain Dimensional region query feature vector See formula (21): (twenty one) Among them, regional query features The spatial mean of RGB features is used to characterize the overall spatial semantics of the current urban area, reflecting the differences in land cover composition, building density and urban form among different areas.

[0135] Furthermore, regional query features Task embedding vector and global semantic mean vector The vectors are concatenated to form a 3D gated input vector, as shown in formula (22): (twenty two) in, The mean of each C-class global feature vector output in step 3 carries the semantics of macro-level city attributes (such as population size, climate type, economic level, etc.), enabling the gating network to perceive the city-level context when calculating modal weights, and further improving the adaptability of weight allocation.

[0136] Then, gating input After passing through a two-layer fully connected network (FC-ReLU-FC) and Softmax normalization, the fusion weight vector of each modality is output, as shown in formula (23): (twenty three) in, , For learnable parameters, , This corresponds to the bias term. Softmax normalization ensures that the modal weights are non-negative and sum to 1, i.e. , Weight vector middle, front The first component corresponds to the fusion weight of the spatial modality, and the second component corresponds to the fusion weight of the global modality of class C.

[0137] In a specific embodiment, the three-dimensional adaptive characteristic refers to the gating network simultaneously receiving task type encoding, regional spatial query features, and city-wide semantic features as input to achieve adaptive weight allocation across the following three dimensions. Firstly, task awareness: specifically, the same urban area outputs different modal weights under different downstream tasks. For example, the carbon emission estimation task assigns higher weights to building height maps and feature segmentation maps, while the population density prediction task assigns higher weights to POI semantic maps.

[0138] Secondly, there is regional awareness. Specifically, the same downstream task outputs different modal weights in different urban areas. For example, the dependence of high-density commercial areas on each modality differs significantly from that of low-density suburbs under the same task.

[0139] Third, global semantic awareness. Specifically, the macro-attributes of a city (such as climate zone type and economic development level) have a moderating effect on the relative importance of each modality. The introduction of global semantic features enables the gating network to automatically adapt to the modality weight distribution of different cities when generalizing between cities.

[0140] Therefore, this application employs a MoE gating mechanism jointly driven by task awareness, region awareness, and global semantic awareness. The gating network simultaneously receives task type encoding, regional spatial query features, and the average macro-semantic value of the city as input, dynamically calculating the fusion weights of each modality. Compared to fusion strategies with fixed weights or only task awareness, this application can adaptively adjust modal contributions based on the combined effects of task type, urban regional differences, and urban macro-attributes, exhibiting stronger generalization capabilities in urban analysis scenarios with diverse tasks and strong regional heterogeneity.

[0141] S62: The corresponding target features are independently transformed by each expert network to obtain the output features; Based on the above embodiments, as an optional embodiment, expert networks are formed by... It consists of several independent expert networks, each of which independently transforms the features of the corresponding modality, and the parameters are not shared between them.

[0142] For spatial modality expert networks used for both Class A and Class B data, feature transformation is performed on the feature maps of each spatial modality using an independent two-layer MLP expert network with identical structure. This is because the features of Class A and Class B data differ in... Figure 3 The spatial feature maps shown in Step 1 are all unified and perfectly aligned in spatial scale, requiring no upsampling or spatial alignment operations. The expert network acts independently on each feature vector of each dimension, as shown in Equation (24). (twenty four) in, For the first Each modality-specific two-layer perceptron (linear layer-ReLU-linear layer, ... ), For the hidden layer dimension, output Each expert network has independent parameters, ensuring that the feature transformation of each mode fully adapts to the feature distribution of that mode and is not affected by interference from other modes.

[0143] For a global modal expert network of class C data, the modalities of each class C are... The global feature vector is transformed using an independent MLP, and the output vector is then broadcast and expanded to [a specific region / location]. The spatial dimension enables global semantic information to participate in the weighted fusion of spatial dimensions, see formula (25): (25) in, For the first A two-layer MLP specific to each Class C modality ( ), Indicates will dimensional vector in Broadcasting and copying in spatial dimensions yields Broadcasting operations uniformly inject macroscopic urban attribute information into the fused features of each spatial location, providing global semantic constraints for local spatial prediction.

[0144] S63: Based on the fusion weights, the output features are weighted and summed to obtain the fusion feature representation.

[0145] Furthermore, the weight vector output by the gating network... The output feature maps of each expert network are weighted and summed to obtain the unified multimodal fusion feature tensor for task awareness, as shown in formula (26): (26) Fusion feature tensor Preserve integrity While achieving spatial resolution, it integrates semantic information from all modalities, and its modal weight allocation is dynamically adjusted according to task type, regional features, and global city semantics, possessing three-dimensional adaptive characteristics.

[0146] In a specific embodiment of feature fusion, the fused feature tensor is... After being compressed into region-level feature vectors by global average pooling, and then subjected to linear projection and nonlinear activation, the following is obtained: 3D task feature vector See formula (27): (27) in, The projected feature dimensions, , For the learnable parameters of the linear projection layer, Indicates to Global average pooling operation in spatial dimensions.

[0147] For the task prediction header, for each downstream task Configure an independent task prediction head with region-level feature vectors. The input is the prediction result for the corresponding task. Each task's prediction head structure is a two-layer fully connected network with independent parameters, as shown in formula (28): (28) in, , For the first Learnable parameters of each task prediction head For hidden layer dimensions, For output dimensions (regression task) Classification tasks (Equal to the number of categories). The downstream urban analysis tasks supported by the embodiments of this application include, but are not limited to, urban carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. Each task outputs regression values ​​or classification results using a spatial grid as the prediction unit.

[0148] It should be noted that, in the specific embodiment, the training objects in step 4 include the MoE fusion module (gated network and expert networks for each modality), the feature aggregation layer, the task type embedding vector, and the prediction heads for each task. The above modules are jointly trained end-to-end using labeled downstream data as supervision signals. The shared Transformer encoder weights pre-trained in step 2 remain completely frozen throughout the entire training process in step 4 and do not participate in gradient calculation and parameter updates.

[0149] In an alternative embodiment, for regression-type downstream tasks, the prediction head outputs continuous regression values, using mean squared error loss, see formula (29): (29) For the downstream task of classification, the prediction head outputs the category probability distribution, and the cross-entropy loss is used, see formula (30): (30) in, For the first Number of labeled samples for each task For the first The true labeled values ​​of each sample. During multi-task joint training, the total loss is the weighted sum of the losses from each task, as shown in formula (31): (31) in, For the first The loss weight coefficients for each task can be set as fixed hyperparameters or learned automatically using an uncertainty-weighted method. Training can employ the AdamW optimizer, and the learning rate can use a preheating plus cosine annealing scheduling strategy.

[0150] In one alternative embodiment, when training for a single downstream task, the gating network receives a fixed task type embedding vector and activates only the corresponding task prediction head. When performing multi-task joint training, task types are randomly sampled in each training batch, and the gating network learns to dynamically adjust modality weights for different tasks. Each task prediction head independently receives labeled supervision for the corresponding task. Both training methods are within the scope of protection of this application.

[0151] Therefore, the prediction method for downstream urban tasks provided in this application can improve training efficiency. Specifically, after the pre-trained encoder weights are frozen, the number of trainable parameters in step 4 is only a fraction to a tenth of the encoder size, and the required number of labeled samples is reduced. This has important practical value in the actual scenario where the cost of labeling urban data is high and labeled samples are scarce.

[0152] On the other hand, different downstream tasks share the same set of pre-trained features and expert network parameters. The gating network adapts to different tasks by outputting different weight allocations. There is no need to train a separate feature transformation network for each task. The parameter efficiency in multi-task scenarios is significantly better than independent training schemes. Moreover, the shared features between different tasks help alleviate the problem of insufficient labeled data for single tasks.

[0153] Furthermore, the gated network consists of two fully connected layers, with a parameter scale much smaller than that of the pre-trained encoder. Even with only a small number of labeled samples for each task, it can quickly learn the dependencies of different tasks and regions on each modality, output reasonable modality weight allocation, and achieve effective downstream task adaptation.

[0154] In the above embodiments, the prediction method for urban downstream tasks has been described in detail. This application also provides an embodiment of a device for predicting urban downstream tasks.

[0155] Figure 8 This is a schematic diagram of the structure of a predictive device for downstream urban tasks provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes: Remote sensing data acquisition module 80 is used to acquire multimodal remote sensing data; The data unification processing module 81 is used to encode multimodal remote sensing data into the same feature space to obtain a unified vectorized representation; The joint training module 82 is used to jointly pre-train a pre-built multimodal pre-trained model using a unified vectorized representation to determine the weights of the shared encoder in the multimodal pre-trained model. The feature extraction module 83 is used to extract features from the multimodal remote sensing data of the target city using a shared encoder to obtain a city feature set. The feature fusion module 84 is used to fuse the urban feature set into a fused feature representation for a specified downstream task through a feature fusion network; the specified downstream task includes at least carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. The task prediction module 85 is used to input the fused feature representation into the independent prediction head of the specified downstream task to obtain the prediction result.

[0156] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 9 As shown, the electronic device includes: a memory 90 for storing computer programs; The processor 91 is configured to execute a computer program to implement the steps of the urban downstream task prediction method as described in the above embodiments.

[0157] The electronic devices provided in this embodiment may include, but are not limited to, laptops or desktop computers.

[0158] The processor 91 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 91 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 91 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 91 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 91 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0159] The memory 90 may include one or more computer-readable storage media, which may be non-transitory. The memory 90 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 90 is used to store at least the following computer program 901, which, after being loaded and executed by the processor 91, is capable of implementing the relevant steps of the urban downstream task prediction method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 90 may also include an operating system 902 and data 903, etc., and the storage method may be temporary storage or permanent storage. The operating system 902 may include Windows, Unix, Linux, etc. The data 903 may include, but is not limited to, relevant data involved in the urban downstream task prediction method.

[0160] In some embodiments, the electronic device may further include a display screen 92, an input / output interface 93, a communication interface 94, a power supply 95, and a communication bus 96.

[0161] Those skilled in the art will understand that Figure 9 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0162] The electronic device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the urban downstream task prediction method in the above embodiments.

[0163] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A method for predicting downstream tasks in cities, characterized in that, The method includes: Acquire multimodal remote sensing data; The multimodal remote sensing data is encoded into the same feature space to obtain a unified vectorized representation; the multimodal remote sensing data includes multiple classes of data, including class A, class B, and class C data; the class A data is subjected to a block embedding operation to obtain a first block label sequence; the class B data is subjected to a cross-modal aligned embedding operation to obtain a second block label sequence; the class C data is subjected to a global projection operation to obtain a global label sequence. The pre-built multimodal pre-trained model is jointly pre-trained using the unified vectorized representation to determine the weights of the shared encoder in the multimodal pre-trained model. Using the shared encoder, feature extraction is performed on multimodal remote sensing data of the target city to obtain a city feature set; The urban feature set is fused into a fused feature representation for a specified downstream task through a feature fusion network; the specified downstream task includes at least carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. The fused feature representation is input into the independent prediction head of the specified downstream task to obtain the prediction result; Joint pre-training of pre-built multimodal pre-trained models includes: A first mask rate is applied to the first block marker sequence, and a second mask rate is applied to the second block marker sequence to obtain a masked block marker sequence; the first mask rate is greater than the second mask rate. The visible markers in the mask block marker sequence are concatenated with the global marker sequence, and then input into the shared encoder for joint encoding to obtain the output sequence. The output sequence is split according to the data types of the multiple data types, and then input into the decoder corresponding to each data type to perform loss reconstruction to obtain the reconstruction loss; Based on the reconstruction loss, a total optimization objective is determined; and the multimodal pre-trained model is jointly trained with the goal of minimizing the total optimization objective. The feature fusion network includes a gating network and an expert network group; through the feature fusion network, the city feature set is fused into a fused feature representation oriented towards a specified downstream task, including: The task type code of the specified downstream task is obtained, and target features are extracted from the city feature set. The target features include regional spatial query features and city global semantic features. The expert network group includes multiple expert networks that correspond one-to-one with the target features. The regional spatial query features are used to characterize the differences in land cover composition, building density, and urban morphology between different regions. The city global semantic features are used to characterize the city's macro-attribute semantics. The city's macro-attribute semantics include population size and climate type. The task type encoding, the regional spatial query features, and the city global semantic features are input into the gating network to determine the fusion weights corresponding to each data type. The corresponding target features are independently transformed by each of the expert networks to obtain the output features; Based on the fusion weights, the output features are weighted and summed to obtain the fusion feature representation.

2. The method for predicting downstream urban tasks as described in claim 1, characterized in that, Encoding the multimodal remote sensing data into the same feature space to obtain a unified vectorized representation includes: Based on preset classification rules, the multimodal remote sensing data is divided into multiple data categories; the spatial resolution of category A data is higher than that of category B data, and category C data is global scalar data without spatial distribution information; The data of type A is segmented and embedded to obtain the first segmented label sequence; Based on the alignment to the first block label sequence, a cross-modal alignment embedding operation is performed on the B-type data to obtain the second block label sequence; A global projection operation is performed on the C-type data to obtain a global label sequence with the same feature dimensions as the first block label sequence; The global tag sequence is used as the head, and it is concatenated with the first block tag sequence and the second block tag sequence to obtain the unified vectorized representation.

3. The method for predicting downstream urban tasks as described in claim 2, characterized in that, The B-type data includes semantic feature maps of interest points; cross-modal alignment embedding is performed on the B-type data to obtain a second block label sequence, including: Map the point location data of the point of interest in the semantic feature map to a spatial grid of the same size as the downsampled data of class A. Extract the semantic embedding vector of each interest point in the semantic feature map of interest points, and perform mean pooling on the semantic embedding vectors within the same spatial grid to obtain the aggregated semantic vector; For target spatial grids in the spatial grid where the amount of interest point data is less than the threshold, a filling vector is generated and filled using the land cover category label at the corresponding location in the land cover segmentation map of the Class A data; The aggregated semantic vector and the padding vector are projected onto the same feature dimension and superimposed with the same positional encoding as the first block label sequence to obtain the second block label sequence corresponding to the interest point semantic feature map.

4. The method for predicting downstream urban tasks as described in claim 3, characterized in that, The Class B data includes surface temperature grids; Perform a cross-modal aligned embedding operation on the B-type data to obtain a second block label sequence, including: The surface temperature grid is resampled to a spatial grid of the same size as the downsampled Class A data, and the average temperature within the corresponding ground area is recorded for each spatial grid. The average temperature is normalized to obtain a normalized temperature value. The normalized temperature value is projected onto the same feature dimension and superimposed with the same position code as the first block label sequence to obtain the second block label sequence corresponding to the surface temperature grid.

5. The method for predicting downstream urban tasks as described in claim 1, characterized in that, The output sequence includes spatial label features corresponding to the A-class data and the B-class data; Based on the reconstruction loss, the overall optimization objective is determined, including: Global average pooling is performed on the spatial label features to obtain the global mean of the spatial features; The global mean of the spatial features is input into the auxiliary prediction head of the C-class data to predict the corresponding scalar value, thereby obtaining the prediction loss. Based on the reconstruction loss and the prediction loss, the overall optimization objective is constructed.

6. The method for predicting downstream urban tasks as described in claim 1, characterized in that, The B-type data includes an interest point semantic feature map, and the corresponding decoder is an interest point semantic map decoder; the reconstruction target of the interest point semantic map decoder is an aggregated semantic vector, and the corresponding reconstruction loss is a cosine similarity loss.

7. The method for predicting downstream urban tasks as described in claim 6, characterized in that, The C-type data includes numerical scalars, categorical scalars, and text scalars; The prediction target of the auxiliary prediction head corresponding to the numerical scalar is the data normalized scalar value, and the corresponding prediction loss is the mean square error loss. The categorical scalar corresponds to the auxiliary prediction head whose prediction target is the category label, and the corresponding prediction loss is the cross-entropy loss. The prediction target of the text-type scalar corresponding auxiliary prediction head is the text semantic embedding vector, and the corresponding prediction loss is the cosine similarity loss.

8. The method for predicting downstream urban tasks as described in claim 2, characterized in that, The city feature set includes a spatial feature grid and a semantic feature vector set; Feature extraction is performed on multimodal remote sensing data of the target city, including: Using the target remote sensing data as input, the step of encoding the multimodal remote sensing data into the same feature space to obtain a unified vectorized representation is performed to obtain a target vectorized representation; The target is vectorized and input into the shared encoder with the weights frozen; and the labeled feature vector output from the last layer of the shared encoder is extracted; the labeled feature vector includes the first labeled feature vector corresponding to the A-class data and the B-class data, and the second labeled feature vector corresponding to the C-class data; The first labeled feature vector is rearranged into a spatial feature map according to its spatial position, and the spatial feature maps are stacked into the spatial feature grid. Based on the second marker feature vector, the semantic feature vector group is constructed.

9. A device for predicting downstream tasks in cities, characterized in that, The device includes: The remote sensing data acquisition module is used to acquire multimodal remote sensing data; A unified data processing module is used to encode the multimodal remote sensing data into the same feature space to obtain a unified vectorized representation; the multimodal remote sensing data includes multiple types of data, including type A data, type B data, and type C data; a block embedding operation is performed on the type A data to obtain a first block label sequence; a cross-modal aligned embedding operation is performed on the type B data to obtain a second block label sequence; and a global projection operation is performed on the type C data to obtain a global label sequence. The joint training module is used to jointly pre-train a pre-built multimodal pre-trained model using the unified vectorized representation to determine the weights of the shared encoder in the multimodal pre-trained model. The feature extraction module is used to extract features from the multimodal remote sensing data of the target city using the shared encoder to obtain a city feature set. The feature fusion module is used to fuse the urban feature set into a fused feature representation for a specified downstream task through a feature fusion network; the specified downstream task includes at least carbon emission concentration estimation, population density prediction, surface temperature prediction, nighttime light intensity prediction, and building density prediction. The task prediction module is used to input the fused feature representation into the independent prediction head of the specified downstream task to obtain the prediction result; This includes joint pre-training of pre-built multimodal pre-trained models, including: A first mask rate is applied to the first block marker sequence, and a second mask rate is applied to the second block marker sequence to obtain a masked block marker sequence; the first mask rate is greater than the second mask rate. The visible markers in the mask block marker sequence are concatenated with the global marker sequence, and then input into the shared encoder for joint encoding to obtain the output sequence. The output sequence is split according to the data types of the multiple data types, and then input into the decoder corresponding to each data type to perform loss reconstruction to obtain the reconstruction loss; Based on the reconstruction loss, a total optimization objective is determined; and the multimodal pre-trained model is jointly trained with the goal of minimizing the total optimization objective. The feature fusion network includes a gating network and an expert network group; through the feature fusion network, the city feature set is fused into a fused feature representation oriented towards a specified downstream task, including: The task type code of the specified downstream task is obtained, and target features are extracted from the city feature set. The target features include regional spatial query features and city global semantic features. The expert network group includes multiple expert networks that correspond one-to-one with the target features. The regional spatial query features are used to characterize the differences in land cover composition, building density, and urban morphology between different regions. The city global semantic features are used to characterize the city's macro-attribute semantics. The city's macro-attribute semantics include population size and climate type. The task type encoding, the regional spatial query features, and the city global semantic features are input into the gating network to determine the fusion weights corresponding to each data type. The corresponding target features are independently transformed by each of the expert networks to obtain the output features; Based on the fusion weights, the output features are weighted and summed to obtain the fusion feature representation.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the prediction method for downstream urban tasks according to any one of claims 1 to 8.

Images