A spatial neighborhood reconstruction auto-encoding multi-modal fusion vector generation method

Abstract

The application discloses a kind of space neighborhood reconstruction self-encoding multimodal fusion vector generation method, belong to geographic information science, computer vision and artificial intelligence cross technical field.The present mask self-encoding technique is only stopped at the technical limitation of grid level spatial unit, cannot process the heterogeneous modal vector of geographical entity with independent identity, the application proposes a kind of geographical entity-oriented multimodal mask self-encoding training framework: the multiple heterogeneous modal vectors of geographical entity with independent entity identification are randomly masked, the heterogeneous modal vector information of adjacent entity in the space neighborhood determined by k-ring algorithm is used, the masked modal is reconstructed by multimodal fusion network using the remaining modal vector of the entity not masked;With the weighted combination of reconstruction loss and contrast learning loss, the fusion network parameters are jointly optimized.The application also provides a joint optimization method and a cold start and gradual switching method.

Description

Technical Field

[0001] This invention relates to the fields of geographic information science, computer vision and artificial intelligence, and in particular to a multimodal fusion vector generation method for reconstructing the spatial neighborhood of geographic entities with independent identities using autoencoders. Background Technology

[0002] With the deep integration of Geographic Information Systems (GIS) and Artificial Intelligence (AI) technologies, vector representation methods for geographic entities have become a core foundation supporting applications such as spatial semantic retrieval, intelligent situational analysis, and automated mapping. Geographic entities possess rich heterogeneous information dimensions: visual features, symbolic representation, geometric location, geographic context, structured attributes, and business knowledge. How to effectively integrate these six heterogeneous modalities into a unified dense vector representation, while simultaneously enabling this vector representation to naturally encode spatial contextual semantic relationships, is a key technical challenge in the current field of geographic AI.

[0003] Among existing technologies, the closest approach is a self-supervised geospatial embedding method that utilizes spatial adjacency relationships as a natural supervisory signal and learns the vector representation of geographic grids through a mask autoencoder. This method discretizes large-scale geographic regions into grid cells, statistically analyzes the built environment features within each cell, rasterizes the feature vectors into image patches, and then performs a mask autoencoder to learn the embedding representation of the grid. However, existing grid-based mask encoding techniques are essentially homogenized representations of regional statistical features. The semantic features of spatially fragmented entities are smoothed out or lost during rasterization, failing to achieve accurate cross-modal alignment at the entity level.

[0004] However, this existing technology has the following limitations: First, it processes grid cells, which are uniformly discrete spatial units without independent identity or multimodal attributes, and cannot handle heterogeneous information such as visual features, symbolic semantics, and structural attributes of geographic entities. Second, the masking object is a rasterized image of the feature vectors of the built environment, belonging to a homogeneous embedding space, and cannot handle heterogeneous modal vectors with different dimensions and semantic spaces. Third, neighborhood relationships are based on regular adjacency between cells, while the spatial distribution between geographic entities is often irregular and sparse, requiring a more flexible neighbor selection mechanism based on discrete grid indexing. Fourth, the loss function only uses pure mean squared error loss and does not introduce contrastive learning loss to enhance the discriminativeness of vector representation. The conclusion section of the published literature of this existing technology explicitly lists "extending beyond the grid" as a research problem to be solved.

[0005] In the field of geospatial embedding learning based on H3 grids, Hex2Vec and GeoVex provide concrete practices for self-supervised learning using hexagonal hierarchical grid systems. Hex2Vec uses the H3 spatial index to statistically analyze OpenStreetMap labels within a region and trains it using a skip-gram model. Its context region selection is based on the neighbors directly adjacent to the target region in the H3 grid. GeoVex is also based on H3 indexes and OSM labels, but uses a hexagonal convolutional autoencoder to generate embeddings, which better considers contextual information from neighboring H3 regions. However, the processing objects of Hex2Vec and GeoVex are still H3 grid cells (grid level), rather than geographic entities with independent identities; the representations they learn are region-level statistical feature embeddings, which cannot handle heterogeneous modal information such as visual features, symbolic semantics, structured attributes, and business knowledge of individual geographic entities; although their spatial neighborhood selection mechanism utilizes the hexagonal structure of H3, it does not use the heterogeneous modal vectors of neighboring entities as reconstruction targets, nor does it employ a modality combination masking strategy to perform reconstruction at the entity level across heterogeneous semantic spaces. Therefore, while Hex2Vec and GeoVex are valuable references in utilizing H3 spatial neighborhoods, they fail to solve the core problem of extending mask autoencoding from the lattice level to the entity level.

[0006] In the field of geospatial location encoding and contrastive learning, Space2Vec, by simulating the multi-scale periodic representation mechanism of grid cells in the mammalian brain, uses sine functions of different frequencies to encode raw geographic coordinates, generating representations that capture absolute location and spatial relationships. This is a landmark foundational work in the field. Loc2Vec uses rasterized OpenStreetMap map data to create multi-channel tensors and learns location embeddings through a convolutional encoder network. Its training employs Triplet Loss to make the distance between geographically close location vectors closer. SatCLIP and GeoCLIP apply the successful experience of vision-language pre-training to geospatial data, aligning satellite images with their corresponding geographic locations into a common embedding space through contrastive learning. However, the above technical solutions all process spatial locations or grid regions, rather than geographic entities with independent identities and multimodal attributes; the representations they learn encode location features or regional features, without involving entity-level multimodal fusion; and their training paradigms—whether skip-gram, Triplet Loss, or CLIP-style contrastive learning—do not employ masked autoencoders to reconstruct missing entity modalities using spatial neighborhood information. Therefore, the aforementioned existing technologies fail to solve the problem of reconstructing the spatial neighborhood of geographical entities with independent identities.

[0007] In the field of multimodal masking autoencoders, existing technologies primarily handle multi-sensor data from the same scene. Each modality physically represents parallel observations of the same object, exhibiting a natural spatiotemporal alignment. Masking strategies are executed across modalities at the same spatial location, rather than across heterogeneous semantic spaces at the entity level. In the audio-visual domain, frameworks exist that jointly optimize reconstruction and contrastive losses, but these only process audio and video modalities, neglecting the geospatial dimension.

[0008] In the field of multimodal feature fusion and metric learning, CN202310727786 discloses a logo recognition method based on multimodal fusion and metric learning. It employs a feature concatenation early fusion strategy to combine visual and textual feature vectors in series, and uses surrogate anchor loss as the metric loss function for training. This technique addresses the visual and textual dual-modal fusion in logo recognition tasks. Its visual features originate from general image perception and do not involve modalities specific to geographic entities such as symbol style, geometric location, and geographic context. Its metric learning serves logo category discrimination and does not employ a functional semantic-driven symbol vector space organization strategy. Its application scenario is commercial logo recognition, which is fundamentally different from the autoencoder problem of spatial neighborhood reconstruction of geographic entities.

[0009] In summary, existing technologies in the field of geospatial embedding learning—including grid-level S2Vec, H3-based Hex2Vec and GeoVex, and location-level Space2Vec, Loc2Vec, and SatCLIP—all deal with spatial units, grid regions, or geographic locations, rather than geographic entities with independent identities and multimodal attributes. Therefore, they cannot solve the problem of heterogeneous modality alignment and fusion at the entity level. In the field of multimodal mask autoencoders, existing technologies mainly process multi-sensor data from the same scene, without performing mask reconstruction across heterogeneous semantic spaces at the entity level. In the field of multimodal feature fusion, existing technologies, such as multimodal fusion and metric learning methods in logo recognition, do not address the six unique heterogeneous modalities of geographic entities and their spatial contextual relationships. Therefore, existing technologies have not yet solved the technical problems of how to extend the idea of ​​spatial neighborhood reconstruction autoencoder from the lattice level to the entity level, how to use the k-ring algorithm to flexibly determine the spatial neighborhood and aggregate the heterogeneous modal vector information of neighboring entities in a hexagonal hierarchical grid system, and how to use a dynamic weighted joint optimization strategy of reconstruction loss and contrastive learning loss to train multimodal fusion networks. Summary of the Invention

[0010] The technical problem to be solved by this invention is to provide a multimodal fusion vector generation method based on spatial neighborhood reconstruction autoencoder. This method can generate a unified fusion vector by using a spatial neighborhood reconstruction autoencoder mechanism to generate multiple heterogeneous modal vector information of geographic entities with independent identities. The generated unified vector not only encodes the multimodal semantics of the entity itself, but also naturally captures spatial contextual relationships. It also provides a joint optimization method and a cold start and gradual switching mechanism, and does not rely on manually labeled data.

[0011] To solve the above-mentioned technical problems, the present invention provides the following technical solution: I. A Multimodal Fusion Vector Generation Method Based on Spatial Neighborhood Reconstruction Autoencoder A multimodal fusion vector generation method for spatial neighborhood reconstruction autoencoder includes the following steps: Step S1: Obtain multiple heterogeneous modal vectors of the target geographic entity. The heterogeneous modal vectors include at least two of the following: visual feature vectors, symbolic style vectors, geometric location vectors, geographic context vectors, structured attribute vectors, and knowledge semantic vectors. The target geographic entity has a unique entity identifier, a timestamp, and a set of heterogeneous modal vectors. Each modal vector is obtained in the following ways: visual feature vectors are extracted from images using a lightweight visual feature extraction network; symbolic style vectors are generated from map symbol parameters using a symbol encoder; geometric location vectors are generated using multi-resolution hierarchical encoding; geographic context vectors are generated using neighborhood statistical compression; structured attribute vectors are generated using category mapping and bucketing encoding; and knowledge semantic vectors are generated using a pre-trained text embedding model.

[0012] Step S2: Perform a random masking operation. Randomly select one or two modal vectors from multiple heterogeneous modal vectors of the target geographic entity for masking, resulting in a masked modal set and a retained modal set. The masking operation employs a modal combination masking strategy: Let the total number of modalities be N, where N is a positive integer not less than 2. In each iteration, randomly select one or two modalities from the N modalities, set the vector value of the selected modalities to zero, and retain the original vector values ​​of the unmasked modalities.

[0013] Step S3: Determine the spatial neighborhood. Based on the spatial discrete grid index of the target geographic entity, the k-ring algorithm is used to determine one or more neighboring geographic entities within its spatial neighborhood. The spatial discrete grid index adopts a hexagonal hierarchical grid system. By mapping the geometric coordinates of the geographic entity to grid cell identifiers, the k-ring algorithm is used to determine neighboring geographic entities within the spatial neighborhood, where k is a positive integer. Within the determined spatial neighborhood, other existing geographic entities are selected as the set of neighboring entities. The neighboring geographic entities are geographic objects with independent identities and heterogeneous modal vector sets. Unlike existing technologies based on regular adjacency, this invention uses the k-ring algorithm to determine the spatial neighborhood, which can flexibly adapt to the irregular spatial distribution of geographic entities.

[0014] Step S4: Modal Projection and Neighborhood Aggregation. Each modal vector in the retained modality set is projected to a unified intermediate dimension through layer normalization and linear transformation, and then concatenated to form a retained modality representation. The projection operation sequentially performs layer normalization and linear transformation on each modal vector, projecting each modal vector to the same preset intermediate dimension. The preset intermediate dimension is determined by the number of heterogeneous modal vectors and the statistical characteristics of the original dimensions of each modality.

[0015] Simultaneously, the unified vectors of the neighboring geographic entities are spatially aggregated using a weighted averaging mechanism to obtain aggregated neighborhood information. Spatial aggregation employs a distance-attenuation-based weighted averaging mechanism: Aggregated Neighborhood Information = Σ_j (w_j × u_j) / Σ_j w_j, where u_j is the unified vector of the j-th neighboring geographic entity, w_j is the attenuation weight based on spatial distance, w_j = exp(-dist_j / σ), dist_j is the spatial distance between the j-th neighboring entity and the target entity, and σ is a preset distance attenuation factor. The value of σ is related to the hierarchical resolution of the spatial discrete grid index.

[0016] Step S5: Multimodal Fusion Network Reconstruction. The concatenated preserved modality representation and aggregated neighborhood information are input into the multimodal fusion network. The fusion network contains at least two hidden layers, each with a predetermined number of neurons. A non-linear activation function is used, and features interact through fully connected layers to generate the reconstructed vectors of the masked modalities. The output layer dimension of the fusion network matches the original dimension of the masked modalities. When masking multiple modalities, the output layer outputs the reconstructed vectors of each masked modality.

[0017] Step S6: Joint Loss Optimization. Calculate the reconstruction loss between the reconstructed vector and the original masked modality vector—the reconstruction loss is the mean squared error loss. Simultaneously, calculate the contrastive learning loss between the unified vector of the target geographic entity and the unified vectors of other entities within the batch—the contrastive learning loss is the InfoNCE loss. Optimize the multimodal fusion network using the joint loss function, which is a weighted combination of the reconstruction loss and the contrastive learning loss. In the joint loss function, the value of the first weight coefficient is greater than the second weight coefficient, and the ratio of the first weight coefficient to the second weight coefficient is dynamically adjusted based on the training phase, modality missing rate, and spatial neighborhood density.

[0018] Step S7: Unified Vector Generation. The optimized modal vectors are input into the fusion network to generate a unified vector representation of the target geographic entity. L2 normalization is used in the unified vector generation process to ensure that the unified vector lies on a unit hypersphere, facilitating cosine similarity calculation.

[0019] II. Joint Optimization Method This invention also provides a multimodal fusion vector joint optimization method for geographic entities. The method includes: masking a portion of multiple heterogeneous modal vectors, retaining the remaining modal vectors; using the retained modal vectors and heterogeneous modal vector information of neighboring geographic entities within the spatial neighborhood of the target geographic entity, reconstructing the masked modality through a multimodal fusion network to obtain the reconstructed vector; calculating the reconstruction loss between the reconstructed vector and the original masked modal vector, and simultaneously calculating the contrastive learning loss between the unified vector of the target geographic entity and the unified vectors of other geographic entities in the batch. The contrastive learning loss uses unified vectors generated by the same geographic entity in different masking rounds as positive samples, unified vectors of geographic entities with the same semantic type within the same spatial neighborhood as positive samples, and unified vectors of other entities within the batch that are not positive samples as negative samples. The multimodal fusion network is optimized using a weighted combination of the reconstruction loss and the contrastive learning loss as the joint loss function. The weighting coefficients of the joint loss function are related to the spatial neighborhood level of the target geographic entity and the modality missing rate of the heterogeneous modal vectors.

[0020] III. Cold Start and Gradual Switching Methods This invention also provides a method for cold start and gradual switching of multimodal fusion vectors for geographic entities. The method includes: in the cold start phase, when the amount of training data has not reached a preset threshold, a rule-based multimodal vector fusion strategy is adopted—normalizing each modality vector and then concatenating them to generate a unified vector through dimensionality reduction; monitoring the accumulation of training data, and when the amount of training data reaches the preset threshold, starting the training of the multimodal fusion network, which adopts a spatial neighborhood reconstruction autoencoder architecture, reconstructing the masked modality using the retained modality vectors of the target geographic entity and the heterogeneous modality vector information of its neighboring geographic entities within its spatial neighborhood; during the training of the fusion network, an early stopping strategy is adopted to prevent overfitting, and the reconstruction error of the training model is evaluated on the validation set; when the reconstruction error of the training model is lower than the reconstruction error of the cold start rule scheme by a preset proportion, the unified vector generation method for newly added geographic entities is switched from the rule-based fusion strategy to the trained fusion network, and the existing geographic entities are gradually recalculated into unified vectors through an asynchronous pipeline, achieving a gradual switching from cold start to the training model. The preset threshold is determined by a combination of the total number of geographic entities, the completeness rate of each modality's data, and the spatial distribution density. The preset ratio is determined by balancing the system's requirements for reconstruction accuracy with the constraints of computing resources. Beneficial effects

[0021] Compared with the prior art, the present invention has the following beneficial effects: (1) It breaks through the limitation of existing technologies that can only handle grid-level homogeneous embeddings, extends the idea of ​​mask self-encoding to the level of geographical entities with independent identities, realizes the unified encoding of heterogeneous semantic modal vectors, and solves the technical problem of entity-level heterogeneous modal alignment.

[0022] (2) By using the k-ring algorithm to determine the spatial neighborhood and using the neighboring entity vectors to participate in the reconstruction, the generated unified vector naturally encodes the spatial context semantics, adapts to the irregular spatial distribution of geographic entities, and solves the technical problems of irregular spatial neighbor selection and spatial context encoding.

[0023] (3) The weighting coefficients of the joint loss function are related to the spatial neighborhood level and the modality missing rate, which enables the fusion network to simultaneously optimize reconstruction accuracy and representation discriminability, unlike the joint loss scheme with fixed weights.

[0024] (4) Modal decoupling design is adopted. Each modal vector is independently projected and aligned before being spliced. When a single modal changes, only incremental recalculation is required.

[0025] (5) The cold start and gradual switching scheme takes into account both startup speed and long-term performance, and solves the cold start technical problem of geographic entity multimodal fusion vector in actual deployment.

[0026] Compared to existing technologies, entity-level unified vectors generated using the method of this invention, applied to the same geographical region and test set, significantly outperform existing grid-level embedding methods in geographic semantic retrieval tasks in terms of both accuracy and recall. Incorporating spatial neighbor information into the reconstruction further improves retrieval accuracy compared to schemes that do not incorporate neighbor information. The modality-independent projection alignment strategy significantly reduces reconstruction errors compared to direct stitching schemes. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall process of the multimodal fusion vector generation method for spatial neighborhood reconstruction autoencoder of the present invention; Figure 2 This is a schematic diagram illustrating the spatial discrete grid index and k-ring neighborhood determination of the present invention; Figure 3 This is a flowchart of the cold start and training model switching strategy of the present invention. Detailed Implementation

[0028] Example 1: Overall Method Flow like Figure 1 As shown in the figure, this embodiment provides an overall process for a multimodal fusion vector generation method based on an autoencoder for spatial neighborhood reconstruction.

[0029] Step S1: Heterogeneous Modal Vector Acquisition. Multiple heterogeneous modal vectors of the target geographic entity are acquired through the following methods: visual feature vectors are extracted from visible light or infrared imagery using a lightweight visual feature extraction network; symbol style vectors are generated from traditional map symbol parameter mapping using a symbol encoder, which encodes the parameter information of map symbols (including style identifiers, colors, sizes, and rotation angles) into dense vector representations; geometric location vectors are generated through multi-resolution hierarchical encoding—by inputting the grid cell identifiers of the macro, meso, and micro levels into independent embedding layers and then concatenating them; geographic context vectors are generated by extracting macro-statistical features—including the quantity distribution, density, average confidence, and distance to the nearest similar elements—of the meso-level grid where the target entity is located and its k-ring neighbors, and then compressing them using a multilayer perceptron; structured attribute vectors are generated by concatenating and compressing semantic types through category mapping and numerical attributes through bucket encoding; and knowledge semantic vectors are generated from business knowledge descriptions through a pre-trained text embedding model.

[0030] Step S2: Random Modal Masking. A modal combination masking strategy is adopted. Let the total number of modalities be N, where N is a positive integer not less than 2. In each training iteration, 1 to 2 modalities are randomly selected from the N modalities. The vector values ​​of the selected modalities are set to zero, while the unmasked modalities retain their original vector values, forming a retained modal set. By limiting the number of modes masked each time to no more than two, it is ensured that the retained modal set always contains sufficient information for subsequent reconstruction.

[0031] Step S3: Spatial Neighborhood Determination. Based on the spatial discrete grid index of the target geographic entity—using a hexagonal hierarchical grid system, specifically by mapping the geometric coordinates (latitude and longitude) of the geographic entity to grid cell identifiers at the corresponding resolution level—the spatial neighborhood is determined using the k-ring algorithm, such as... Figure 2 As shown, k is a positive integer (e.g., k=2), meaning it includes the central cell and all neighboring cells within 1 and 2 steps of its distance. Within the defined spatial neighborhood, the database is queried to filter out other existing geographic entities as the set of neighboring entities. Unlike existing technologies based on rule-based adjacency, this embodiment uses the k-ring algorithm to determine the neighborhood on a hexagonal grid. The neighborhood defined by this algorithm is all cells centered on the target cell that are reachable within exactly k steps, which can flexibly adapt to the irregular spatial distribution of geographic entities. The geometric position vector can be generated by concatenating grid cell identifiers of different resolution levels (e.g., 3 levels) after embedding layer encoding, with each level covering a different spatial scale range.

[0032] Step S4: Modal Projection and Neighborhood Aggregation. Each modal vector in the retained modality set undergoes layer normalization and linear transformation sequentially. After being projected onto a unified preset intermediate dimension, they are concatenated in a preset order to form the retained modal representation. The projection operation first performs layer normalization on each modal vector to eliminate dimensional differences and distribution shifts, and then projects it onto the same intermediate dimension (e.g., 256 dimensions) through an independent linear transformation matrix. The preset intermediate dimension is determined by the number of heterogeneous modal vectors and the statistical characteristics of the original dimensions of each modality. Simultaneously, a weighted spatial aggregation based on distance decay is performed on the unified vectors of neighboring geographic entities: Aggregated neighborhood information = Σ_j(exp(-dist_j / σ) × u_j) / Σ_j(exp(-dist_j / σ)), where dist_j is the spatial distance between the target entity and the j-th neighboring entity, u_j is the unified vector of the j-th neighboring entity, and σ is a preset distance decay factor. The value of σ is related to the hierarchical resolution of the spatial discrete grid index (e.g., σ is approximately 100 meters), so that the closer the neighbors are, the greater their contribution weight.

[0033] Step S5: Reconstruct the masked modality using a multimodal fusion network. The fusion network contains at least two hidden layers, each with a predetermined number of neurons (e.g., 1024), employing a non-linear activation function (e.g., ReLU), and using fully connected layers to achieve feature interaction between modalities. The input layer receives the concatenated vector preserving the modalities and the concatenated neighborhood information. The output layer's dimension matches the original dimension of the masked modality, generating the reconstructed vector of the masked modality. When masking multiple modalities simultaneously, the output layer outputs the reconstructed vectors of each masked modality separately.

[0034] Step S6: Joint Loss Optimization. The joint loss function is a weighted average of the mean squared error reconstruction loss and the InfoNCE contrastive learning loss. The reconstruction loss measures the difference between the reconstructed vector and the original masked modality vector. The contrastive learning loss uses the unified vector of the same entity under different masking rounds and the unified vector of entities of the same semantic type in the same neighborhood as positive samples, and other entities in the batch as negative samples. The joint loss function is L_total = α × MSE + (1-α) × InfoNCE, where the first weight coefficient α is greater than the second weight coefficient (1-α), and the weight ratio is dynamically adjusted by the training phase, modality missing rate, and spatial neighborhood density. An adaptive optimizer is used for training.

[0035] Step S7: Generate a unified vector. After training, input all modal vectors of the target geographic entity (without masking) into the fusion network, and output the final unified vector after L2 normalization, so that the unified vector lies on the unit hypersphere, which facilitates the calculation of cosine similarity.

[0036] Example 2: Cold Start and Gradual Switching like Figure 3 As shown, this embodiment describes the gradual switching process from cold start to training model, including the specific operations in four stages.

[0037] Phase 1 (Cold Start): In the initial stage of system deployment, when the number of geographic entities in the database has not reached a preset threshold, the cold start scheme is initiated. L2 normalization is performed on each modality vector: v_normalized = v / ||v||_2, where ||v||_2 is the L2 norm of vector v, thus eliminating modulus differences between modalities. The normalized modality vectors are then directly concatenated according to a preset concatenation order (e.g., visual → symbolic → geometric → context → attribute → knowledge) to generate an initial concatenated vector. Principal component analysis (PCA) is performed on the initial concatenated vector to reduce its dimension to the target dimension, retaining the top principal components with the largest variance contribution. The dimensionality-reduced vector is stored as the unified cold start vector in the vector database, using the entity identifier as the index key. This phase does not rely on training data and can be used immediately after the system goes live.

[0038] Phase Two (Data Accumulation): Monitoring the amount of training data accumulated. When the number of geographic entities with complete multimodal data in the database reaches a preset threshold—which is determined by the total number of geographic entities, the completeness rate of each modality's data, and the spatial distribution density (e.g., 1000 geographic entities with complete six modal data)—the system automatically starts the training data collection process and marks the entity data that meets the conditions as training sample candidates.

[0039] Phase 3 (Model Training): The multimodal fusion network is trained using the collected training data. An early stopping strategy is employed during training to prevent overfitting: after each training epoch, the reconstruction error is calculated on the validation set. If the reconstruction error on the validation set does not decrease for several consecutive epochs (e.g., 5 epochs), training is automatically stopped and the current optimal model parameters are saved.

[0040] Phase Four (Gradual Switching): Evaluate the reconstruction error of the trained model on the test set. When the reconstruction error of the trained model is lower than the reconstruction error of the cold start rule scheme by a preset percentage—this percentage is determined by balancing the system's requirements for reconstruction accuracy with the constraints of computing resources (e.g., below 20%)—a gradual switch is executed. First, the unified vector generation method for newly added geographic entities is switched from the rule-based fusion strategy to the trained fusion network; for existing geographic entities, vector recalculation and replacement are performed one by one through an asynchronous task queue to avoid impacting the performance of online services. The cold start scheme is retained as a fallback option during the switch.

[0041] Example 3: Incremental Vector Recalculation This embodiment takes the user's correction of entity attributes in situation analysis as an example to describe the specific implementation process of incremental vector recalculation.

[0042] Scenario: In situational analysis scenarios, entity attributes generated by automatic identification systems may contain errors. Users can correct these attributes through an interactive interface. For example, a user might change the semantic type of a geographic entity from "unknown vehicle" to "warning post" and adjust its symbol style accordingly.

[0043] Step 1: Receive Correction Instruction. The system receives the user's correction instruction through the human-computer interaction interface. The correction instruction includes the unique identifier of the target entity, the corrected semantic type, and the corrected symbolic parameters. The system first verifies the validity of the target entity identifier, then compares the differences in fields before and after the correction to determine the scope of the impact.

[0044] Step 2: Update entity attributes. The system updates the semantic type field and symbolic parameter field of the target entity in the database. The update operation is completed within a transaction to ensure data consistency.

[0045] Step 3: Trigger Incremental Vector Recalculation. After the attribute update is complete, the system automatically triggers the incremental vector recalculation task. The key is that, thanks to the modal decoupling design of this invention—each modal vector is independently projected and aligned before being concatenated in step S4—the incremental recalculation only needs to recalculate the vectors of the affected modalities and the final unified vector. Specifically: the symbolic style vector is recalculated (because the symbolic parameters have changed); the structured attribute vector is recalculated (because the semantic type has changed); the updated symbolic style vector and structured attribute vector are reconcatenated with the unchanged visual feature vector, geometric location vector, geographic context vector, and knowledge semantic vector, and input into the trained multimodal fusion network to generate the updated unified vector. Other unaffected modal vectors remain unchanged and do not need to be recalculated.

[0046] Step 4: Cascaded Update. Since the unified vector encodes spatial context relationships—a characteristic imparted by the spatial neighbor information introduced during the autoencoder training process for spatial neighborhood reconstruction—changes in the unified vector of a corrected entity will affect the geographic context vectors of other entities within its spatial neighborhood. The system automatically queries all neighboring entities within the grid cell containing the entity and its k-ring neighborhood, marks the geographic context vectors of these neighboring entities as pending updates, and triggers a cascaded recalculation of the context vector.

[0047] Step 5: Synchronize and Update. Write the updated unified vector and geographic context vector to the vector database and update the vector index. Push the change event to all connected clients via a publish-subscribe mechanism to ensure that the situation map of each client is synchronized within the preset time window.

[0048] This embodiment demonstrates the advantages of modal decoupling design in practical applications: a single attribute correction only triggers the recalculation of a small number of modal vectors and unified vectors, avoiding the performance bottleneck of full recalculation, and enabling the system to meet the high real-time requirements of situational analysis scenarios.

Claims

1. A multimodal fusion vector generation method for spatial neighborhood reconstruction autoencoder, characterized in that, Includes the following steps: Step S1: Obtain multiple heterogeneous modal vectors of the target geographic entity. The heterogeneous modal vectors include at least two of the following: visual feature vectors, symbolic style vectors, geometric location vectors, geographic context vectors, structured attribute vectors, and knowledge semantic vectors. The target geographic entity has a unique entity identifier, a timestamp, and a set of heterogeneous modal vectors. Step S2: Perform a random masking operation, select one or two modal vectors from the multiple heterogeneous modal vectors of the target geographic entity for masking, and obtain a masked modal set and a retained modal set; Step S3: Based on the unique identifier of the target geographic entity and its associated geometric coordinates, filter out neighboring geographic entities with independent multimodal attributes from the irregularly distributed discrete entity set through spatial discrete grid indexing and k-ring algorithm; Step S4: Project each modal vector in the retained modal set of the target geographic entity to a unified intermediate dimension through layer normalization and linear transformation and then concatenate them. At the same time, spatially aggregate the unified vectors of the neighboring geographic entities through a weighted average mechanism to obtain aggregated neighborhood information. Step S5: Input the concatenated preserved modality representation and the aggregated neighborhood information into the multimodal fusion network. The fusion network contains at least two hidden layers, each with a preset number of neurons. It uses a nonlinear activation function and performs feature interaction through a fully connected layer to generate the reconstructed vector of the masked modality. Step S6: Calculate the reconstruction loss between the reconstructed vector and the original masked modal vector, and simultaneously calculate the contrastive learning loss between the unified vector of the target geographic entity and the unified vectors of other entities in the batch. Optimize the multimodal fusion network with a joint loss function, where the joint loss function is a weighted combination of the reconstruction loss and the contrastive learning loss. Step S7: Input the optimized modal vectors into the fusion network to generate a unified vector representation of the target geographic entity.

2. The method according to claim 1, characterized in that, It also includes joint optimization methods, specifically including: A portion of the heterogeneous modal vectors is masked, while the remaining modal vectors are retained. Using the retained modal vectors and the heterogeneous modal vector information of neighboring geographic entities within the spatial neighborhood of the target geographic entity, the masked modality is reconstructed through a multimodal fusion network to obtain the reconstructed vector. Calculate the reconstruction loss between the reconstructed vector and the original masked modality vector, and simultaneously calculate the contrastive learning loss between the unified vector of the target geographic entity and the unified vectors of other geographic entities in the batch. The contrastive learning loss uses the unified vectors generated by the same geographic entity under different masking rounds as positive samples, the unified vectors of geographic entities with the same semantic type in the same spatial neighborhood as positive samples, and the unified vectors of other entities in the batch that do not belong to the positive samples as negative samples. The multimodal fusion network is optimized by using a weighted combination of the reconstruction loss and the contrastive learning loss as a joint loss function, wherein the weighting coefficients of the joint loss function are related to the spatial neighborhood level to which the target geographic entity belongs and the modality missing rate of the heterogeneous modality vector.

3. The method according to claim 1, characterized in that, It also includes cold start and gradual switching methods, specifically including: During the cold start phase, when the amount of training data has not reached the preset threshold, a rule-based multimodal vector fusion strategy is adopted: each modal vector is normalized and then concatenated to generate a unified vector through dimensionality reduction. Monitor the amount of training data accumulated, and when the amount of training data reaches the preset threshold, start training of the multimodal fusion network; During the training of the fusion network, an early stopping strategy is employed to prevent overfitting, and the reconstruction error of the trained model is evaluated on the validation set. When the reconstruction error of the training model is lower than the reconstruction error of the cold start rule scheme by a preset ratio, a switch is triggered, and the unified vector generation method of the newly added geographic entities is switched from the rule-based fusion strategy to the fusion network obtained by training. The existing geographic entities are gradually recalculated with unified vectors through an asynchronous pipeline.

4. The method according to claim 1, characterized in that, The random masking operation in step S2 adopts a modal combination masking strategy. In each iteration, one or two modalities are randomly selected from N modalities for masking. The mask value is set to a zero vector, and the unmasked modalities retain their original vector values. N is a positive integer not less than 2.

5. The method according to claim 1, characterized in that, The spatial discrete grid index in step S3 adopts a hexagonal hierarchical grid system. By mapping the geometric coordinates of geographic entities to grid cell identifiers, the k-ring algorithm is used to determine the neighboring geographic entities in the spatial neighborhood, where k is a positive integer.

6. The method according to claim 5, characterized in that, The spatial discrete grid system adopts a multi-resolution hierarchical index, including macroscopic, mesoscopic and microscopic levels, each level covering a different preset area range. The geometric position vector is generated by inputting the grid cell identifiers of each level into independent embedding layers and splicing them together.

7. The method according to claim 1, characterized in that, The projection operation in step S4 performs layer normalization and linear transformation on each modal vector in sequence, projecting each modal vector to the same preset intermediate dimension. The preset intermediate dimension is determined by the number of heterogeneous modal vectors and the statistical characteristics of the original dimensions of each modality.

8. The method according to claim 1, characterized in that, The spatial aggregation in step S4 adopts a distance-attenuation-based weighted average mechanism: Aggregated neighborhood information = Σ_j (w_j × u_j) / Σ_j w_j, where u_j is the unified vector of the j-th neighboring geographic entity, w_j is the attenuation weight based on spatial distance, w_j = exp(-dist_j / σ), dist_j is the spatial distance between the j-th neighboring entity and the target entity, and σ is a preset distance attenuation factor. The value of σ is related to the hierarchical resolution of the spatial discrete grid index.

9. The method according to claim 1 or 2, characterized in that, The reconstruction loss is mean squared error loss, the contrastive learning loss is InfoNCE loss, and the value of the first weight coefficient in the joint loss function is greater than that of the second weight coefficient. The ratio of the first weight coefficient to the second weight coefficient is dynamically adjusted by the training phase, modality missing rate, and spatial neighborhood density.

10. The method according to claim 1 or 3, characterized in that, In the multimodal fusion network, the number of neurons in each hidden layer decreases or remains the same layer by layer. The output layer dimension of the fusion network matches the original dimension of the masked modality. When masking multiple modalities, the output layer outputs the reconstruction vector of each masked modality.

11. The method according to claim 1, characterized in that, The unified vector generation in step S7 uses L2 normalization to make the unified vector lie on the unit hypersphere, which facilitates the calculation of cosine similarity.

12. The method according to claim 3, characterized in that, The preset threshold is determined by a combination of the total number of geographic entities, the completeness rate of each modality data, and the spatial distribution density. The preset ratio is determined by balancing the system's requirements for reconstruction accuracy with the constraints of computing resources.

13. The method according to claim 1 or 2, characterized in that, The unified vector of the target geographic entity is generated through the forward propagation of the fusion network. During the training process, the fusion network learns the nonlinear interaction relationships between various modalities and the semantics of spatial neighborhood context. The unified vector encodes the multimodal semantics of the target geographic entity itself and the contextual information from the spatial neighborhood.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 13.

15. A multimodal fusion vector generation device for spatial neighborhood reconstruction autoencoder, characterized in that, include: The modality acquisition module is used to acquire multiple heterogeneous modality vectors of the target geographic entity; The mask module is used to perform random masking operations; The spatial indexing module is used to determine nearby geographic entities based on a spatial discrete grid index. The projection stitching module is used to project vectors of various modalities onto a unified dimension and stitch them together; the neighborhood aggregation module is used to spatially aggregate the unified vectors of neighboring geographic entities. The fusion network module is used to generate the reconstructed vector of the masked modality and the final unified vector through a multilayer perceptron; the optimization module is used to optimize the parameters of the fusion network module with a joint loss function; the cold start module is used to adopt a rule-based fusion strategy when training data is insufficient and switch to the training model when data accumulation reaches a preset threshold.

Images