Entity alignment method based on large language model adaptive fusion

Abstract

The invention discloses an entity alignment method based on large language model adaptive fusion. The method comprises the steps that multi-modal data is preprocessed, multi-modal data features are obtained, and the multi-modal data features comprise semantic features corresponding to text data, image features corresponding to image data and topological relation features corresponding to geographic space data; mapping the multi-modal data features to a unified semantic space to obtain an entity representation vector in the unified semantic space; for the entity representation vectors in the unified semantic space, dynamic weights are distributed according to confidence coefficients of different modal data, then weighted fusion entity representation is calculated based on the dynamic weights, alignment is carried out through comparative learning and attention mechanism matching, and aligned multi-modal fusion entity representation is obtained; and for the aligned multi-modal fusion entity representation, performing entity matching and geographic coordinate calibration to obtain an entity matching result. According to the invention, the accuracy and robustness of entity alignment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to an entity alignment method based on large language model adaptive fusion. BACKGROUND

[0002] With the rapid development of Internet of Things, artificial intelligence and geographic information technology, the fusion and entity alignment of multi-modal data have become the core technical requirements in the fields of smart city, transportation planning, environmental monitoring, etc. Entity alignment aims to identify and associate different representations of the same entity from heterogeneous data sources. However, existing methods face many challenges when dealing with multi-modal data, mainly including the following problems: 1) semantic gap between modalities. The semantic expressions of different modal data are significantly different. Text description may emphasize functional attributes, image focuses on visual features, and geographic data focuses on spatial coordinates. Traditional methods such as rule-based feature mapping or simple embedding alignment are difficult to capture cross-modal deep semantic association, resulting in limited alignment results due to shallow feature matching. 2) Data quality imbalance and noise interference. The quality of multi-modal data in actual scenarios is significantly different, for example, text may have description ambiguity; images are often affected by light changes, occlusions or low resolution; and geographic coordinates have collection errors. Existing methods such as multi-modal joint embedding use fixed weight fusion strategy, without considering the information reliability of each modality in a single sample. 3) Limitations of heterogeneous data modeling. The topological relationship of geographic spatial data and the semantic association of image / text are difficult to model uniformly. Traditional graph neural networks usually process spatial relationships with static adjacency matrices, ignoring dynamic spatio-temporal changes. In addition, existing methods are insufficient in designing multi-modal feature interaction mechanisms, such as text-image attention only stays at the feature concatenation level, without realizing fine-grained semantic alignment. 4) Trade-off between alignment accuracy and computational efficiency. The trade-off between alignment accuracy and computational efficiency. High-precision alignment usually relies on complex models such as multi-modal Transformer, but the computational overhead limits real-time applications. For example, the proposed hypergraph convolution network, although it improves the accuracy by 8%, the inference time increases by 3 times, which is difficult to meet the timeliness requirements of city-level entity alignment. In addition, existing methods pay insufficient attention to geographic calibration, relying only on coordinate distance threshold for post-processing, without embedding spatial constraints in the feature fusion stage, resulting in multiple iterations of error correction.

[0003] In summary, the existing entity alignment method is difficult to cope with the semantic gap and quality difference between text, image and geospatial data due to excessive reliance on a single data source or static fusion strategy, resulting in limited alignment accuracy and inability to dynamically adapt to noisy interference and incomplete information scenarios, which has significant limitations and one-sidedness. In addition, in traditional multi-modal fusion, there are still deficiencies in dynamic evaluation of feature confidence, cross-modal fine-grained interaction and embedded optimization of geospatial constraints, lacking precise modeling of the internal association of heterogeneous data, thereby restricting the practicality and reliability of large-scale entity alignment. Therefore, how to effectively fuse multi-source heterogeneous data, dynamically balance the quality of each modal information through an adaptive mechanism, and achieve cross-modal semantic consistency enhancement and geospatial precise calibration has become a difficult problem that the existing technology needs to break through. SUMMARY

[0004] The purpose of the present application is to overcome the above-mentioned defects of the prior art and provide an entity alignment method based on adaptive fusion of large language models. The method comprises the following steps:

[0005] Preprocessing the multi-modal data to obtain multi-modal data features, wherein the multi-modal data includes text data, image data and geospatial data, and the multi-modal data features include semantic features corresponding to the text data, image features corresponding to the image data and topological relationship features corresponding to the geospatial data;

[0006] Mapping the multi-modal data features to a unified semantic space to obtain entity representation vectors in the unified semantic space;

[0007] For the entity representation vectors in the unified semantic space, dynamically assigning weights according to the confidence of different modal data, and then calculating the weighted fusion entity representation based on the dynamic weights, and performing alignment through contrast learning and attention mechanism matching to obtain aligned multi-modal fusion entity representation;

[0008] For the aligned multi-modal fusion entity representation, entity matching and geospatial coordinate calibration are performed to obtain the entity matching result.

[0009] Compared with the prior art, the entity alignment method based on adaptive fusion of large language models provided by the present application performs entity alignment based on dynamic weight assignment and adaptive fusion of contrast learning, improves the alignment robustness in noisy environments by introducing confidence-driven weight adjustment, improved deep network architecture (such as ResNet with embedded channel-space dual attention, GraphSAGE with dynamic edge weight in space-time and joint training strategy), solves the problems of multi-modal data conflict, semantic inconsistency and static model rigidity, and provides an efficient and reliable entity alignment solution for smart city, geographic information system and other scenarios.

[0010] Other features of the present application, its nature and advantages will become more apparent from the detailed description of exemplary embodiments of the application which follows, in conjunction with the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0012] Figure 1 is a flow chart of a large language model based adaptive fusion entity alignment method according to an embodiment of the present application;

[0013] Figure 2 is a data flow diagram of a large language model based adaptive fusion entity alignment method according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present application unless otherwise specifically stated.

[0015] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the scope of the application, its application, or uses.

[0016] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0017] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of exemplary embodiments can have different values.

[0018] It should be noted that like reference numerals and letters refer to like items in the drawings, and thus, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0019] In general, this invention improves the alignment accuracy and robustness of text, image and geospatial data in a unified semantic space by introducing text semantic extraction based on large language models (such as DeepSeek) into traditional multimodal entity alignment methods, embedding attention mechanisms into CNN (convolutional neural networks), and introducing dynamic graph update mechanisms into GNN (graph neural networks), and combining adaptive dynamic weight allocation with contrastive learning-attention mechanisms. This effectively solves the problems of semantic inconsistency of heterogeneous data, high data noise and insufficient alignment accuracy in the existing technology, and provides efficient and accurate technical support for complex spatial queries and knowledge graph construction.

[0020] Specifically, combined Figure 1 and Figure 2 As shown, the provided entity alignment method based on adaptive fusion of a large language model includes the following steps:

[0021] Step S110 , preprocessing the acquired multimodal data to obtain multimodal data features, wherein the multimodal data features include semantic features corresponding to the text data, image features corresponding to the image data, and topological relationship features corresponding to the geographic space data.

[0022] Step S110 implements multimodal data acquisition and preprocessing. For example, text data, image data, and geospatial data related to geographic entities are acquired. Text descriptions include descriptions of subway stations, buildings, public facilities, etc. Semantic features are then extracted from the text data to generate text embedding vectors (or semantic features); visual features are extracted from the image data to generate image embedding vectors (or image features); and topological features are extracted from the geospatial data, primarily by analyzing its structure and the relationships between nodes to extract valuable features and generate geospatial embedding vectors (or topological relationship features).

[0023] 1) Preprocessing of text data

[0024] Specifically, text data preprocessing includes basic cleaning of the raw text, such as removing special symbols and HTML tags, performing word segmentation and stop word filtering, and normalization. The cleaned text is fed into the encoder of an improved large language model, which uses multi-head self-attention to capture global context. The output is then embedded in the MGAP (Multi-Granularity Adaptive Pooling) module. Through a learnable gating network, word vector weights are dynamically assigned based on the information content of each word, sentence, or paragraph, enhancing key information. Multi-level features are then weighted and aggregated. The resulting multi-scale semantic feature vector is then normalized and reduced in dimensionality to generate the final high-dimensional but computationally friendly text embedding representation.

[0025] In an embodiment, the encoder of the large language model selects DeepSeek-Encoder. The DeepSeek-Encoder is a Transformer architecture-based encoder, including twelve layers of multi-head self-attention modules and feedforward networks. After the output of the last layer of the DeepSeek-Encoder, a multi-granularity adaptive pooling module (MGAP) is embedded. The MGAP module dynamically allocates word-level, sentence-level, and paragraph-level vector weights through a learnable gating network. The MGAP module first performs gated weighted aggregation on the word-level vector, then performs average pooling on the sentence-level and paragraph-level vectors and weights them through the gating network, and finally linearly combines the aggregation results of each granularity with learnable fusion weights to generate a multi-granularity text embedding representation.

[0026] 2) Preprocessing of image data

[0027] In an embodiment, image enhancement (such as adjusting brightness and contrast), size cropping to 224x224, and noise removal are performed on the image data for preprocessing. The preprocessed image is input into a convolutional neural network (CNN) such as a ResNet 50 backbone network, and a channel attention module SE Block is embedded after the CNN to strengthen key area features, generating an image embedding representation. The SE Block automatically focuses on the channels related to geographic entities in the image, suppresses background interference, and extracts the resulting feature map through global pooling and normalization to generate an image embedding representation:

[0028] F out =F in ×σ(W2ReLU(W1GAP(F in ))) (1)

[0029] wherein GAP is global average pooling, W1 and W2 are trainable parameters, σ is a Sigmoid function, F in represents the feature input into the SE-ResNet 50 (i.e., the network combining ResNet 50 and embedded channel attention module), and F out represents the generated image embedding representation.

[0030] 3) Preprocessing of geospatial data

[0031] In an embodiment, the latitude and longitude of each geographic entity and its adjacency / road network topological relationship are collected and converted into the WGS84 coordinate system. The structured topological graph is input into a graph neural network GraphSAGE (Graph Sample and Aggregated) backbone network for graph node embedding learning, and a dynamic edge weight update mechanism is introduced at each message passing, represented as:

[0032]

[0033] wherein, denotes the edge weight between node u and node v in the kth iteration, sim geo denotes the similarity calculated based on geospatial proximity, which is the weighted sum of the geographical distance (e.g. Euclidean distance) and the trajectory overlap within the time window of node u and v. (u, v) denotes a pair of adjacent nodes in the geospatial topology graph, and k denotes the number of message passing iterations of the graph neural network. geo The similarity is calculated based on spatiotemporal proximity (distance, time window), and a is a dynamic update rate that can be set according to actual needs, used to control the balance between historical weight and new weight.

[0034] The dynamically updated edge weight enables the model to automatically adjust the weight of aggregated neighbor information in response to noise or sudden changes in topology, and ultimately generate geospatial embedding representation through node feature aggregation and normalization.

[0035] In summary, the text data processing uses an improved DeepSeek-Encoder, which embeds a multi-granularity adaptive pooling MGAP module based on the traditional Transformer architecture. Through a learnable gating mechanism, the module dynamically allocates word vector weights, strengthens key entities, and supports dynamic semantic aggregation from word level, sentence level to paragraph level, extracting multi-granularity semantic features. The image data processing uses an improved ResNet-50 network, which embeds a channel attention module after the convolutional layer, automatically focusing on the key areas related to geographic entities in the image, enhancing the ability to extract key features under noise interference, and improving the visual feature extraction effect. The geospatial data processing uses an improved GraphSAGE graph neural network, which introduces a dynamic edge weight update mechanism, and optimizes the modeling of topological relationships in combination with the spatiotemporal proximity of geographic entities.

[0036] In step S120, the multi-modal features are mapped to a unified semantic space to obtain entity representation vectors in the unified semantic space.

[0037] The goal of step S120 is to map the embedding representations of text, image and geospatial data into a unified high-dimensional semantic space, thereby providing a common representation basis for subsequent entity alignment. The multi-modal features can be mapped to a unified semantic space through a pre-trained multi-modal fusion model.

[0038] In one embodiment, a multi-flow Transformer fusion model (or TriFuser-X) is constructed, and independent encoder branches are designed to process text embedding, image embedding and geospatial embedding respectively, and fine-grained interaction is achieved through cross-modal attention. The text embedding, image embedding and geospatial embedding obtained above are input into a pre-trained multi-modal fusion model, and the text, image and geospatial embedding vectors are mapped to a unified semantic space through a cross-modal attention mechanism. The multi-modal fusion model can be trained jointly, and the reconstruction loss and the contrast loss are combined to map the data of each modality to a unified high-dimensional semantic space, so as to achieve semantic consistency and complementarity. In the unified semantic space, for each entity, the contrast loss (InfoNCE) and the modal reconstruction loss (MSE) are combined to achieve end-to-end optimization and adaptive alignment, and the entire network is optimized through end-to-end back propagation.

[0039] Specifically, for the multi-flow Transformer fusion model TriFuser-X, after preliminary processing, each branch enters a separate encoder, and local features are extracted in these encoders; then, in the cross-modal fusion layer, the features of each branch are interacted through the multi-head self-attention mechanism, so that the text, image and geographic information can complement each other. To ensure the semantic consistency of the fused features, when pre-training the multi-modal fusion model, a joint training loss is designed, including a reconstruction loss (for restoring the original information) and a contrast learning loss (for narrowing the distance between different modalities of the same entity and widening the distance between different entities), and the entire network is optimized through end-to-end back propagation.

[0040] The features processed by the cross-modal fusion network are output to a unified high-dimensional semantic space to form global embedding representation. At this time, the representation of each entity integrates information from three modalities of text, image and geospatial, and the features of each modality remain highly consistent in the semantic space, which facilitates subsequent entity alignment and matching.

[0041] 1) Constructing a multi-modal input branch

[0042] The multi-modal input branch includes a text branch, an image branch and a geospatial branch.

[0043] For the text branch, the extracted text embedding vector is used as input. The vector is processed by normalization and dimension reduction to ensure data density and computational efficiency.

[0044] For the image branch, a CNN (such as an improved ResNet-50) is used, and an image embedding is added after the convolutional layer as input. The attention module further enhances the expression ability of visual features by weighting the key regions in the feature map.

[0045] For the geospatial branch, GraphSAGE based on graph attention network is used, and a dynamic graph update module is embedded in the geospatial embedding representation as input. The dynamic graph update module can dynamically adjust the node features to capture the real spatial structure after complex topological relationship and noise interference.

[0046] 2) Cross-modal fusion network design

[0047] In one implementation, a multi-stream Transformer fusion model TriFuser-X is constructed, which includes three independent encoder branches, namely: a text encoder, for example, a 12-layer Transformer, input Et, output dimension 768; an image encoder, for example, a 6-layer Transformer, input Et, dimension reduced to 768; a geographic encoder, for example, a 4-layer Transformer, input Et, dimension increased to 768.

[0048] The cross-modal fusion network adopts a cross-modal attention mechanism, for example, a multi-head cross-attention layer (Multi-Head Cross-Attention) is used to calculate the interaction weights between text-image, image-geography, and text-geography. For example, the interaction weights can be uniformly represented as:

[0049]

[0050] where Q mx is the query vector from modality x, K my is the key vector from modality y, V my is the query vector from modality y, d k is the dimension of the key vector, is the interaction weight matrix between modality x and modality y.

[0051] For the cross-modal fusion network, a joint training strategy is designed, for example, a hybrid loss is designed, including a contrastive loss and a reconstruction loss. The contrastive loss (InfoNCE) is used to maximize the similarity of multi-modal representations of the same entity (positive sample pair) and minimize the similarity of different entities (negative sample pair), to ensure that tight clusters are formed in the fusion space. The modality reconstruction loss (MSE) is used to reconstruct the original data from the unified embedding through the decoder, to ensure semantic preservation and prevent the contrastive loss from losing single-modal details, to achieve complementary fusion of multi-source information. Through the joint training strategy, end-to-end semantic consistency and complementary mapping are achieved. The features processed by the cross-modal fusion network are output to a unified high-dimensional semantic space to form a global embedding representation.

[0052] In summary, the multi-stream Transformer fusion model TriFuser-X includes three independent encoder branches, respectively processing text, image and geographic embedding; the cross-modal attention mechanism adopts a multi-head cross-attention layer to calculate the interaction weights between text-image, image-geography, and text-geography, realizing feature complementation; the joint training strategy designs a hybrid loss function, including the contrast loss InfoNCE and the modal reconstruction loss MSE. The InfoNCE contrast loss is used to measure the closeness of the cross-modal representations of the same entity and the separability of the representations of different entities, and then the modal reconstruction loss is calculated, which is mainly used to measure whether the fusion representation can accurately reconstruct the original embedding of each modality to ensure information fidelity. Finally, the two are combined into a total loss according to the preset weight and a back propagation and parameter update are performed.

[0053] In step S130, for the entity representation vector in the unified semantic space, an attention mechanism is used to dynamically allocate weights according to the confidence and contribution of each modality feature, and then a weighted fusion entity representation is calculated based on the dynamic weights, and the multi-modal fusion entity representation is obtained by aligning through contrast learning and attention mechanism matching.

[0054] For each geographic entity, the quality and information richness of the text, image and geographic space features are evaluated respectively. For example, the text embedding generated by DeepSeek can be combined with the semantic integrity score to determine the reliability of the description. For image data with more noise, the visual feature clarity and background interference can be scored. Using an attention mechanism, weights are dynamically allocated according to the confidence and contribution of each modality feature, for example, a higher weight is given to a text with clear and semantically complete description, and a moderate weight is allocated to an image feature with high quality after noise filtering. Finally, a weighted fusion entity representation is generated, which can reflect the complementarity of the information of each modality.

[0055] Specifically, for each entity to be aligned, the feature quality indicators in the text, image and geographic space data are calculated respectively. The text branch uses the semantic integrity score output by the DeepSeek-Encoder as the reliability of the text feature. The image branch evaluates the key region score and image noise indicator output by the attention module. The geographic space branch measures the local consistency and neighborhood signal-to-noise ratio calculated during the node aggregation process.

[0056] The fusion representations of the same entity in different modalities are taken as positive samples to form a positive sample pair, and the fusion representations of semantically similar but actually different entities are selected as negative samples to form a negative sample pair. The model is trained using contrast loss functions such as InfoNCE, so that the distance between positive samples in the unified semantic space is as close as possible, while the distance between negative samples is as far as possible, thereby improving the discrimination and alignment accuracy of the entity representation.

[0057] In the fused semantic space, a cross-modal attention layer is set to automatically focus on the features most relevant to the current alignment task. This attention layer fine-tunes the fused entity representation, adaptively amplifies the weights of key features according to the context information, further filters noise information, and ensures efficient matching between multi-modal features.

[0058] Step S130 implements an adaptive alignment mechanism by designing adaptive dynamic weight distribution, contrastive learning, and attention mechanism to achieve fine alignment of multi-modal data. The following will specifically introduce the embodiments of dynamic weight distribution, contrastive learning strategy, and attention mechanism matching.

[0059] 1) Dynamic weight distribution

[0060] Text confidence evaluation S t A pre-trained semantic integrity analysis model (e.g., based on the Transformer architecture) is used to parse the grammatical structure of the text description and calculate the information density.

[0061] Image confidence evaluation Si, using an improved ResNet-50 network, embedding a channel attention module SE Block after the Conv4_x layer. The calculation formula is:

[0062] Si = Sigmoid(β·‖F image ‖2+γ·Entropy(F image )) (4)

[0063] Where F image is the image feature extracted by ResNet-50. ‖F image ‖2 represents the L2 norm of the image feature, which measures the feature strength. Entropy(F image ) is the information entropy of the image feature, which evaluates the feature diversity. β and γ are learnable parameters that balance strength and diversity.

[0064] Geographical confidence evaluation S g An improved GraphSAGE graph neural network is used to calculate the local consistency score of nodes through a dynamic edge weight update mechanism (formula 2), which is specifically:

[0065]

[0066] Where N(u) is the neighbor set of node u, |N(u)| is the number of neighbors of node u, is the dynamic edge weight between node u and node v in the kth iteration, sim geo (u,v) represents the geographical spatio-temporal proximity similarity.

[0067] According to the confidence score (S t , Si, S g ) of each modality, the contribution rate of each modality feature in the fusion representation is adaptively adjusted by a dynamic weight distribution formula:

[0068]

[0069] where s m represents the confidence score of modality m, m represents the modality of data, m' represents traversing all modalities, T represents a temperature coefficient for adjusting the smoothness of weight distribution, which can be set according to actual needs. t represents text, i represents image, and g represents geographic spatial data.

[0070] According to the evaluation results of each modality, the contribution rate of each modality feature in the fusion representation is dynamically adjusted by an adaptive algorithm. Finally, the weighted fused entity representation is generated by using the following formula:

[0071] F=w t ·F text +w i ·F image +w g ·F geo (7)

[0072] where w t , w i , and w g are the dynamic weights of text, image, and geographic data, respectively, and their values are adaptively determined according to the confidence of each feature. F text represents the feature of text data, F image represents the image feature, and F geo represents the feature of geographic spatial data.

[0073] 2) Contrastive learning strategy

[0074] In the contrastive learning strategy, for each anchor representation z i (from modality m), its modality confidence w m is used as a weighting coefficient to adjust the similarity calculation in the InfoNCE loss:

[0075]

[0076] where L InfoNCE represents the InfoNCE contrastive loss function, is the positive sample, is the negative sample, and τ is the temperature coefficient. τ→0 means that the similarity difference is amplified, and the model pays more attention to difficult samples. τ→∞: the similarity difference is smoothed, and the model treats all samples equally. w mdynamic weights of modal m, cosine similarity between anchor sample and positive sample.

[0077] In the Hard Negative Mining (HNM) step, the weighted similarity prior is used to filter out "hard negative samples", i.e., the negative samples with the highest similarity to the anchor in the high-weight modal are selected first for InfoNCE calculation, so that the model can learn on the most discriminative samples.

[0078] 3) Attention mechanism matching

[0079] In the attention mechanism matching module, the three-modal weights are integrated into the generation of the learnable query vector, and the dynamic weights are used for the generation of the key / value, so that the attention output focuses more on the information channel of the high-quality modal, and the final output is a set of finely aligned multi-modal entity representations.

[0080] The above weighted InfoNCE and attention module output jointly participate in backpropagation, ensuring that the alignment accuracy and information integrity of the three modalities are optimized simultaneously in a single training, achieving true end-to-end adaptive alignment.

[0081] In summary, the dynamic weight allocation module is based on feature confidence scores, where the text branch uses the semantic integrity score output by the DeepSeek-Encoder as the reliability of the text features. The image branch is evaluated based on the key region score output by the attention module and the image noise indicator. The geospatial branch is measured by the local consistency and neighborhood signal-to-noise ratio calculated during the node aggregation process. The contrast learning strategy uses the hard negative sample mining technique to select difficult negative samples, enhancing the intra-class closeness of multi-modal representations of the same entity. The attention mechanism matching deploys a learnable query vector in a unified semantic space, focusing on key feature regions through scaled dot product attention.

[0082] Step 140, using the unified multi-modal fusion entity representation, calculate the similarity score between entities, match the text description of geographical entities with corresponding entities in image data and geospatial data, and calibrate with geographic coordinate information.

[0083] This step S140 aims to use the multi-modal unified representation to calculate the similarity score between entities, match the text description of geographical entities with corresponding entities in image and geospatial data, and calibrate with geographic coordinate information to ensure the accuracy and robustness of the alignment results. Based on the above-mentioned construction of multi-modal unified semantic representation, the accurate matching of geographical entities in different source data is realized, and the spatial consistency calibration is combined with the geographic coordinate information to ensure the accuracy and spatial reasonableness of the final matching results.

[0084] In one embodiment, the similarity between entities is calculated using cosine similarity based on the weighted fused entity representation obtained in step S130. According to a preset similarity threshold, the entity pairs with higher matching degree are preliminarily screened out. For the preliminarily matched entity pairs, the corresponding geographic coordinate information is extracted. The actual geographic distance between the entity pairs is calculated using the spatial indexing technology KD-Tree, and compared with the preset distance threshold to exclude the matching pairs with obvious deviation in spatial position. If inconsistency is detected in the geographic coordinates in the text, image and geographic data, a correction algorithm based on geocoding is introduced, i.e. the coordinates are fine-tuned by a geographic coordinate mapping network based on deep learning, to ensure that the final matched entities are highly consistent in spatial position.

[0085] When there is a conflict between different modalities (e.g. the text description and the image information are not completely consistent), a hybrid expert system can be used for decision-making, i.e. combining dynamic weights, attention scores and geographic coordinate calibration results for weighted decision-making to output the final alignment result. Finally, a multiple verification mechanism is used to post-process the matching result to ensure that all aligned entities meet the predetermined requirements in terms of semantics and spatial position.

[0086] In the unified semantic space, each entity is matched and calibrated to ensure the final alignment accuracy of multi-modal data, and the specific implementation includes multi-modal entity matching and geographic coordinate calibration.

[0087] 1) Multi-modal entity matching

[0088] For each pair of entity representations, the similarity score in the unified semantic space is calculated, and the similarity calculation adopts a weighted fusion strategy, which is represented as:

[0089] Sim(e1,e2) = 0.7 · cos(v1,v2) + 0.3 · exp(-0.001 · Haversine(c1,c2)) (9)

[0090] where (e1,e2) represents the two entities to be matched, v1,v2 represents the vector representation of entities e1 and e2 in the unified semantic space, c1,c2 represents the geographic coordinates (latitude and longitude) of entities e1 and e2. cos(v1,v2) represents the cosine similarity of vectors v1 and v2, Haversine(c1,c2) represents the geographic distance calculated based on the Haversine formula, and exp(-0.001 · Haversine(c1,c2)) represents mapping the geographic distance to a similarity score (value range (0,1]).

[0091] A similarity threshold δ is set to screen possible matching entity pairs, and the entity pairs that meet the threshold requirement are selected as the preliminary matching result to form a matching candidate set M.

[0092] 2) Geographical coordinate calibration

[0093] The geographical coordinate information carried by each pair of entities in the input matching candidate set is inputted, and the geographical spatial distance of each pair of entities (e i ,e j ) is calculated. The R-Tree spatial index is used to quickly retrieve the adjacent entities, and if the geographical distance of the matching pair exceeds the threshold, the re-matching mechanism is triggered. For the matching pair with reasonable distance, the alignment result is updated to "final confirmed matching". The final entity matching result is represented as:

[0094] M * = {(e i ,e j ) | dist(e i ,e j ) ≤ δ} (10)

[0095] Wherein, M * is the final matching result set, which ensures that the entity pair meets the consistency in semantics and geographical space. (e i ,e j ) represents an entity pair composed of entity e i and entity e j , δ is the set distance threshold value for final calibration. M represents the preliminary matching candidate set, which is screened out by formula (9). dist(e i ,e j ) represents the distance between entity e i and entity e j .

[0096] In summary, the cosine similarity and the Euclidean distance are weighted to calculate the entity matching, and a dynamic threshold value (based on K-means clustering analysis) is set. The geographical coordinate calibration combines the spatial index (such as R-Tree) and the Haversine distance calculation. If the geographical distance of the matching entity pair exceeds the preset threshold, the re-matching mechanism is triggered. Through this design, the accuracy and robustness of the alignment result are ensured.

[0097] The application can be applied to various scenarios, such as spatial query, knowledge graph construction, etc. For example, for knowledge graph construction in urban traffic management, entities such as subway stations and bus stops may be described by different data sources. Text data is used to describe station names, operating hours, service descriptions (such as "A subway station, open from 6:00 to 23:00 on weekdays, with barrier-free access"), etc. Image data is used to describe street view images (including station boards, entrances, waiting areas), station facility photos (such as ticket vending machines, security equipment), etc. Geospatial data is used to describe GPS coordinates (such as longitude 116.4074°, latitude 39.9042°), station topological relationships (such as connection paths between adjacent stations), etc. The application process of the application includes:

[0098] 1) Data preprocessing and feature extraction

[0099] Text data uses an improved DeepSeek model to extract semantic features, identify key entities (such as "A subway station") and attributes (such as "barrier-free access").

[0100] Image data extracts visual features through ResNet-50+SE Block, focusing on key areas such as station board text and entrance structure.

[0101] Geographic data uses dynamic edge weight GraphSAGE to model station topological relationships and dynamically update proximity weights during morning and evening peak hours.

[0102] 2) Unified semantic mapping

[0103] Through the TriFuser-X multi-modal fusion model, text, image, and geographic features are mapped to a unified semantic space to generate a fusion representation vector v. For example, the fusion vector v of the text "A subway station" and the corresponding street view image combines semantic description, visual features, and coordinate information.

[0104] 3) Dynamic weight distribution and alignment

[0105] Weight calculation: if the text description of a station is complete (such as confidence S t = 0.8), the image is blurred (such as Si = 0.3), and the geographic data is accurate (such as S g = 0.9), then the dynamic weight is:

[0106]

[0107] w i ≈ 0.01

[0108] w g ≈ 0.87

[0109] Alignment operation: By comparing the similarity of multi-modal representations of the same site through contrastive learning, negative samples with similar names but deviated locations (e.g., "A subway station" and "A bus station") are excluded.

[0110] Geographic coordinate calibration:

[0111] The geographic distance between the preliminary matched entity pair (e.g., text "A subway station" and image site) is 150 meters, which exceeds the threshold δ = 50 meters, triggering the re-matching mechanism:

[0112] Re-matching strategy: Combine spatial index (R-Tree) to retrieve nearby sites, re-evaluate semantic similarity, and finally match to the correct entity with a coordinate deviation of <10 meters.

[0113] Output results:

[0114] Generate aligned entity nodes in the knowledge graph, associate text descriptions, image links, and accurate coordinates, support intelligent navigation and facility query.

[0115] In summary, the present application uses deep neural networks (including convolutional neural networks and graph neural networks) and combines them with advanced large language models (such as DeepSeek) to process multi-modal data. Through this combination, joint semantic analysis of text, images, and geographic spatial data can be achieved, allowing for the deep mining of hidden semantic information in different modal data and effective integration and correlation of this information. In terms of entity alignment, the present application exhibits good performance, achieving accurate entity alignment and ensuring that the same entities in different data sources are accurately identified and matched, especially suitable for smart city, geographic information system (GIS), and cross-modal retrieval scenarios. Compared with existing technologies, the advantages of the present application mainly lie in the following aspects:

[0116] 1) In view of the differences in quality of different modal data, the present application uses adaptive feature fusion technology to dynamically adjust the weights of each modal information, ensuring the accuracy and robustness of the fusion results.

[0117] 2) The present application uses the powerful semantic understanding ability of large language models to map text, images, and geographic data to a unified semantic space, improving the collaborative representation learning effect between different modalities through multi-modal collaborative representation learning, thereby improving the accuracy of entity alignment.

[0118] 3) The present application introduces a multi-modal variational auto-encoding method to actively complete the missing modal information, enhancing the model's ability to handle incomplete data.

[0119] 4) The application further improves the accuracy and practicability of entity alignment by modeling the relationship between different modalities through an inter-modal contrast learning method.

[0120] The application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0121] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magnetically encoded device such as magnetic strip cards, an optically encoded device such as a compact disc (CD) or DVD, and / or any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0122] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0123] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0124] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0125] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0126] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0127] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0128] Embodiments of the present application have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the described embodiments are possible, and all such modifications and variations are intended to be within the scope of the described embodiments. The description used herein is intended to best explain the principles of the embodiments, the practical application, and the best mode of the present application, to make this disclosure understood in the art. The scope of the present application is defined by the appended claims.

Claims

1. An entity alignment method based on adaptive fusion of a large language model, comprising the following steps: Preprocessing the multimodal data to obtain multimodal data features, wherein the multimodal data includes text data, image data, and geospatial data, and the multimodal data features include semantic features corresponding to the text data, image features corresponding to the image data, and topological relationship features corresponding to the geospatial data; Mapping the multimodal data features to a unified semantic space to obtain an entity representation vector in the unified semantic space; For the entity representation vectors in the unified semantic space, dynamic weights are assigned according to the confidence of the different modal data, and then a weighted fusion entity representation is calculated based on the dynamic weights. After alignment through contrastive learning and attention mechanism matching, an aligned multimodal fusion entity representation is obtained; For the aligned multimodal fusion entity representation, entity matching and geographic coordinate calibration are performed to obtain entity matching results.

2. The method according to claim 1, characterized in that The semantic features are extracted using a large language model, which is the DeepSeek model. Based on the Transformer architecture, a multi-granularity adaptive pooling module is embedded. Word vector weights are dynamically allocated through a learnable gating mechanism to achieve dynamic semantic aggregation from word level, sentence level to paragraph level, and extract multi-granularity semantic features. The image features are extracted using a ResNet network. The ResNet network embeds a channel attention module after the convolution layer to focus on areas related to geographic entities in the image. The topological relationship features are extracted using GraphSAGE, a graph neural network for graph node embedding learning, which introduces a dynamic edge weight update mechanism and optimizes topological relationship modeling in combination with the spatiotemporal proximity of geographic entities.

3. The method according to claim 2, characterized in that The image features are obtained based on the following formula: F out =F in ×σ(W2ReLU(W1GAP(F in ))) Among them, GAP represents global average pooling, W1 and W2 are training parameters, σ is the Sigmoid function, and F in Represents the features input to the ResNet network, F out Embedded representation of generated image features.

4. The method according to claim 2, characterized in that The dynamic edge weight update mechanism is expressed as: Where α is the update rate, represents the edge weight between node u and node v in the kth iteration, sim geo represents the similarity calculated based on geographic spatiotemporal proximity, (u, v) represents a pair of adjacent nodes in the geospatial topological graph, and k represents the number of message passing iterations.

5. The method according to claim 2, characterized in that The entity representation of the weighted fusion is set as: F=w t ·F text +w i ·F image +w g ·F geo Among them, w t is the dynamic weight of text modal data, w i is the dynamic weight of image modality data, w g is the dynamic weight of the geographic data modality. The dynamic weight of each modality data is calculated uniformly according to the following formula: Among them, s m represents the confidence score of modality m, m represents the modality of the data, m′ represents traversing all modalities, T represents the temperature coefficient, t represents text, i represents image, and g represents geospatial data.

6. The method according to claim 1, characterized in that The multimodal data features are mapped to a unified semantic space using a pre-trained multimodal fusion model, and during the pre-training process, the training accuracy is measured based on contrast loss and modality reconstruction loss.

7. The method according to claim 5, characterized in that The confidence score Si of the image modality is determined according to the following formula: Si=Sigmoid(β·‖F image ‖2+γ·Entropy(F image )) Among them, F image Image features extracted by ResNet network. ‖F image ‖2 represents the L2 norm of the image feature, Entropy(F image ) is the information entropy of the image features, β and γ are learnable parameters; Confidence score S of geographic data modality g Determined according to the following formula: Among them, N(u) is the neighbor set of node u, |N(u)| is the number of neighbors of node u, is the dynamic edge weight between node u and node v in the kth iteration, sim geo (u,v) represents the similarity of geographic spatiotemporal proximity.

8. The method according to claim 1, characterized in that The entity matching and geographic coordinate calibration include: For each pair of entity representations, calculate their similarity score in the unified semantic space, expressed as: Sim(e1,e2)=0.7·cos(v1,v2)+0.3·exp(-0.001·Haversine(c1,c2)) Where (e1, e2) represents the two entities to be matched, v1, v2 represent the vector representations of entities e1 and e2 in the unified semantic space, c1, c2 represent the geographic coordinates of entities e1 and e2, cos(v1, v2) represents the cosine similarity between vectors v1 and v2, and Haversine(c1, c2) represents the geographic distance calculated based on the Haversine formula. According to the set similarity threshold δ, a matching candidate set M is formed; Input the geographic coordinate information of each pair of entities in the matching candidate set M, and for each pair of entities (e i ,e j ), calculate its geographic space distance, and then determine the entity matching result according to the set distance threshold, which is expressed as: M * ={(and i ,And j )∈M│dist(e i ,And j )≤δ} Among them, M * is the entity matching result set, (e i ,e j ) represents entity e i and entity e j The entity pair is composed of δ, which is the set distance threshold, dist(e i ,e j ) represents entity e i and entity e j The distance between them.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images